ClouderaManager 启动spark时内存限制问题

Spark On Yarn执行中executor内存限制问题

15/10/16 16:56:04 INFO RMProxy: Connecting to ResourceManager at InterFinance01/10.238.18.80:8032

15/10/16 16:56:04 INFO Client: Requesting a new application from cluster with 3 NodeManagers
15/10/16 16:56:04 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (1024 MB per container)
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!
        at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:191)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
       

解决方法如下：

从以上红色字体可以看到验证application需要的内存超过container的最大内存

所以配置yarn中的yarn.scheduler.maximum-allocation-mb这一个参数，使container的最大内存达到(1024+384 MB)错误提示的内存要求