Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

Spark submit任务到Spark集群时,会出现如下异常:

Exception 1:Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

查看Spark logs文件spark-Spark-org.apache.spark.deploy.master.Master-1-hadoop1.out发现:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第1张图片

此时的Spark Web UI界面如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第2张图片

Reason:

之前在Spark配置文件spark-env.sh中,SPARK_LOCAL_IP的配置是localhost,内存为512M,所以在Spark UI界面中Workers显示均对应到主机hadoop1默认的的localhost,只是给它分配的workers对应不同的端口而已,而之前最大内存为2.9G,所以5 * 512M > 2.9G的,故上述报错。

Solution:

修改Spark配置文件spark-env.sh,将SPARK_LOCAL_IP的localhost修改为对应的主机名称(hadoop1,hadoop2...),并修改SPARK_WORKER_MEMORY的内存配置 < 对应机器分配的内存 即可

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第3张图片

 

提交任务(WordCount)到Spark集群中,相应脚本如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第4张图片

执行脚本,运行Spark任务,过程如下:

./runSpark.sh

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第5张图片

 对应的WordCount结果如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第6张图片

此时对应的Spark运行UI界面如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory_第7张图片

你可能感兴趣的:(resource)