Spark任务提交时报错bad substitution

Spark任务提交时报错bad substitution

问题描述

在集群中使用spark-submit提交spark任务时，报错

Exception message: /hadoop/yarn/local/usercache/qxadmin/appcache/application_1631068541144_0002/container_e10_1631068541144_0002_01_000001/launch_container.sh:行22: $PWD:$PWD/spark_conf:KaTeX parse error: Expected group after '_' at position 5: PWD/_̲_spark_libs__/*…HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/:/usr/hdp/current/hadoop-client/lib/:/usr/hdp/current/hadoop-hdfs-client/:/usr/hdp/current/hadoop-hdfs-client/lib/:/usr/hdp/current/hadoop-yarn-client/:/usr/hdp/current/hadoop-yarn-client/lib/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/\ast :$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/:$PWD/mr-framework/hadoop/share/hadoop/common/\ast :$PWD/mr-framework/hadoop/share/hadoop/common/lib/:$PWD/mr-framework/hadoop/share/hadoop/yarn/\ast :$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/\ast :$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/\ast :/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.$hdp.version.jar:/etc/hadoop/conf/secure:$PWD/spark_conf/hadoop_conf: 坏的替换

如下图所示：

产生原因

hadoop的mapred-site.xml文件中中引用了hdp.version变量，但是该变量在配置中找不到。

解决方案

1.在集群管理页面中，yarn组件及MapReduce2组件，手动添加hdp.version变量。值需要去集群找，如下图。

2.客户端机器上， /usr/hdp/2.5.0.0-1245/spark2/conf/spark-default.conf 路径下，添加

spark.driver.extraJavaOptions -Dhdp.version=2.5.0.0-1245
spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.0.0-1245

3.客户端机器上，找到yarn和MapReduce2组件的mapred-site.xml文件和yarn-site.xml文件，在文件里分别加上

    <property>
        <name>hdp.versionname>
        <value>2.7.3.2.5.0.0-1245value>
    property>

4.服务器集群三台，/usr/hdp/2.5.0.0-1245/spark2/conf/spark-default.conf路径下，都添加

spark.driver.extraJavaOptions -Dhdp.version=2.5.0.0-1245
spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.0.0-1245

traJavaOptions -Dhdp.version=2.5.0.0-1245

不同的hdp集群可能有不同的情况，目前我这里少了任何一个环节都不行。