spark1.63升级到spark2.3.1环境变量配置

由于spark2.3需要jdk1.8的支持，因此spark-submit与spark-sql需要指定jdk版本，具体方式见测试用例

1.修改个人用户配置文件.bashrc ，注释以前的spark环境与java环境，添加

vi ~/.bashrc

#export SPARK_HOME=/opt/core/spark-1.6.3-bin-hadoop2.6 #export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera export SPARK_HOME=/opt/core/spark-2.3.1-bin-hadoop2.6 export JAVA_HOME=/usr/local/jdk1.8.0_161 export PATH=${SPARK_HOME}/bin:${JAVA_HOME}/bin:$PATH

保存之后  执行：source .bashrc

到此配置完成

 

 

2.运行计算圆周率测试用例：

spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 1 \
--conf "spark.executorEnv.JAVA_HOME=/usr/local/jdk1.8.0_161" \
--conf "spark.yarn.appMasterEnv.JAVA_HOME=/usr/local/jdk1.8.0_161" \
--queue root.bigdata.statistics \
/opt/core/spark/examples/jars/spark-examples*.jar 10

3.启动spark-sql样例：

spark-sql --master yarn \
--deploy-mode client \
--driver-memory 2g \
--executor-memory 8g \
--executor-cores 4 \
--conf "spark.executorEnv.JAVA_HOME=/usr/local/jdk1.8.0_161" \
--conf "spark.yarn.appMasterEnv.JAVA_HOME=/usr/local/jdk1.8.0_161" \
--queue root.bigdata.statistics