配置和启动Spark | local模式以及standalone模式

Spark Local模式

直接在 $SPARK_HOME/bin/spark-shell 中运行spark RDD程序，在UI界面http://nodename:4040/中可以看到jobs的执行情况

[elon@spark ~]$ spark-shell --master local
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/01/22 18:46:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://spark:4040
Spark context available as 'sc' (master = local, app id = local-1516618007091).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val source = sc.wholeTextFiles("/data")
source: org.apache.spark.rdd.RDD[(String, String)] = /data MapPartitionsRDD[1] at wholeTextFiles at :24

scala> source.collect()
res0: Array[(String, String)] =
Array((hdfs://spark:9000/data/data1.txt,"Hello,I'm fine!
How old are you?
What's your name?
I'm a student and come from China.
GoodBye~
"), (hdfs://spark:9000/data/data2.txt,"192.168.1.101
192.168.1.202
Apache.org
Hadoop
Spark
"), (hdfs://spark:9000/data/data3.txt,"GoodBye My dear!
Nice to meet you next time
"))

Spark standalone模式

配置SPARK_MASTER_IP环境变量

首先在$SPARK_HOME/conf/spark-env.sh中设置spark环境变量export SPARK_MASTER_IP=ipAddress，指定Master运行的主机IP地址，如果是在集群中执行时，有多个节点组件连接到Master时，具体配置看starting-a-cluster-manually

手动启动群集

你可以通过执行以下命令开启Master：

./$SPARK_HOME/sbin/start-master.sh

随之MasterUI界面也就启动了，具体 http://hadoop:8080/，在该页面可以查看到master-spark-URL

相应的，如果将其他从节点组件连接到Master时，可以在从节点主机上键入以下命令：

./$SPARK_HOME/sbin/start-slave.sh 

将应用程序连接到群集

因此，对集群运行交互式Spark shell时，通过以下命令行：

./$SPARK_HOME/bin/spark-shell --master spark://IP:PORT

此时可以查看Spark shell application UI http://hadoop:4041

● 详细配置，请移步到官方文档 spark-standalone

转载请注明出处：http://blog.csdn.net/coder__cs/article/details/79118590
本文出自【elon33的博客】