sxjlinux

mac系统下安装hadoop+hbase+spark单机版+IDEA和scala编写使用spark来计算的helloworld程序

一、启动ssh服务

1、首先找到系统偏好设置，如下图所示：

2、然后选择共享，如下图所示：

3、然后将远程登录选上，并选择所有用户，如下图所示：

4、此时即可通过ssh登录了。

二、建立ssh无密码登录本机

1、ssh生成密钥有rsa和dsa两种生成方式，默认情况下采用rsa方式，首先主机上创建ssh-key，这里我们采用rsa方式。使用如下命令(P是要大写的，后面跟""，表示无密码)

ssh-keygen -t rsa

如下图所示：

2、此时会在~/.ssh/下生成两个文件：id_rsa和id_rsa.pub这两个文件是成对出现的，进入到该目录查看，如下图所示：

3、将id_rsa.pub文件authorized_keys授权文件中，开始是没有authorized_keys文件的，只需要执行如下命令即可：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

4、如果存在known_hosts文件则将其删除掉，然后修改authorized_keys的权限使用如下命令修改：

chmod og-wx ~/.ssh/authorized_keys

5、然后修改$HOME的权限为750，使用如下命令修改：

chmod 750 $HOME

如下图所示：

6、然后输入ssh localhost测试即可成功，如下图所示：

三、安装hadoop

1、首先从https://hadoop.apache.org/releases.html下载，如下版本：

2、这里选择hadoop2.7.7的Binary版本。

3、将下载好的放置在某个目录，我的是放在/sunxj/InstallFile/如下图所示：

5、使用如下命令解压：

tar -xzvf hadoop-2.7.7.tar.gz

6、进入到hadoop-2.7.7/etc/hadoop/修改到hadoop-env.sh，修改export JAVA_HOME对应的目录，需要找到java的安装目录，我的是：/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/

如下图所示：

7、在hadoop-2.7.7目录中一个hdfs目录和三个子目录，如

hadoop-2.7.3/hdfs
hadoop-2.7.3/hdfs/tmp
hadoop-2.7.3/hdfs/name
hadoop-2.7.3/hdfs/data

如下图所示：

8、配置core-site.xml文件，使用如下命令打开：

vim etc/hadoop/core-site.xml

然后在中如下配置是读写sequence file 的 buffer size,可减少 I/O 次数。在大型的 Hadoop cluster，建议可设定为 65536 到 131072，默认值 4096.按照教程配置了131072：


   hadoop.tmp.dir
   file:/sunxj/InstallFile/hadoop-2.7.7/hdfs/tmp
   A base for other temporary directories.
 
 
  io.file.buffer.size
   131072
 
 
   fs.defaultFS
   hdfs://localhost:9000

如下图所示：

注意：第一个属性中的value和我们之前创建的/sunxj/InstallFile/hadoop-2.7.7/hdfs/tmp路径要一致。

9、在mapred-env.sh加入JAVA_HOME，如下图所示：

12、在yarn-env.sh加入JAVA_HOME，如下图所示：

13、配置hdfs-site.xml，使用如下命令打开文件

vim etc/hadoop/hdfs-site.xml

然后在中加入以下代码：

         
                 dfs.replication
                 1
         
         
                 dfs.namenode.name.dir
                 file:/sunxj/InstallFile/hadoop-2.7.7/hdfs/name
                 true
        
         
                 dfs.datanode.data.dir
                 file:/sunxj/InstallFile/hadoop-2.7.7/hdfs/data
                 true
         
         
                 dfs.namenode.secondary.http-address
                 localhost:9001
         
         
                 dfs.webhdfs.enabled
                 true
         
         
                 dfs.permissions
                 false

注意：其中第二个dfs.namenode.name.dir和dfs.datanode.data.dir的value和之前创建的/hdfs/name和/hdfs/data路径一致；

如下图所示：

14、复制mapred-site.xml.template文件，并命名为mapred-site.xml，使用如下命令拷贝

cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml

并编辑mapred-site.xml，在标签中添加以下代码：

 
                 mapreduce.framework.name
                 yarn
         
          
                mapreduce.jobhistory.address 
                localhost:10020
        
        
                mapreduce.jobhistory.webapp.address 
                localhost:19888
        
    
        mapred.job.tracker
        localhost:8021

如下图所示：

15、配置yarn-site.xml，使用如下命令打开

vim etc/hadoop/yarn-site.xml

然后在标签中添加以下代码：

        
                 yarn.resourcemanager.address
                 localhost:18040
         
         
                 yarn.resourcemanager.scheduler.address
                 localhost:18030
         
         
                 yarn.resourcemanager.webapp.address
                 localhost:18088
         
         
                 yarn.resourcemanager.resource-tracker.address
                 localhost:18025
         
         
                 yarn.resourcemanager.admin.address
                 localhost:18141
         
         
                 yarn.nodemanager.aux-services
                 mapreduce_shuffle
         
         
                yarn.nodemanager.auxservices.mapreduce.shuffle.class
                 org.apache.hadoop.mapred.ShuffleHandler

如下图所示：

16、配置hadoop环境变量，使用sudo vim /etc/profile打开文件，添加入下代码：

export HADOOP_HOME=/sunxj/InstallFile/hadoop-2.7.7
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

如下图所示：

18、输入如下命令使配置立即生效

source /etc/profile

19、此时使用hadoop用户登录通过如下命令进行格式化

hdfs namenode -format

如下图所示：

注意：/sunxj/InstallFile/hdfs/name/current目录具有写入权限

20、然后使用start-all.sh启动，如下图所示：

21、然后输入jps查看进程，如下图所示：

22、通过web查看集群运行情况，YARN的web页面，然后端口是用yarn-site.xml配置文件中的yarn.resourcemanager.webapp.address指定的，我们配置的是18088，那么在浏览器中输入：http://localhost:18088即可打开界面，如下图所示：

界面中显示的记录则是执行的任务个数。

23、HDFS界面如果没有更改端口，则默认的端口是50070：http://localhost:50070，如下图所示：

24、用自带的样例测试hadoop集群能不能正常跑任务,使用如下命令测试：

hadoop jar /sunxj/InstallFile/hadoop-2.7.7/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar pi 10 10

运行结果如下：

Number of Maps  = 10
Samples per Map = 10
19/01/10 01:28:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
19/01/10 01:28:30 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:18040
19/01/10 01:28:31 INFO input.FileInputFormat: Total input paths to process : 10
19/01/10 01:28:31 INFO mapreduce.JobSubmitter: number of splits:10
19/01/10 01:28:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547054421110_0001
19/01/10 01:28:32 INFO impl.YarnClientImpl: Submitted application application_1547054421110_0001
19/01/10 01:28:32 INFO mapreduce.Job: The url to track the job: http://192.168.0.104:18088/proxy/application_1547054421110_0001/
19/01/10 01:28:32 INFO mapreduce.Job: Running job: job_1547054421110_0001
19/01/10 01:28:41 INFO mapreduce.Job: Job job_1547054421110_0001 running in uber mode : false
19/01/10 01:28:41 INFO mapreduce.Job:  map 0% reduce 0%
19/01/10 01:28:58 INFO mapreduce.Job:  map 30% reduce 0%
19/01/10 01:28:59 INFO mapreduce.Job:  map 60% reduce 0%
19/01/10 01:29:09 INFO mapreduce.Job:  map 70% reduce 0%
19/01/10 01:29:10 INFO mapreduce.Job:  map 100% reduce 0%
19/01/10 01:29:11 INFO mapreduce.Job:  map 100% reduce 100%
19/01/10 01:29:12 INFO mapreduce.Job: Job job_1547054421110_0001 completed successfully
19/01/10 01:29:12 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=226
		FILE: Number of bytes written=1355618
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2690
		HDFS: Number of bytes written=215
		HDFS: Number of read operations=43
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Job Counters 
		Launched map tasks=10
		Launched reduce tasks=1
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=125151
		Total time spent by all reduces in occupied slots (ms)=11133
		Total time spent by all map tasks (ms)=125151
		Total time spent by all reduce tasks (ms)=11133
		Total vcore-milliseconds taken by all map tasks=125151
		Total vcore-milliseconds taken by all reduce tasks=11133
		Total megabyte-milliseconds taken by all map tasks=128154624
		Total megabyte-milliseconds taken by all reduce tasks=11400192
	Map-Reduce Framework
		Map input records=10
		Map output records=20
		Map output bytes=180
		Map output materialized bytes=280
		Input split bytes=1510
		Combine input records=0
		Combine output records=0
		Reduce input groups=2
		Reduce shuffle bytes=280
		Reduce input records=20
		Reduce output records=0
		Spilled Records=40
		Shuffled Maps =10
		Failed Shuffles=0
		Merged Map outputs=10
		GC time elapsed (ms)=989
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
		Total committed heap usage (bytes)=1911554048
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=1180
	File Output Format Counters 
		Bytes Written=97
Job Finished in 41.692 seconds
Estimated value of Pi is 3.20000000000000000000

25、那么此时hadoop安装成功。

四、安装hbase

1、在https://hbase.apache.org/downloads.html下载hbase，我们选择2.0.4版本的bin，如下图所示：

2、将下载好的hbase-2.0.4-bin.tar.gz放置在/sunxj/InstallFile/目录，如下图所示：

3、然后使用如下命令进行解压：

tar -xzvf hbase-2.0.4-bin.tar.gz

4、配置hbase的环境变量编辑 /etc/profile 文件，通过如下命令进行编辑：

sudo vim /etc/profile

然后在该文件中加入：

export HBASE_HOME=/sunxj/InstallFile/hbase-2.0.4
export PATH=$HBASE_HOME/bin:$PATH

如下图所示：

5、保存退出，然后执行：source /etc/profile命令使之生效.

6、然后即可通过如下命令查看版本：

hbase version

如下图所示：

7、修改配置文件，切换到 /sunxj/InstallFile/hbase-2.0.4/conf下,如下图所示：

8、使用如下命令在hbase-2.0.4创建一个pids，用于存放用于运行hbase进程的pid文件：

mkdir /sunxj/InstallFile/hbase-2.0.4/pids

如下图所示：

9、修改hbase-env.sh，使用如下命令编辑：

vim hbase-env.sh

然后在文件末尾加入如下信息：

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home
export HADOOP_HOME=/sunxj/InstallFile/hadoop-2.7.7
export HBASE_HOME=/sunxj/InstallFile/hbase-2.0.4
export HBASE_CLASSPATH=/sunxj/InstallFile/hadoop-2.7.7/etc/hadoop
export HBASE_PID_DIR=/sunxj/InstallFile/hbase-2.0.4/pids
export HBASE_MANAGES_ZK=false

如下图所示：

其中HBASE_PID_DIR的路径就是刚才创建的pids文件夹路径，HBASE_MANAGES_ZK则表示启用hbase自己的zookeeper。

10、使用如下命令创建一个tmp文件夹，如下图所示：

mkdir /sunxj/InstallFile/hbase-2.0.4/tmp

如下图所示：

11、然后修改 hbase-site.xml，编辑hbase-site.xml 文件，在添加如下配置

        
                hbase.rootdir
                hdfs://localhost:9000/hbase
                The directory shared byregion servers.
        
        
        
                hbase.tmp.dir
                /sunxj/InstallFile/hbase-2.0.4/tmp
        
        
        
                hbase.cluster.distributed
                false
        
        
        
                hbase.master
                localhost:60000

如下图所示：

12、先启动Hadoop后在启动hbase，使用如下命令进行启动：

start-all.sh 
start-hbase.sh

如下图所示：

13、然后用jps查看进程情况，如下图所示：

14、然后通过web地址来查看，地址为：http://localhost16010，如下图所示：

15、此时说明hbase集群已经搭建好了，那么可以通过以下命令进入hbase的shell命令命令行：

hbase shell

如下图所示：

16、然后输入status可以查看hbase的状态，如下图所示：

意思是1个主hbase并且是活动的，0个是备用节点，一共有1个服务

17、通过如下命令创建一个表：

create 'person','name','age'

如下图所示：

其中person是表名，name和age是字段名

18、开始向表中插入数据使用如下命令：

put 'person','1','name','sun'
put 'person','1','age',24

意思是向person的第一行中的name列插入sun，向person的第一行中的age列插入24

如下图所示：

19、通过scan '表名'来查看表中所有的记录，如下图所示：

20、具体的请看下表操作方法：

HBase Shell的一些基本操作命令，列出了几个常用的HBase Shell命令，如下：

名称	命令表达式
查看存在哪些表	list
创建表	create '表名称', '列名称1','列名称2','列名称N'
添加记录	put '表名称', '行名称', '列名称:', '值'
查看记录	get '表名称', '行名称'
查看表中的记录总数	count '表名称'
删除记录	delete '表名' ,'行名称' , '列名称'
删除一张表	先要屏蔽该表，才能对该表进行删除，第一步 disable '表名称' 第二步 drop '表名称'
查看所有记录	scan "表名称"
查看某个表某个列中所有数据	scan "表名称" , ['列名称:']
更新记录	就是重写一遍进行覆

五、安装spark

1、安装scala环境，在mac上直接执行如下指令即可安装：

brew install scala

如下图所示：

2、然后执行scala -version查看scala版本，并检测是否配置成功，如下图所示：

3、从http://spark.apache.org/downloads.html下载spark，选择spark和hadoop版本，如下图所示：

4、然后将压缩包放在/sunxj/InstallFile/目录，如下图所示：

5、此时权限哪里多了一个@这是mac自带的，可以通过 xattr -c -r *命令去掉，如下图所示：

6、然后使用如下命令解压spark：

tar -xzvf spark-2.4.0-bin-hadoop2.7.tgz

7、进入到/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/conf/目录修改配置文件，如下图所示：

8、复制spark-env.sh.template并重命名为spark-env.sh，如下图所示：

9、然后编辑spark-env.sh，在文件的末尾添加入下配置：

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home
export HADOOP_HOME=/sunxj/InstallFile/hadoop-2.7.7
export HADOOP_CONF_DIR=/sunxj/InstallFile/hadoop-2.7.7/etc/hadoop
export SPARK_WORKER_MEMORY=500m
export SPARK_WORKER_CORES=1

如下图所示：

注意：如果存在export SPARK_MASTER_IP则需要将此项注释掉。然后保存退出。

10、配置环境变量，编辑/etc/profile文件，在最后加入如下配置：

export SPARK_HOME=/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7
export PATH="$SPARK_HOME/bin:$PATH"

如下图所示：

11、然后使用source /etc/profile使之配置生效，依次启动启动hadoop、hbase、spark

命令如下：

(1)、start-all.sh（注意：有时候第一次启动时namenode节点都是standby状态，需要停止在启动即可）

(2)、start-hbase.sh

(3)、 /sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/sbin/start-all.sh 如下图所示：

12、然后查看进程，如下图所示：

13、此时Master进程和Worker进程都以启动成功，

14、打开web页面查看http://localhost:8080/，如下图所示：

如果主机名不好记可以更改成一个其他的名称，比如我更改为了：sunxj-mac-air，然后重启hadoop、hbase、spark，再次查看web如下图所示：

15、测试，通过spark-shell 命令进行测试，在master节点上输入该命令进入scala环境，如下图所示：

16、然后创建一个worldcount.txt文件，使用如下命令创建：

vim worldcount.txt

文件内容如下：

hello
hello
world
world
hello
linux
spark
window
linux
spark
spark
linux
hello
sunxj
window

如下图所示：

17、然后通过如下命令在hadoop的fs系统中创建一个user_data的文件夹：

hadoop fs -mkdir /user_data

如下图所示：

18、然后通过如下命令查看hadoop的fs系统目录信息：

hadoop fs -ls /

如下图所示：

19、此时发现有有一个警告 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable，那么更改hadoop-env.sh文件，在该文件中加入如下代码：

export HADOOP_OPTS="-Djava.library.path=/sunxj/InstallFile/hadoop-2.7.7/lib/native"

如下图所示：

20、重启hadoop在次查看还有有警告，如下图所示：

21、更改core-site.xml 文件，在该文件中加入如下代码：


   hadoop.native.lib
   false

如下图所示：

22、重启hadoop在次查看还有有警告，如下图所示：

23、此警告不影响程序执行，先跳过，以后看如何解决，那么通过如下命令将文件上传至hdfs系统：

hadoop fs -put worldcount.txt /user_data

如下图所示：

24、然后在spark即scala命令行依次输入如下代码：

val file=sc.textFile("hdfs://localhost:9000/user_data/worldcount.txt")
val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
rdd.collect()
rdd.foreach(println)

如下图所示：

25、此时spark搭建完毕，注意在写hdfs路径时，可以通过hadoop fs -ls hdfs://localhost:9000/，一步一步的找路径，如下图所示：

26、最后可写两个脚本start-hadoop-hbase-spark.sh和stop-hadoop-hbase-spark.sh来统一启动和停止这些服务，我们在/sunxj/InstallFile目录创建一个hadoop-hbase-spark目录，在次目录中创建两个脚本文件，使用如下命令创建：

vim start-hadoop-hbase-spark.sh

文件内容如下：

start-all.sh
start-hbase.sh
/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/sbin/start-all.sh

如下图所示：

27、用如下命令创建stop-hadoop-hbase-spark.sh文件：

vim stop-hadoop-hbase-spark.sh

文件内容如下：

/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/sbin/stop-all.sh 
stop-hbase.sh
stop-all.sh

如下图所示：

29、修改权限为777，如下图所示：

30、然后将该目录添加到/etc/profile文件中，添加入如下代码：

export HADOOP_HBASE_SPARK_START_STOP_SH_HOME=/sunxj/InstallFile/hadoop-hbase-spark
export PATH=$HADOOP_HBASE_SPARK_START_STOP_SH_HOME:$PATH

如下图所示：

31、然后使用source /etc/profile命令使之生效，然后在终端直接输入start-hadoop-hbase-spark.sh启动hadoop、hbase、spark，如下图所示：

32、在终端直接输入stop-hadoop-hbase-spark.sh停止spark、hbase、hadoop，如下图所示：

六、使用IDEA+scala编写使用spark来计算的helloworld程序

1、首先打开IDEA，如下图所示：

2、然后找到右下角位置的Configure->Plugins，如下图所示：

3、在Marketplace搜索scala查看scala插件是否安装，如果已安装Installed则是灰色的，未安装则可以进行安装，如下图所示：

4、在安装好之后开始创建工程，如下图所示：

5、选择Maven，选择SDK，然后选择Next，如下图所示：

6、输入GroupId和ArtifactId，然后Next，如下图所示：

7、输入Project name的名称，然后点击Finish，如下图所示：

8、然后选择Project Structure,如下图所示：

9、然后选择Libraries,接着选择+号->Scala SDK，如下图所示：

10、此时会让其选择版本，如果此时没有想要的版本可以选择Download...按钮，如下图所示：

11、此时就会弹出让其下载的版本，并且可以选择下载那个版本，如下图所示：

12、选择好之后即可开始下载，下载时间比较长：

13、再次打开即可发现scala的版本已经被选择，或者直接可以选择，如下图所示：

14、点击OK之后会让选择项目，如下图所示：

15、点击OK之后即可添加Scala的Lib，如下图所示：

16、最后在点击OK进行应用，在配置之前要确定spark使用的scala的版本，登录到spark的集群查看scala的版本，可以通过spark-shell 命令进行测试，在spark上输入该命令进入scala环境如下图所示：

否则会出现如下错误：

/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:50675,suspend=y,server=n -javaagent:/Users/sunxiaoju/Library/Caches/IntelliJIdea2018.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/tools.jar:/sunxj/work/git/sparktest/target/classes:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.8.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.8.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/srcs/scala-reflect-2.12.8-sources.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/srcs/scala-library-2.12.8-sources.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-core_2.12/2.4.0/spark-core_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/Users/sunxiaoju/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/Users/sunxiaoju/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill_2.12/0.9.3/chill_2.12-0.9.3.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/sunxiaoju/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-launcher_2.12/2.4.0/spark-launcher_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-kvstore_2.12/2.4.0/spark-kvstore_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-common_2.12/2.4.0/spark-network-common_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-shuffle_2.12/2.4.0/spark-network-shuffle_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-unsafe_2.12/2.4.0/spark-unsafe_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/Users/sunxiaoju/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/sunxiaoju/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/sunxiaoju/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/sunxiaoju/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/Users/sunxiaoju/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/sunxiaoju/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/sunxiaoju/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/sunxiaoju/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-library/2.12.7/scala-library-2.12.7.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-jackson_2.12/3.5.3/json4s-jackson_2.12-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-core_2.12/3.5.3/json4s-core_2.12-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-ast_2.12/3.5.3/json4s-ast_2.12-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-scalap_2.12/3.5.3/json4s-scalap_2.12-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/Users/sunxiaoju/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.12/2.6.7.1/jackson-module-scala_2.12-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-reflect/2.12.1/scala-reflect-2.12.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/sunxiaoju/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/sunxiaoju/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/sunxiaoju/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/sunxiaoju/.m2/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-tags_2.12/2.4.0/spark-tags_2.12-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-client/3.1.1/hadoop-client-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-common/3.1.1/hadoop-common-3.1.1.jar:/Users/sunxiaoju/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar:/Users/sunxiaoju/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/sunxiaoju/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/sunxiaoju/.m2/repository/org/eclipse/jetty/jetty-servlet/9.3.19.v20170502/jetty-servlet-9.3.19.v20170502.jar:/Users/sunxiaoju/.m2/repository/org/eclipse/jetty/jetty-security/9.3.19.v20170502/jetty-security-9.3.19.v20170502.jar:/Users/sunxiaoju/.m2/repository/org/eclipse/jetty/jetty-webapp/9.3.19.v20170502/jetty-webapp-9.3.19.v20170502.jar:/Users/sunxiaoju/.m2/repository/org/eclipse/jetty/jetty-xml/9.3.19.v20170502/jetty-xml-9.3.19.v20170502.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-servlet/1.19/jersey-servlet-1.19.jar:/Users/sunxiaoju/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/sunxiaoju/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-configuration2/2.1.1/commons-configuration2-2.1.1.jar:/Users/sunxiaoju/.m2/repository/com/google/re2j/re2j/1.1/re2j-1.1.jar:/Users/sunxiaoju/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/sunxiaoju/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-auth/3.1.1/hadoop-auth-3.1.1.jar:/Users/sunxiaoju/.m2/repository/com/nimbusds/nimbus-jose-jwt/4.41.1/nimbus-jose-jwt-4.41.1.jar:/Users/sunxiaoju/.m2/repository/com/github/stephenc/jcip/jcip-annotations/1.0-1/jcip-annotations-1.0-1.jar:/Users/sunxiaoju/.m2/repository/net/minidev/json-smart/2.3/json-smart-2.3.jar:/Users/sunxiaoju/.m2/repository/net/minidev/accessors-smart/1.2/accessors-smart-1.2.jar:/Users/sunxiaoju/.m2/repository/org/ow2/asm/asm/5.0.4/asm-5.0.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-client/2.12.0/curator-client-2.12.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/htrace/htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-simplekdc/1.0.1/kerb-simplekdc-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-client/1.0.1/kerb-client-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerby-config/1.0.1/kerby-config-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-core/1.0.1/kerb-core-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerby-pkix/1.0.1/kerby-pkix-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerby-asn1/1.0.1/kerby-asn1-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerby-util/1.0.1/kerby-util-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-common/1.0.1/kerb-common-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-crypto/1.0.1/kerb-crypto-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-util/1.0.1/kerb-util-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/token-provider/1.0.1/token-provider-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-admin/1.0.1/kerb-admin-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-server/1.0.1/kerb-server-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerb-identity/1.0.1/kerb-identity-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/kerby/kerby-xdr/1.0.1/kerby-xdr-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/woodstox/woodstox-core/5.0.3/woodstox-core-5.0.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/3.1.1/hadoop-hdfs-client-3.1.1.jar:/Users/sunxiaoju/.m2/repository/com/squareup/okhttp/okhttp/2.7.5/okhttp-2.7.5.jar:/Users/sunxiaoju/.m2/repository/com/squareup/okio/okio/1.6.0/okio-1.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-api/3.1.1/hadoop-yarn-api-3.1.1.jar:/Users/sunxiaoju/.m2/repository/javax/xml/bind/jaxb-api/2.2.11/jaxb-api-2.2.11.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-client/3.1.1/hadoop-yarn-client-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.1/hadoop-mapreduce-client-core-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-common/3.1.1/hadoop-yarn-common-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/eclipse/jetty/jetty-util/9.3.19.v20170502/jetty-util-9.3.19.v20170502.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-core/1.19/jersey-core-1.19.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/jsr311-api/1.1.1/jsr311-api-1.1.1.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-client/1.19/jersey-client-1.19.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.7.8/jackson-module-jaxb-annotations-2.7.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.7.8/jackson-jaxrs-json-provider-2.7.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.7.8/jackson-jaxrs-base-2.7.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.1/hadoop-mapreduce-client-jobclient-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/3.1.1/hadoop-mapreduce-client-common-3.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-annotations/3.1.1/hadoop-annotations-3.1.1.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" com.sunxj.sparktest.WordCount
Connected to the target VM, address: '127.0.0.1:50675', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/06 22:53:38 INFO SparkContext: Running Spark version 2.4.0
19/01/06 22:53:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/06 22:53:40 INFO SparkContext: Submitted application: Spark 学习
19/01/06 22:53:40 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/06 22:53:40 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/06 22:53:40 INFO SecurityManager: Changing view acls groups to: 
19/01/06 22:53:40 INFO SecurityManager: Changing modify acls groups to: 
19/01/06 22:53:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/06 22:53:41 INFO Utils: Successfully started service 'sparkDriver' on port 50679.
19/01/06 22:53:41 INFO SparkEnv: Registering MapOutputTracker
19/01/06 22:53:41 INFO SparkEnv: Registering BlockManagerMaster
19/01/06 22:53:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/06 22:53:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/06 22:53:41 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-d772314d-fc3a-4ee0-8a69-241a5d73e78b
19/01/06 22:53:41 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/06 22:53:41 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/06 22:53:42 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/06 22:53:42 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/06 22:53:42 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.0.108:7077...
19/01/06 22:53:43 INFO TransportClientFactory: Successfully created connection to /192.168.0.108:7077 after 133 ms (0 ms spent in bootstraps)
19/01/06 22:54:02 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.0.108:7077...
19/01/06 22:54:22 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.0.108:7077...
19/01/06 22:54:42 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/01/06 22:54:42 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/01/06 22:54:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50690.
19/01/06 22:54:42 INFO NettyBlockTransferService: Server created on 192.168.0.104:50690
19/01/06 22:54:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/06 22:54:42 INFO SparkUI: Stopped Spark web UI at http://192.168.0.104:4040
19/01/06 22:54:42 INFO StandaloneSchedulerBackend: Shutting down all executors
19/01/06 22:54:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/01/06 22:54:42 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/01/06 22:54:42 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/01/06 22:54:42 INFO MemoryStore: MemoryStore cleared
19/01/06 22:54:42 INFO BlockManager: BlockManager stopped
19/01/06 22:54:42 INFO BlockManagerMaster: BlockManagerMaster stopped
19/01/06 22:54:42 WARN MetricsSystem: Stopping a MetricsSystem that is not running
19/01/06 22:54:42 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/01/06 22:54:42 INFO SparkContext: Successfully stopped SparkContext
19/01/06 22:54:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 50690, None)
19/01/06 22:54:42 ERROR SparkContext: Error initializing SparkContext.
java.lang.NullPointerException
	at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:64)
	at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:252)
	at org.apache.spark.SparkContext.(SparkContext.scala:510)
	at com.sunxj.sparktest.WordCount$.main(WordCount.scala:13)
	at com.sunxj.sparktest.WordCount.main(WordCount.scala)
19/01/06 22:54:42 INFO SparkContext: SparkContext already stopped.
Exception in thread "main" java.lang.NullPointerException
	at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:64)
	at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:252)
	at org.apache.spark.SparkContext.(SparkContext.scala:510)
	at com.sunxj.sparktest.WordCount$.main(WordCount.scala:13)
	at com.sunxj.sparktest.WordCount.main(WordCount.scala)
19/01/06 22:54:43 INFO ShutdownHookManager: Shutdown hook called
19/01/06 22:54:43 INFO ShutdownHookManager: Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-10c156e0-e3bf-44aa-92a7-77a4080a4c71
Disconnected from the target VM, address: '127.0.0.1:50675', transport: 'socket'

Process finished with exit code 1

如下图所示：

17、然后在pom.xml中添加相关依赖，如下配置：



    4.0.0

    spark
    spark-test
    1.0-SNAPSHOT

    
        
            org.apache.spark
            spark-core_2.11
            2.3.2
        
        
            org.apache.hadoop
            hadoop-client
            2.7.7

18、然后创建一个com.sunxj.sparktest包，并在该包中创建一个scala的类WordCount，如下代码所示：

package com.sunxj.sparktest

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("Spark 学习")
      .setMaster("spark://sunxj-mac-air.local:7077")
      .set("spark.executor.memory", "500m")
      .set("spark.cores.max", "1")


    val sc = new SparkContext(conf)
    //val line = sc.textFile(args(0))
    val file=sc.textFile("hdfs://localhost:9000/user_data/worldcount.txt")
    val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
    rdd.collect()
    rdd.foreach(println)
  }
}
lect()
    rdd.foreach(println)
  }
}

如下图所示：

19、其中spark://sunxj-mac-air.local:7077是spark集群中的ALIVE，如下图所示：

20、其中hdfs://localhost:9000是hadoop集群中的一个active节点,如下图所示：

而在hadoop的hdfs系统中是存在一个worldcount.txt文件的，位置在：hdfs://localhost:9000/user_data/worldcount.txt，如下图所示：

21、然后选择Run->Debug如下图所示：

23、然后选择WordCount，如下图所示：

24、此时会出现以下错误：

Connected to the target VM, address: '127.0.0.1:50188', transport: 'socket'
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
	at org.apache.spark.internal.config.ConfigHelpers$.stringToSeq(ConfigBuilder.scala:48)
	at org.apache.spark.internal.config.TypedConfigBuilder$$anonfun$toSequence$1.apply(ConfigBuilder.scala:124)
	at org.apache.spark.internal.config.TypedConfigBuilder$$anonfun$toSequence$1.apply(ConfigBuilder.scala:124)
	at org.apache.spark.internal.config.TypedConfigBuilder.createWithDefault(ConfigBuilder.scala:142)
	at org.apache.spark.internal.config.package$.(package.scala:152)
	at org.apache.spark.internal.config.package$.(package.scala)
	at org.apache.spark.SparkConf$.(SparkConf.scala:668)
	at org.apache.spark.SparkConf$.(SparkConf.scala)
	at org.apache.spark.SparkConf.set(SparkConf.scala:94)
	at org.apache.spark.SparkConf.set(SparkConf.scala:83)
	at org.apache.spark.SparkConf.setAppName(SparkConf.scala:120)
	at com.sunxj.sparktest.WordCount$.main(WordCount.scala:8)
	at com.sunxj.sparktest.WordCount.main(WordCount.scala)
Disconnected from the target VM, address: '127.0.0.1:50188', transport: 'socket'

Process finished with exit code 1

如下图所示：

25、出现此问题是由于pom.xml选的scala的版本，与IDEA选择的SDK版本不一致，首先pom.xml选择的是2.11.12，而在idea也要选择2.11.12，如下图所示：

26、然后再次执行debug，此时没有出现错误，如下代码所示：

/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:50386,suspend=y,server=n -javaagent:/Users/sunxiaoju/Library/Caches/IntelliJIdea2018.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/tools.jar:/sunxj/work/sparktest/target/classes:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-core_2.11/2.3.2/spark-core_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro/1.7.7/avro-1.7.7.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/Users/sunxiaoju/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill_2.11/0.8.4/chill_2.11-0.8.4.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/kryo-shaded/3.0.3/kryo-shaded-3.0.3.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/sunxiaoju/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill-java/0.8.4/chill-java-0.8.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/xbean/xbean-asm5-shaded/4.4/xbean-asm5-shaded-4.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-launcher_2.11/2.3.2/spark-launcher_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-kvstore_2.11/2.3.2/spark-kvstore_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-common_2.11/2.3.2/spark-network-common_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.3.2/spark-network-shuffle_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.3.2/spark-unsafe_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/net/java/dev/jets3t/jets3t/0.9.4/jets3t-0.9.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar:/Users/sunxiaoju/.m2/repository/commons-codec/commons-codec/1.11/commons-codec-1.11.jar:/Users/sunxiaoju/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/sunxiaoju/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.52/bcprov-jdk15on-1.52.jar:/Users/sunxiaoju/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/1.1/java-xmlbuilder-1.1.jar:/Users/sunxiaoju/.m2/repository/net/iharder/base64/2.3.8/base64-2.3.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/sunxiaoju/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/sunxiaoju/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/sunxiaoju/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/sunxiaoju/.m2/repository/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar:/Users/sunxiaoju/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/sunxiaoju/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/sunxiaoju/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/sunxiaoju/.m2/repository/commons-net/commons-net/2.2/commons-net-2.2.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-jackson_2.11/3.2.11/json4s-jackson_2.11-3.2.11.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-core_2.11/3.2.11/json4s-core_2.11-3.2.11.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scalap/2.11.0/scalap-2.11.0.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-compiler/2.11.0/scala-compiler-2.11.0.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.1/scala-parser-combinators_2.11-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/Users/sunxiaoju/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/sunxiaoju/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/sunxiaoju/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/sunxiaoju/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/sunxiaoju/.m2/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-tags_2.11/2.3.2/spark-tags_2.11-2.3.2.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-client/2.7.7/hadoop-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-common/2.7.7/hadoop-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/sunxiaoju/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/sunxiaoju/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/sunxiaoju/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/sunxiaoju/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/sunxiaoju/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/sunxiaoju/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/sunxiaoju/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/sunxiaoju/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/sunxiaoju/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/sunxiaoju/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.7/hadoop-auth-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/sunxiaoju/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/sunxiaoju/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.7/hadoop-mapreduce-client-app-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.7/hadoop-mapreduce-client-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.7.7/hadoop-yarn-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.7.7/hadoop-yarn-server-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.7/hadoop-mapreduce-client-shuffle-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.7.7/hadoop-yarn-api-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.7/hadoop-mapreduce-client-core-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.7/hadoop-yarn-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/sunxiaoju/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.7/hadoop-mapreduce-client-jobclient-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-annotations/2.7.7/hadoop-annotations-2.7.7.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/srcs/scala-reflect-2.11.12-sources.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/srcs/scala-library-2.11.12-sources.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" com.sunxj.sparktest.WordCount
Connected to the target VM, address: '127.0.0.1:50386', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/12 21:04:01 INFO SparkContext: Running Spark version 2.3.2
19/01/12 21:04:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/12 21:04:03 INFO SparkContext: Submitted application: Spark 学习
19/01/12 21:04:03 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/12 21:04:03 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/12 21:04:03 INFO SecurityManager: Changing view acls groups to: 
19/01/12 21:04:03 INFO SecurityManager: Changing modify acls groups to: 
19/01/12 21:04:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/12 21:04:04 INFO Utils: Successfully started service 'sparkDriver' on port 50391.
19/01/12 21:04:05 INFO SparkEnv: Registering MapOutputTracker
19/01/12 21:04:05 INFO SparkEnv: Registering BlockManagerMaster
19/01/12 21:04:05 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/12 21:04:05 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/12 21:04:05 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-48e50794-9702-4c62-96ac-2cbf1df3a320
19/01/12 21:04:05 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/12 21:04:05 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/12 21:04:06 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
19/01/12 21:04:06 INFO Utils: Successfully started service 'SparkUI' on port 4041.
19/01/12 21:04:06 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4041
19/01/12 21:04:07 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/12 21:04:07 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 104 ms (0 ms spent in bootstraps)
19/01/12 21:04:07 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190112210407-0000
19/01/12 21:04:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50394.
19/01/12 21:04:07 INFO NettyBlockTransferService: Server created on 192.168.0.104:50394
19/01/12 21:04:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/12 21:04:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/0 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/0 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:08 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 50394, None)
19/01/12 21:04:08 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:50394 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 50394, None)
19/01/12 21:04:08 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 50394, None)
19/01/12 21:04:08 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 50394, None)
19/01/12 21:04:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/0 is now RUNNING
19/01/12 21:04:09 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/12 21:04:11 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/12 21:04:11 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/12 21:04:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:50394 (size: 23.7 KB, free: 912.3 MB)
19/01/12 21:04:11 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:16
19/01/12 21:04:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/0 is now EXITED (Command exited with code 1)
19/01/12 21:04:12 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/0 removed: Command exited with code 1
19/01/12 21:04:12 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/1 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:12 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/1 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:12 INFO BlockManagerMaster: Removal of executor 0 requested
19/01/12 21:04:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/1 is now RUNNING
19/01/12 21:04:12 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 0
19/01/12 21:04:13 INFO BlockManagerMasterEndpoint: Trying to remove executor 0 from BlockManagerMaster.
19/01/12 21:04:15 INFO FileInputFormat: Total input paths to process : 1
19/01/12 21:04:15 INFO SparkContext: Starting job: collect at WordCount.scala:18
19/01/12 21:04:16 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:17)
19/01/12 21:04:16 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:18) with 2 output partitions
19/01/12 21:04:16 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:18)
19/01/12 21:04:16 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/12 21:04:16 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/12 21:04:16 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17), which has no missing parents
19/01/12 21:04:16 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 912.0 MB)
19/01/12 21:04:16 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/12 21:04:16 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:50394 (size: 2.9 KB, free: 912.3 MB)
19/01/12 21:04:16 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
19/01/12 21:04:16 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/12 21:04:16 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/12 21:04:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/1 is now EXITED (Command exited with code 1)
19/01/12 21:04:16 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/1 removed: Command exited with code 1
19/01/12 21:04:16 INFO BlockManagerMaster: Removal of executor 1 requested
19/01/12 21:04:16 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
19/01/12 21:04:16 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 1
19/01/12 21:04:16 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/2 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:16 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/2 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/2 is now RUNNING
19/01/12 21:04:19 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/2 is now EXITED (Command exited with code 1)
19/01/12 21:04:19 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/2 removed: Command exited with code 1
19/01/12 21:04:19 INFO BlockManagerMaster: Removal of executor 2 requested
19/01/12 21:04:19 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 2
19/01/12 21:04:19 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
19/01/12 21:04:19 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/3 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:19 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/3 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:19 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/3 is now RUNNING
19/01/12 21:04:22 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/3 is now EXITED (Command exited with code 1)
19/01/12 21:04:22 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/3 removed: Command exited with code 1
19/01/12 21:04:22 INFO BlockManagerMaster: Removal of executor 3 requested
19/01/12 21:04:22 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 3
19/01/12 21:04:22 INFO BlockManagerMasterEndpoint: Trying to remove executor 3 from BlockManagerMaster.
19/01/12 21:04:22 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/4 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:22 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/4 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:22 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/4 is now RUNNING
19/01/12 21:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/4 is now EXITED (Command exited with code 1)
19/01/12 21:04:26 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/4 removed: Command exited with code 1
19/01/12 21:04:26 INFO BlockManagerMaster: Removal of executor 4 requested
19/01/12 21:04:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 4
19/01/12 21:04:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
19/01/12 21:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/5 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/5 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/5 is now RUNNING
19/01/12 21:04:30 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/5 is now EXITED (Command exited with code 1)
19/01/12 21:04:30 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/5 removed: Command exited with code 1
19/01/12 21:04:30 INFO BlockManagerMaster: Removal of executor 5 requested
19/01/12 21:04:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 5
19/01/12 21:04:30 INFO BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster.
19/01/12 21:04:30 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/6 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:30 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/6 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:30 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/6 is now RUNNING
19/01/12 21:04:31 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 21:04:33 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/6 is now EXITED (Command exited with code 1)
19/01/12 21:04:33 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/6 removed: Command exited with code 1
19/01/12 21:04:33 INFO BlockManagerMaster: Removal of executor 6 requested
19/01/12 21:04:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 6
19/01/12 21:04:33 INFO BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster.
19/01/12 21:04:33 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/7 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:33 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/7 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:33 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/7 is now RUNNING
19/01/12 21:04:36 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/7 is now EXITED (Command exited with code 1)
19/01/12 21:04:36 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/7 removed: Command exited with code 1
19/01/12 21:04:36 INFO BlockManagerMaster: Removal of executor 7 requested
19/01/12 21:04:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 7
19/01/12 21:04:36 INFO BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster.
19/01/12 21:04:36 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/8 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:36 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/8 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:36 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/8 is now RUNNING
19/01/12 21:04:39 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/8 is now EXITED (Command exited with code 1)
19/01/12 21:04:39 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/8 removed: Command exited with code 1
19/01/12 21:04:39 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/9 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:39 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/9 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:39 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
19/01/12 21:04:39 INFO BlockManagerMaster: Removal of executor 8 requested
19/01/12 21:04:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 8
19/01/12 21:04:39 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/9 is now RUNNING
19/01/12 21:04:42 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/9 is now EXITED (Command exited with code 1)
19/01/12 21:04:42 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/9 removed: Command exited with code 1
19/01/12 21:04:42 INFO BlockManagerMaster: Removal of executor 9 requested
19/01/12 21:04:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 9
19/01/12 21:04:42 INFO BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster.
19/01/12 21:04:42 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/10 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:42 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/10 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:42 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/10 is now RUNNING
19/01/12 21:04:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/10 is now EXITED (Command exited with code 1)
19/01/12 21:04:45 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/10 removed: Command exited with code 1
19/01/12 21:04:45 INFO BlockManagerMaster: Removal of executor 10 requested
19/01/12 21:04:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 10
19/01/12 21:04:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster.
19/01/12 21:04:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/11 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/11 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/11 is now RUNNING
19/01/12 21:04:46 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 21:04:48 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/11 is now EXITED (Command exited with code 1)
19/01/12 21:04:48 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/11 removed: Command exited with code 1
19/01/12 21:04:48 INFO BlockManagerMasterEndpoint: Trying to remove executor 11 from BlockManagerMaster.
19/01/12 21:04:48 INFO BlockManagerMaster: Removal of executor 11 requested
19/01/12 21:04:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 11
19/01/12 21:04:48 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/12 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:48 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/12 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:48 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/12 is now RUNNING
19/01/12 21:04:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/12 is now EXITED (Command exited with code 1)
19/01/12 21:04:51 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/12 removed: Command exited with code 1
19/01/12 21:04:51 INFO BlockManagerMaster: Removal of executor 12 requested
19/01/12 21:04:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 12
19/01/12 21:04:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 12 from BlockManagerMaster.
19/01/12 21:04:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/13 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/13 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/13 is now RUNNING
19/01/12 21:04:53 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/13 is now EXITED (Command exited with code 1)
19/01/12 21:04:53 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/13 removed: Command exited with code 1
19/01/12 21:04:53 INFO BlockManagerMaster: Removal of executor 13 requested
19/01/12 21:04:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 13
19/01/12 21:04:53 INFO BlockManagerMasterEndpoint: Trying to remove executor 13 from BlockManagerMaster.
19/01/12 21:04:53 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/14 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:53 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/14 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:53 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/14 is now RUNNING
19/01/12 21:04:57 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/14 is now EXITED (Command exited with code 1)
19/01/12 21:04:57 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/14 removed: Command exited with code 1
19/01/12 21:04:57 INFO BlockManagerMaster: Removal of executor 14 requested
19/01/12 21:04:57 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 14
19/01/12 21:04:57 INFO BlockManagerMasterEndpoint: Trying to remove executor 14 from BlockManagerMaster.
19/01/12 21:04:57 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/15 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:04:57 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/15 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:04:57 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/15 is now RUNNING
19/01/12 21:05:01 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 21:05:02 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/15 is now EXITED (Command exited with code 1)
19/01/12 21:05:02 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/15 removed: Command exited with code 1
19/01/12 21:05:02 INFO BlockManagerMaster: Removal of executor 15 requested
19/01/12 21:05:02 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 15
19/01/12 21:05:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 15 from BlockManagerMaster.
19/01/12 21:05:02 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/16 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:02 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/16 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:02 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/16 is now RUNNING
19/01/12 21:05:06 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/16 is now EXITED (Command exited with code 1)
19/01/12 21:05:06 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/16 removed: Command exited with code 1
19/01/12 21:05:06 INFO BlockManagerMasterEndpoint: Trying to remove executor 16 from BlockManagerMaster.
19/01/12 21:05:06 INFO BlockManagerMaster: Removal of executor 16 requested
19/01/12 21:05:06 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 16
19/01/12 21:05:06 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/17 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:06 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/17 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:06 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/17 is now RUNNING
19/01/12 21:05:09 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/17 is now EXITED (Command exited with code 1)
19/01/12 21:05:09 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/17 removed: Command exited with code 1
19/01/12 21:05:09 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/18 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:09 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/18 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:09 INFO BlockManagerMaster: Removal of executor 17 requested
19/01/12 21:05:09 INFO BlockManagerMasterEndpoint: Trying to remove executor 17 from BlockManagerMaster.
19/01/12 21:05:09 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 17
19/01/12 21:05:09 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/18 is now RUNNING
19/01/12 21:05:13 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/18 is now EXITED (Command exited with code 1)
19/01/12 21:05:13 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/18 removed: Command exited with code 1
19/01/12 21:05:13 INFO BlockManagerMasterEndpoint: Trying to remove executor 18 from BlockManagerMaster.
19/01/12 21:05:13 INFO BlockManagerMaster: Removal of executor 18 requested
19/01/12 21:05:13 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 18
19/01/12 21:05:13 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/19 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:13 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/19 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:13 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/19 is now RUNNING
19/01/12 21:05:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/19 is now EXITED (Command exited with code 1)
19/01/12 21:05:16 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/19 removed: Command exited with code 1
19/01/12 21:05:16 INFO BlockManagerMaster: Removal of executor 19 requested
19/01/12 21:05:16 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 19
19/01/12 21:05:16 INFO BlockManagerMasterEndpoint: Trying to remove executor 19 from BlockManagerMaster.
19/01/12 21:05:16 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/20 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:16 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/20 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/20 is now RUNNING
19/01/12 21:05:16 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 21:05:20 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/20 is now EXITED (Command exited with code 1)
19/01/12 21:05:20 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/20 removed: Command exited with code 1
19/01/12 21:05:20 INFO BlockManagerMaster: Removal of executor 20 requested
19/01/12 21:05:20 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 20
19/01/12 21:05:20 INFO BlockManagerMasterEndpoint: Trying to remove executor 20 from BlockManagerMaster.
19/01/12 21:05:20 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/21 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:20 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/21 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:20 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/21 is now RUNNING
19/01/12 21:05:23 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/21 is now EXITED (Command exited with code 1)
19/01/12 21:05:23 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/21 removed: Command exited with code 1
19/01/12 21:05:23 INFO BlockManagerMaster: Removal of executor 21 requested
19/01/12 21:05:23 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 21
19/01/12 21:05:23 INFO BlockManagerMasterEndpoint: Trying to remove executor 21 from BlockManagerMaster.
19/01/12 21:05:24 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/22 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:24 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/22 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:24 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/22 is now RUNNING
19/01/12 21:05:27 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/22 is now EXITED (Command exited with code 1)
19/01/12 21:05:27 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/22 removed: Command exited with code 1
19/01/12 21:05:27 INFO BlockManagerMaster: Removal of executor 22 requested
19/01/12 21:05:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 22
19/01/12 21:05:27 INFO BlockManagerMasterEndpoint: Trying to remove executor 22 from BlockManagerMaster.
19/01/12 21:05:27 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/23 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:27 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/23 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:27 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/23 is now RUNNING
19/01/12 21:05:30 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/23 is now EXITED (Command exited with code 1)
19/01/12 21:05:30 INFO StandaloneSchedulerBackend: Executor app-20190112210407-0000/23 removed: Command exited with code 1
19/01/12 21:05:30 INFO BlockManagerMasterEndpoint: Trying to remove executor 23 from BlockManagerMaster.
19/01/12 21:05:30 INFO BlockManagerMaster: Removal of executor 23 requested
19/01/12 21:05:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 23
19/01/12 21:05:30 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112210407-0000/24 on worker-20190112203002-192.168.0.104-49749 (192.168.0.104:49749) with 1 core(s)
19/01/12 21:05:30 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112210407-0000/24 on hostPort 192.168.0.104:49749 with 1 core(s), 500.0 MB RAM
19/01/12 21:05:30 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112210407-0000/24 is now RUNNING
19/01/12 21:05:31 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Disconnected from the target VM, address: '127.0.0.1:50386', transport: 'socket'
19/01/12 21:05:33 INFO SparkContext: Invoking stop() from shutdown hook
19/01/12 21:05:33 INFO SparkUI: Stopped Spark web UI at http://192.168.0.104:4041
19/01/12 21:05:33 INFO DAGScheduler: Job 0 failed: collect at WordCount.scala:18, took 78.338755 s

27、此时会重复的执行，可以通过http://localhost:8080来查看对应的错误，首先打开网页，点击对应的Application ID，如下图所示：

28、此时会有很多执行任务，如下图所示：

29、然后点击stderr就会显示出错信息，如下错误提示：

Spark Executor Command: "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java" "-cp" "/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/conf/:/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/jars/*:/sunxj/InstallFile/hadoop-2.7.7/etc/hadoop/" "-Xmx500M" "-Dspark.driver.port=50391" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:50391" "--executor-id" "0" "--hostname" "192.168.0.104" "--cores" "1" "--app-id" "app-20190112210407-0000" "--worker-url" "spark://[email protected]:49749"
========================================

Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
	at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:200)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:221)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

如下图所示：

30、出现此错误是需要设置应用程序的 VM options或者在代码中设置，如下图所示：

31、再次启动会卡住，如下信息：

Connected to the target VM, address: '127.0.0.1:53635', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/12 22:42:03 INFO SparkContext: Running Spark version 2.3.2
19/01/12 22:42:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/12 22:42:04 INFO SparkContext: Submitted application: Spark 学习
19/01/12 22:42:04 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/12 22:42:04 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/12 22:42:04 INFO SecurityManager: Changing view acls groups to: 
19/01/12 22:42:04 INFO SecurityManager: Changing modify acls groups to: 
19/01/12 22:42:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/12 22:42:05 INFO Utils: Successfully started service 'sparkDriver' on port 53641.
19/01/12 22:42:05 INFO SparkEnv: Registering MapOutputTracker
19/01/12 22:42:05 INFO SparkEnv: Registering BlockManagerMaster
19/01/12 22:42:05 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/12 22:42:05 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/12 22:42:05 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-01859ab4-0601-4945-8e28-d71dd73bffdf
19/01/12 22:42:05 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/12 22:42:05 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/12 22:42:05 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/12 22:42:05 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/12 22:42:05 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/12 22:42:06 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 52 ms (0 ms spent in bootstraps)
19/01/12 22:42:06 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190112224206-0000
19/01/12 22:42:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53643.
19/01/12 22:42:06 INFO NettyBlockTransferService: Server created on 192.168.0.104:53643
19/01/12 22:42:06 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/12 22:42:06 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 53643, None)
19/01/12 22:42:06 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:53643 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 53643, None)
19/01/12 22:42:06 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 53643, None)
19/01/12 22:42:06 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 53643, None)
19/01/12 22:42:06 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/12 22:42:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/12 22:42:08 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/12 22:42:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:53643 (size: 23.7 KB, free: 912.3 MB)
19/01/12 22:42:08 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:17
19/01/12 22:42:09 INFO FileInputFormat: Total input paths to process : 1
19/01/12 22:42:10 INFO SparkContext: Starting job: collect at WordCount.scala:21
19/01/12 22:42:10 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:20)
19/01/12 22:42:10 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:21) with 2 output partitions
19/01/12 22:42:10 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:21)
19/01/12 22:42:10 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/12 22:42:10 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/12 22:42:10 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20), which has no missing parents
19/01/12 22:42:10 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 912.0 MB)
19/01/12 22:42:10 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/12 22:42:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:53643 (size: 2.9 KB, free: 912.3 MB)
19/01/12 22:42:10 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
19/01/12 22:42:10 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20) (first 15 tasks are for partitions Vector(0, 1))
19/01/12 22:42:10 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/12 22:42:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:42:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:42:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:43:10 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:43:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:43:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:43:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:44:10 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:44:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:44:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:44:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:45:10 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:45:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:45:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:45:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:46:10 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:46:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:46:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/01/12 22:46:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

而web界面则显示等待状态，如下图所示：

32、那么需要将spark-env.sh文件中的设置的大一下，将500m设置成2046m，如下图所示：

33、然后重启spark，再次执行,而此时会卡住，如下信息所示：

Connected to the target VM, address: '127.0.0.1:53794', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/12 22:51:56 INFO SparkContext: Running Spark version 2.3.2
19/01/12 22:51:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/12 22:51:57 INFO SparkContext: Submitted application: Spark 学习
19/01/12 22:51:57 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/12 22:51:57 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/12 22:51:57 INFO SecurityManager: Changing view acls groups to: 
19/01/12 22:51:57 INFO SecurityManager: Changing modify acls groups to: 
19/01/12 22:51:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/12 22:51:58 INFO Utils: Successfully started service 'sparkDriver' on port 53799.
19/01/12 22:51:58 INFO SparkEnv: Registering MapOutputTracker
19/01/12 22:51:58 INFO SparkEnv: Registering BlockManagerMaster
19/01/12 22:51:58 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/12 22:51:58 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/12 22:51:58 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-01a7fbd5-c0da-4103-9358-6e5a2d0c6802
19/01/12 22:51:58 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/12 22:51:58 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/12 22:51:58 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/12 22:51:58 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/12 22:51:59 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/12 22:51:59 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 46 ms (0 ms spent in bootstraps)
19/01/12 22:51:59 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190112225159-0000
19/01/12 22:51:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53801.
19/01/12 22:51:59 INFO NettyBlockTransferService: Server created on 192.168.0.104:53801
19/01/12 22:51:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/12 22:51:59 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112225159-0000/0 on worker-20190112225146-192.168.0.104-53789 (192.168.0.104:53789) with 1 core(s)
19/01/12 22:51:59 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112225159-0000/0 on hostPort 192.168.0.104:53789 with 1 core(s), 512.0 MB RAM
19/01/12 22:51:59 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 53801, None)
19/01/12 22:51:59 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:53801 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 53801, None)
19/01/12 22:51:59 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112225159-0000/0 is now RUNNING
19/01/12 22:51:59 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 53801, None)
19/01/12 22:51:59 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 53801, None)
19/01/12 22:52:00 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/12 22:52:02 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/12 22:52:03 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/12 22:52:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:53801 (size: 23.7 KB, free: 912.3 MB)
19/01/12 22:52:03 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:17
19/01/12 22:52:04 INFO FileInputFormat: Total input paths to process : 1
19/01/12 22:52:04 INFO SparkContext: Starting job: collect at WordCount.scala:21
19/01/12 22:52:05 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:20)
19/01/12 22:52:05 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:21) with 2 output partitions
19/01/12 22:52:05 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:21)
19/01/12 22:52:05 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/12 22:52:05 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/12 22:52:05 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20), which has no missing parents
19/01/12 22:52:05 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 912.0 MB)
19/01/12 22:52:05 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/12 22:52:05 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:53801 (size: 2.9 KB, free: 912.3 MB)
19/01/12 22:52:05 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
19/01/12 22:52:05 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20) (first 15 tasks are for partitions Vector(0, 1))
19/01/12 22:52:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/12 22:52:05 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.104:53813) with ID 0
19/01/12 22:52:06 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7886 bytes)
19/01/12 22:52:06 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:53815 with 93.3 MB RAM, BlockManagerId(0, 192.168.0.104, 53815, None)

34、再次查看web界面发现是一致处于RUNNING状态，如下图所示：

35、点进去然后查看stdout显示如下信息:

2019-01-12 22:52:02 INFO  CoarseGrainedExecutorBackend:2566 - Started daemon with process name: [email protected]
2019-01-12 22:52:02 INFO  SignalUtils:54 - Registered signal handler for TERM
2019-01-12 22:52:02 INFO  SignalUtils:54 - Registered signal handler for HUP
2019-01-12 22:52:02 INFO  SignalUtils:54 - Registered signal handler for INT
2019-01-12 22:52:03 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-12 22:52:04 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-12 22:52:04 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-12 22:52:04 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-12 22:52:04 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-12 22:52:04 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-12 22:52:04 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:53799 after 182 ms (0 ms spent in bootstraps)
2019-01-12 22:52:05 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-12 22:52:05 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-12 22:52:05 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-12 22:52:05 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-12 22:52:05 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-12 22:52:05 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:53799 after 2 ms (0 ms spent in bootstraps)
2019-01-12 22:52:05 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-a5a62153-d136-4feb-b6ad-0f7671f58e2e/executor-2aac416a-c07d-4bb9-84a3-6c7c85d6fcb5/blockmgr-66574664-328e-4bff-9600-68de97420077
2019-01-12 22:52:05 INFO  MemoryStore:54 - MemoryStore started with capacity 93.3 MB
2019-01-12 22:52:05 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://[email protected]:53799
2019-01-12 22:52:05 INFO  WorkerWatcher:54 - Connecting to worker spark://[email protected]:53789
2019-01-12 22:52:06 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:53789 after 60 ms (0 ms spent in bootstraps)
2019-01-12 22:52:06 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2019-01-12 22:52:06 INFO  WorkerWatcher:54 - Successfully connected to spark://[email protected]:53789
2019-01-12 22:52:06 INFO  Executor:54 - Starting executor ID 0 on host 192.168.0.104
2019-01-12 22:52:06 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53815.
2019-01-12 22:52:06 INFO  NettyBlockTransferService:54 - Server created on 192.168.0.104:53815
2019-01-12 22:52:06 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-01-12 22:52:06 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(0, 192.168.0.104, 53815, None)
2019-01-12 22:52:06 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(0, 192.168.0.104, 53815, None)
2019-01-12 22:52:06 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(0, 192.168.0.104, 53815, None)
2019-01-12 22:52:06 ERROR Inbox:91 - Ignoring error
java.io.EOFException
	at java.io.DataInputStream.readFully(DataInputStream.java:197)
	at java.io.DataInputStream.readUTF(DataInputStream.java:609)
	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
	at org.apache.spark.scheduler.TaskDescription$$anonfun$decode$1.apply(TaskDescription.scala:134)
	at org.apache.spark.scheduler.TaskDescription$$anonfun$decode$1.apply(TaskDescription.scala:133)
	at scala.collection.immutable.Range.foreach(Range.scala:160)
	at org.apache.spark.scheduler.TaskDescription$.decode(TaskDescription.scala:133)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:96)
	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

如下图所示：

36、这是由于项目中引用的spark的jar和集群中的jar不通，我们的spark使用的是scala的版本为2.11.12,spark的版本是2.4.0，因此需要选择的是2.11.12版本的2.4.0，在maven仓库中选择2.4.0的2.11,如下图所示：

37、所以将pom.xml中的版本更换一下，如下所示：



    4.0.0

    spark
    spark-test
    1.0-SNAPSHOT

    
        
            org.apache.spark
            spark-core_2.11
            2.4.0
        
        
            org.apache.hadoop
            hadoop-client
            2.7.7

38、等下载好依赖后重新执行，此时出现如下错误：

/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:54379,suspend=y,server=n -javaagent:/Users/sunxiaoju/Library/Caches/IntelliJIdea2018.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/tools.jar:/sunxj/work/sparktest/target/classes:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/srcs/scala-reflect-2.11.12-sources.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/srcs/scala-library-2.11.12-sources.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-core_2.11/2.4.0/spark-core_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/Users/sunxiaoju/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/Users/sunxiaoju/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/sunxiaoju/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-launcher_2.11/2.4.0/spark-launcher_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-kvstore_2.11/2.4.0/spark-kvstore_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-common_2.11/2.4.0/spark-network-common_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.4.0/spark-network-shuffle_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.4.0/spark-unsafe_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/Users/sunxiaoju/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/sunxiaoju/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/sunxiaoju/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/sunxiaoju/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/Users/sunxiaoju/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/sunxiaoju/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/sunxiaoju/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/sunxiaoju/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-jackson_2.11/3.5.3/json4s-jackson_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-core_2.11/3.5.3/json4s-core_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-ast_2.11/3.5.3/json4s-ast_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-scalap_2.11/3.5.3/json4s-scalap_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/Users/sunxiaoju/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/sunxiaoju/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/sunxiaoju/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/sunxiaoju/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/sunxiaoju/.m2/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-tags_2.11/2.4.0/spark-tags_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-client/2.7.7/hadoop-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-common/2.7.7/hadoop-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/sunxiaoju/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/sunxiaoju/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/sunxiaoju/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/sunxiaoju/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/sunxiaoju/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/sunxiaoju/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/sunxiaoju/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/sunxiaoju/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/sunxiaoju/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/sunxiaoju/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.7/hadoop-auth-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/sunxiaoju/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/sunxiaoju/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.7/hadoop-mapreduce-client-app-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.7/hadoop-mapreduce-client-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.7.7/hadoop-yarn-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.7.7/hadoop-yarn-server-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.7/hadoop-mapreduce-client-shuffle-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.7.7/hadoop-yarn-api-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.7/hadoop-mapreduce-client-core-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.7/hadoop-yarn-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/sunxiaoju/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.7/hadoop-mapreduce-client-jobclient-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-annotations/2.7.7/hadoop-annotations-2.7.7.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" com.sunxj.sparktest.WordCount
Connected to the target VM, address: '127.0.0.1:54379', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/12 23:15:01 INFO SparkContext: Running Spark version 2.4.0
19/01/12 23:15:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/12 23:15:02 INFO SparkContext: Submitted application: Spark 学习
19/01/12 23:15:03 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/12 23:15:03 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/12 23:15:03 INFO SecurityManager: Changing view acls groups to: 
19/01/12 23:15:03 INFO SecurityManager: Changing modify acls groups to: 
19/01/12 23:15:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/12 23:15:03 INFO Utils: Successfully started service 'sparkDriver' on port 54382.
19/01/12 23:15:04 INFO SparkEnv: Registering MapOutputTracker
19/01/12 23:15:04 INFO SparkEnv: Registering BlockManagerMaster
19/01/12 23:15:04 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/12 23:15:04 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/12 23:15:04 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-3b98740f-c5dc-43b8-b8fd-acfb4be4bb5d
19/01/12 23:15:04 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/12 23:15:04 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/12 23:15:04 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/12 23:15:04 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/12 23:15:04 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/12 23:15:04 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 67 ms (0 ms spent in bootstraps)
19/01/12 23:15:05 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190112231505-0001
19/01/12 23:15:05 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190112231505-0001/0 on worker-20190112225146-192.168.0.104-53789 (192.168.0.104:53789) with 1 core(s)
19/01/12 23:15:05 INFO StandaloneSchedulerBackend: Granted executor ID app-20190112231505-0001/0 on hostPort 192.168.0.104:53789 with 1 core(s), 512.0 MB RAM
19/01/12 23:15:05 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54384.
19/01/12 23:15:05 INFO NettyBlockTransferService: Server created on 192.168.0.104:54384
19/01/12 23:15:05 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/12 23:15:05 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190112231505-0001/0 is now RUNNING
19/01/12 23:15:05 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 54384, None)
19/01/12 23:15:05 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:54384 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 54384, None)
19/01/12 23:15:05 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 54384, None)
19/01/12 23:15:05 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 54384, None)
19/01/12 23:15:05 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/12 23:15:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/12 23:15:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/12 23:15:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:54384 (size: 23.7 KB, free: 912.3 MB)
19/01/12 23:15:07 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:17
19/01/12 23:15:09 INFO FileInputFormat: Total input paths to process : 1
19/01/12 23:15:09 INFO SparkContext: Starting job: collect at WordCount.scala:21
19/01/12 23:15:10 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.104:54388) with ID 0
19/01/12 23:15:10 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:20)
19/01/12 23:15:10 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:21) with 2 output partitions
19/01/12 23:15:10 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:21)
19/01/12 23:15:10 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/12 23:15:10 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/12 23:15:10 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20), which has no missing parents
19/01/12 23:15:10 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 912.0 MB)
19/01/12 23:15:10 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/12 23:15:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:54384 (size: 2.9 KB, free: 912.3 MB)
19/01/12 23:15:10 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
19/01/12 23:15:10 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:20) (first 15 tasks are for partitions Vector(0, 1))
19/01/12 23:15:10 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/12 23:15:10 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:54390 with 93.3 MB RAM, BlockManagerId(0, 192.168.0.104, 54390, None)
19/01/12 23:15:11 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:11 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:54390 (size: 2.9 KB, free: 93.3 MB)
19/01/12 23:15:12 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, 192.168.0.104, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

19/01/12 23:15:12 INFO TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 1]
19/01/12 23:15:12 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 2, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 3, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID 2) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 2]
19/01/12 23:15:12 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 3]
19/01/12 23:15:12 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 4]
19/01/12 23:15:12 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 6, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 5]
19/01/12 23:15:12 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID 7, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/12 23:15:12 INFO TaskSetManager: Lost task 0.3 in stage 0.0 (TID 6) on 192.168.0.104, executor 0: java.lang.ClassCastException (cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 6]
19/01/12 23:15:12 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
19/01/12 23:15:12 INFO TaskSchedulerImpl: Cancelling stage 0
19/01/12 23:15:12 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
19/01/12 23:15:12 INFO TaskSchedulerImpl: Stage 0 was cancelled
19/01/12 23:15:12 INFO DAGScheduler: ShuffleMapStage 0 (map at WordCount.scala:20) failed in 1.827 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, 192.168.0.104, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
19/01/12 23:15:12 INFO DAGScheduler: Job 0 failed: collect at WordCount.scala:21, took 2.828873 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, 192.168.0.104, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
	at com.sunxj.sparktest.WordCount$.main(WordCount.scala:21)
	at com.sunxj.sparktest.WordCount.main(WordCount.scala)
Caused by: java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/01/12 23:15:12 INFO SparkContext: Invoking stop() from shutdown hook
19/01/12 23:15:12 INFO SparkUI: Stopped Spark web UI at http://192.168.0.104:4040
19/01/12 23:15:12 INFO StandaloneSchedulerBackend: Shutting down all executors
19/01/12 23:15:12 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/01/12 23:15:12 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:160)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:140)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:655)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:275)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)
19/01/12 23:15:12 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/01/12 23:15:12 INFO MemoryStore: MemoryStore cleared
19/01/12 23:15:12 INFO BlockManager: BlockManager stopped
19/01/12 23:15:12 INFO BlockManagerMaster: BlockManagerMaster stopped
19/01/12 23:15:12 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/01/12 23:15:12 INFO SparkContext: Successfully stopped SparkContext
19/01/12 23:15:12 INFO ShutdownHookManager: Shutdown hook called
19/01/12 23:15:12 INFO ShutdownHookManager: Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-6c739ca4-8a49-4f30-8e45-e13fd8236513
Disconnected from the target VM, address: '127.0.0.1:54379', transport: 'socket'

Process finished with exit code 1

web界面的stdout显示如下错误：

2019-01-12 23:15:07 INFO  CoarseGrainedExecutorBackend:2566 - Started daemon with process name: [email protected]
2019-01-12 23:15:07 INFO  SignalUtils:54 - Registered signal handler for TERM
2019-01-12 23:15:07 INFO  SignalUtils:54 - Registered signal handler for HUP
2019-01-12 23:15:07 INFO  SignalUtils:54 - Registered signal handler for INT
2019-01-12 23:15:08 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-12 23:15:08 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-12 23:15:08 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-12 23:15:08 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-12 23:15:08 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-12 23:15:08 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-12 23:15:09 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:54382 after 185 ms (0 ms spent in bootstraps)
2019-01-12 23:15:09 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-12 23:15:09 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-12 23:15:09 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-12 23:15:09 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-12 23:15:09 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-12 23:15:09 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:54382 after 2 ms (0 ms spent in bootstraps)
2019-01-12 23:15:10 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-a5a62153-d136-4feb-b6ad-0f7671f58e2e/executor-402760db-d784-40a1-acd5-5b30d25e7aff/blockmgr-2bd08250-c6ba-40a1-b378-8005e79f250d
2019-01-12 23:15:10 INFO  MemoryStore:54 - MemoryStore started with capacity 93.3 MB
2019-01-12 23:15:10 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://[email protected]:54382
2019-01-12 23:15:10 INFO  WorkerWatcher:54 - Connecting to worker spark://[email protected]:53789
2019-01-12 23:15:10 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:53789 after 18 ms (0 ms spent in bootstraps)
2019-01-12 23:15:10 INFO  WorkerWatcher:54 - Successfully connected to spark://[email protected]:53789
2019-01-12 23:15:10 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2019-01-12 23:15:10 INFO  Executor:54 - Starting executor ID 0 on host 192.168.0.104
2019-01-12 23:15:10 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54390.
2019-01-12 23:15:10 INFO  NettyBlockTransferService:54 - Server created on 192.168.0.104:54390
2019-01-12 23:15:10 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-01-12 23:15:10 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(0, 192.168.0.104, 54390, None)
2019-01-12 23:15:10 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(0, 192.168.0.104, 54390, None)
2019-01-12 23:15:10 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(0, 192.168.0.104, 54390, None)
2019-01-12 23:15:11 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 0
2019-01-12 23:15:11 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2019-01-12 23:15:11 INFO  TorrentBroadcast:54 - Started reading broadcast variable 1
2019-01-12 23:15:11 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:54384 after 3 ms (0 ms spent in bootstraps)
2019-01-12 23:15:11 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 93.3 MB)
2019-01-12 23:15:11 INFO  TorrentBroadcast:54 - Reading broadcast variable 1 took 585 ms
2019-01-12 23:15:12 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 93.3 MB)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 1
2019-01-12 23:15:12 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 2
2019-01-12 23:15:12 INFO  Executor:54 - Running task 0.1 in stage 0.0 (TID 2)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 0.1 in stage 0.0 (TID 2)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 3
2019-01-12 23:15:12 INFO  Executor:54 - Running task 1.1 in stage 0.0 (TID 3)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 1.1 in stage 0.0 (TID 3)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 4
2019-01-12 23:15:12 INFO  Executor:54 - Running task 0.2 in stage 0.0 (TID 4)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 0.2 in stage 0.0 (TID 4)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 5
2019-01-12 23:15:12 INFO  Executor:54 - Running task 1.2 in stage 0.0 (TID 5)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 1.2 in stage 0.0 (TID 5)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 6
2019-01-12 23:15:12 INFO  Executor:54 - Running task 0.3 in stage 0.0 (TID 6)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 0.3 in stage 0.0 (TID 6)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-01-12 23:15:12 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 7
2019-01-12 23:15:12 INFO  Executor:54 - Running task 1.3 in stage 0.0 (TID 7)
2019-01-12 23:15:12 ERROR Executor:91 - Exception in task 1.3 in stage 0.0 (TID 7)
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:88)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

如下图所示：

39、此时需要先打包成一个jar，首先使用IDEA打包jar，选择File->Project Structure如下图所示：

40、然后选择Artifacts，选择+号，如下图所示：

41、然后选择JAR->From modules with dependencies...，如下图所示：

42、然后选择Main Class，选择copy to the output directory and link via manifest（此项不是将所有依赖包打包成一个jar包的），然后将MANIFEST.MF的目录选在根目录，如下图所示：

43、然后选择Build->Build Artifacts...如下图所示：

44、进行编译，此时会在根目录中生成一个out目录，如下图所示：

45、然后直接使用如下命令执行：

spark-submit out/artifacts/sparktest_jar/sparktest.jar

如下图所示：

打印信息如下，此时没有错误：

2019-01-13 21:26:33 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-13 21:26:33 INFO  SparkContext:54 - Running Spark version 2.4.0
2019-01-13 21:26:33 INFO  SparkContext:54 - Submitted application: Spark 学习
2019-01-13 21:26:33 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-13 21:26:33 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-13 21:26:33 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-13 21:26:33 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-13 21:26:33 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-13 21:26:34 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 50430.
2019-01-13 21:26:34 INFO  SparkEnv:54 - Registering MapOutputTracker
2019-01-13 21:26:34 INFO  SparkEnv:54 - Registering BlockManagerMaster
2019-01-13 21:26:34 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-01-13 21:26:34 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2019-01-13 21:26:34 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-38b355e2-2a7a-4d6a-b4ae-3b691277a4bb
2019-01-13 21:26:34 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2019-01-13 21:26:34 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2019-01-13 21:26:34 INFO  log:192 - Logging initialized @3102ms
2019-01-13 21:26:34 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-01-13 21:26:34 INFO  Server:419 - Started @3241ms
2019-01-13 21:26:34 INFO  AbstractConnector:278 - Started ServerConnector@3b4ef7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-01-13 21:26:34 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@251ebf23{/jobs,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3773862a{/jobs/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2472c7d8{/jobs/job,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22175d4f{/jobs/job/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9fecdf1{/stages,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3b809711{/stages/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3b0f7d9d{/stages/stage,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@63034ed1{/stages/stage/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@232024b9{/stages/pool,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@55a8dc49{/stages/pool/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a415aa9{/storage,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@53cdecf6{/storage/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@71ea1fda{/storage/rdd,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@62b3df3a{/storage/rdd/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@420745d7{/environment,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7e11ab3d{/environment/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5fa47fea{/executors,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2392212b{/executors/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5b43e173{/executors/threadDump,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@28f8e165{/executors/threadDump/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@545f80bf{/static,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e406694{/,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ab9b447{/api,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15b986cd{/jobs/job/kill,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6bb7cce7{/stages/stage/kill,null,AVAILABLE,@Spark}
2019-01-13 21:26:35 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
2019-01-13 21:26:35 INFO  SparkContext:54 - Added JAR file:/sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar at spark://192.168.0.104:50430/jars/sparktest.jar with timestamp 1547385995160
2019-01-13 21:26:35 INFO  StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://sunxj-mac-air.local:7077...
2019-01-13 21:26:35 INFO  TransportClientFactory:267 - Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 62 ms (0 ms spent in bootstraps)
2019-01-13 21:26:35 INFO  StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20190113212635-0005
2019-01-13 21:26:35 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20190113212635-0005/0 on worker-20190113204613-192.168.0.104-49258 (192.168.0.104:49258) with 1 core(s)
2019-01-13 21:26:35 INFO  StandaloneSchedulerBackend:54 - Granted executor ID app-20190113212635-0005/0 on hostPort 192.168.0.104:49258 with 1 core(s), 512.0 MB RAM
2019-01-13 21:26:35 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50432.
2019-01-13 21:26:35 INFO  NettyBlockTransferService:54 - Server created on 192.168.0.104:50432
2019-01-13 21:26:35 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-01-13 21:26:35 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20190113212635-0005/0 is now RUNNING
2019-01-13 21:26:35 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.0.104, 50432, None)
2019-01-13 21:26:35 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.0.104:50432 with 366.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 50432, None)
2019-01-13 21:26:35 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.0.104, 50432, None)
2019-01-13 21:26:35 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 50432, None)
2019-01-13 21:26:36 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@529cfee5{/metrics/json,null,AVAILABLE,@Spark}
2019-01-13 21:26:36 INFO  StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
2019-01-13 21:26:37 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 242.5 KB, free 366.1 MB)
2019-01-13 21:26:38 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.4 KB, free 366.0 MB)
2019-01-13 21:26:38 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.0.104:50432 (size: 23.4 KB, free: 366.3 MB)
2019-01-13 21:26:38 INFO  SparkContext:54 - Created broadcast 0 from textFile at WordCount.scala:16
2019-01-13 21:26:39 INFO  FileInputFormat:249 - Total input paths to process : 1
2019-01-13 21:26:39 INFO  SparkContext:54 - Starting job: collect at WordCount.scala:18
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Registering RDD 3 (map at WordCount.scala:17)
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Got job 0 (collect at WordCount.scala:18) with 2 output partitions
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Final stage: ResultStage 1 (collect at WordCount.scala:18)
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Parents of final stage: List(ShuffleMapStage 0)
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Missing parents: List(ShuffleMapStage 0)
2019-01-13 21:26:39 INFO  DAGScheduler:54 - Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17), which has no missing parents
2019-01-13 21:26:39 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 366.0 MB)
2019-01-13 21:26:39 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 366.0 MB)
2019-01-13 21:26:40 INFO  BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on 192.168.0.104:50432 (size: 2.9 KB, free: 366.3 MB)
2019-01-13 21:26:40 INFO  SparkContext:54 - Created broadcast 1 from broadcast at DAGScheduler.scala:1161
2019-01-13 21:26:40 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
2019-01-13 21:26:40 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2019-01-13 21:26:40 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.104:50435) with ID 0
2019-01-13 21:26:40 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
2019-01-13 21:26:40 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.0.104:50437 with 93.3 MB RAM, BlockManagerId(0, 192.168.0.104, 50437, None)
2019-01-13 21:26:41 INFO  BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on 192.168.0.104:50437 (size: 2.9 KB, free: 93.3 MB)
2019-01-13 21:26:41 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.0.104:50437 (size: 23.4 KB, free: 93.3 MB)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 2903 ms on 192.168.0.104 (executor 0) (1/2)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 129 ms on 192.168.0.104 (executor 0) (2/2)
2019-01-13 21:26:43 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool 
2019-01-13 21:26:43 INFO  DAGScheduler:54 - ShuffleMapStage 0 (map at WordCount.scala:17) finished in 3.406 s
2019-01-13 21:26:43 INFO  DAGScheduler:54 - looking for newly runnable stages
2019-01-13 21:26:43 INFO  DAGScheduler:54 - running: Set()
2019-01-13 21:26:43 INFO  DAGScheduler:54 - waiting: Set(ResultStage 1)
2019-01-13 21:26:43 INFO  DAGScheduler:54 - failed: Set()
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 3.2 KB, free 366.0 MB)
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 2027.0 B, free 366.0 MB)
2019-01-13 21:26:43 INFO  BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on 192.168.0.104:50432 (size: 2027.0 B, free: 366.3 MB)
2019-01-13 21:26:43 INFO  SparkContext:54 - Created broadcast 2 from broadcast at DAGScheduler.scala:1161
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
2019-01-13 21:26:43 INFO  TaskSchedulerImpl:54 - Adding task set 1.0 with 2 tasks
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Starting task 0.0 in stage 1.0 (TID 2, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
2019-01-13 21:26:43 INFO  BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on 192.168.0.104:50437 (size: 2027.0 B, free: 93.3 MB)
2019-01-13 21:26:43 INFO  MapOutputTrackerMasterEndpoint:54 - Asked to send map output locations for shuffle 0 to 192.168.0.104:50435
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Starting task 1.0 in stage 1.0 (TID 3, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 0.0 in stage 1.0 (TID 2) in 186 ms on 192.168.0.104 (executor 0) (1/2)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 1.0 in stage 1.0 (TID 3) in 36 ms on 192.168.0.104 (executor 0) (2/2)
2019-01-13 21:26:43 INFO  TaskSchedulerImpl:54 - Removed TaskSet 1.0, whose tasks have all completed, from pool 
2019-01-13 21:26:43 INFO  DAGScheduler:54 - ResultStage 1 (collect at WordCount.scala:18) finished in 0.276 s
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Job 0 finished: collect at WordCount.scala:18, took 4.272676 s
2019-01-13 21:26:43 INFO  SparkContext:54 - Starting job: foreach at WordCount.scala:19
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Got job 1 (foreach at WordCount.scala:19) with 2 output partitions
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Final stage: ResultStage 3 (foreach at WordCount.scala:19)
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Parents of final stage: List(ShuffleMapStage 2)
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Missing parents: List()
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Submitting ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 3.2 KB, free 366.0 MB)
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 2010.0 B, free 366.0 MB)
2019-01-13 21:26:43 INFO  BlockManagerInfo:54 - Added broadcast_3_piece0 in memory on 192.168.0.104:50432 (size: 2010.0 B, free: 366.3 MB)
2019-01-13 21:26:43 INFO  SparkContext:54 - Created broadcast 3 from broadcast at DAGScheduler.scala:1161
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
2019-01-13 21:26:43 INFO  TaskSchedulerImpl:54 - Adding task set 3.0 with 2 tasks
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Starting task 0.0 in stage 3.0 (TID 4, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
2019-01-13 21:26:43 INFO  BlockManagerInfo:54 - Added broadcast_3_piece0 in memory on 192.168.0.104:50437 (size: 2010.0 B, free: 93.3 MB)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Starting task 1.0 in stage 3.0 (TID 5, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 0.0 in stage 3.0 (TID 4) in 78 ms on 192.168.0.104 (executor 0) (1/2)
2019-01-13 21:26:43 INFO  TaskSetManager:54 - Finished task 1.0 in stage 3.0 (TID 5) in 37 ms on 192.168.0.104 (executor 0) (2/2)
2019-01-13 21:26:43 INFO  TaskSchedulerImpl:54 - Removed TaskSet 3.0, whose tasks have all completed, from pool 
2019-01-13 21:26:43 INFO  DAGScheduler:54 - ResultStage 3 (foreach at WordCount.scala:19) finished in 0.127 s
2019-01-13 21:26:43 INFO  DAGScheduler:54 - Job 1 finished: foreach at WordCount.scala:19, took 0.137270 s
2019-01-13 21:26:43 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2019-01-13 21:26:43 INFO  AbstractConnector:318 - Stopped Spark@3b4ef7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-01-13 21:26:43 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.0.104:4040
2019-01-13 21:26:43 INFO  StandaloneSchedulerBackend:54 - Shutting down all executors
2019-01-13 21:26:43 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down
2019-01-13 21:26:43 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2019-01-13 21:26:43 INFO  MemoryStore:54 - MemoryStore cleared
2019-01-13 21:26:43 INFO  BlockManager:54 - BlockManager stopped
2019-01-13 21:26:43 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2019-01-13 21:26:43 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2019-01-13 21:26:43 INFO  SparkContext:54 - Successfully stopped SparkContext
2019-01-13 21:26:43 INFO  ShutdownHookManager:54 - Shutdown hook called
2019-01-13 21:26:43 INFO  ShutdownHookManager:54 - Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-b60acfc8-02f8-422a-a062-87898e3df769
2019-01-13 21:26:43 INFO  ShutdownHookManager:54 - Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-9b46be2d-e1cc-4443-a8d7-5d3ca85df1fe

46、然后进入到web页面看一下打印日志：

2019-01-13 21:26:37 INFO  CoarseGrainedExecutorBackend:2566 - Started daemon with process name: [email protected]
2019-01-13 21:26:37 INFO  SignalUtils:54 - Registered signal handler for TERM
2019-01-13 21:26:37 INFO  SignalUtils:54 - Registered signal handler for HUP
2019-01-13 21:26:37 INFO  SignalUtils:54 - Registered signal handler for INT
2019-01-13 21:26:38 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-13 21:26:38 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-13 21:26:38 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-13 21:26:38 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-13 21:26:38 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-13 21:26:38 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-13 21:26:39 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:50430 after 173 ms (0 ms spent in bootstraps)
2019-01-13 21:26:39 INFO  SecurityManager:54 - Changing view acls to: sunxiaoju
2019-01-13 21:26:39 INFO  SecurityManager:54 - Changing modify acls to: sunxiaoju
2019-01-13 21:26:39 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-01-13 21:26:39 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-01-13 21:26:39 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
2019-01-13 21:26:39 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:50430 after 1 ms (0 ms spent in bootstraps)
2019-01-13 21:26:39 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-1ee810c4-971f-43df-81b1-80a52c2bd18e/executor-c927c25f-e5e6-40c9-a3e8-75dd2b465a08/blockmgr-93424b8a-f753-4ef4-a7ca-019159ff24bd
2019-01-13 21:26:39 INFO  MemoryStore:54 - MemoryStore started with capacity 93.3 MB
2019-01-13 21:26:40 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://[email protected]:50430
2019-01-13 21:26:40 INFO  WorkerWatcher:54 - Connecting to worker spark://[email protected]:49258
2019-01-13 21:26:40 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:49258 after 18 ms (0 ms spent in bootstraps)
2019-01-13 21:26:40 INFO  WorkerWatcher:54 - Successfully connected to spark://[email protected]:49258
2019-01-13 21:26:40 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2019-01-13 21:26:40 INFO  Executor:54 - Starting executor ID 0 on host 192.168.0.104
2019-01-13 21:26:40 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50437.
2019-01-13 21:26:40 INFO  NettyBlockTransferService:54 - Server created on 192.168.0.104:50437
2019-01-13 21:26:40 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-01-13 21:26:40 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(0, 192.168.0.104, 50437, None)
2019-01-13 21:26:40 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(0, 192.168.0.104, 50437, None)
2019-01-13 21:26:40 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(0, 192.168.0.104, 50437, None)
2019-01-13 21:26:40 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 0
2019-01-13 21:26:40 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2019-01-13 21:26:40 INFO  Executor:54 - Fetching spark://192.168.0.104:50430/jars/sparktest.jar with timestamp 1547385995160
2019-01-13 21:26:40 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:50430 after 2 ms (0 ms spent in bootstraps)
2019-01-13 21:26:40 INFO  Utils:54 - Fetching spark://192.168.0.104:50430/jars/sparktest.jar to /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-1ee810c4-971f-43df-81b1-80a52c2bd18e/executor-c927c25f-e5e6-40c9-a3e8-75dd2b465a08/spark-3ad2372c-ca16-49d2-94f6-aff6eeb911f5/fetchFileTemp6857335042780793583.tmp
2019-01-13 21:26:40 INFO  Utils:54 - Copying /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-1ee810c4-971f-43df-81b1-80a52c2bd18e/executor-c927c25f-e5e6-40c9-a3e8-75dd2b465a08/spark-3ad2372c-ca16-49d2-94f6-aff6eeb911f5/9479687331547385995160_cache to /sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/work/app-20190113212635-0005/0/./sparktest.jar
2019-01-13 21:26:40 INFO  Executor:54 - Adding file:/sunxj/InstallFile/spark-2.4.0-bin-hadoop2.7/work/app-20190113212635-0005/0/./sparktest.jar to class loader
2019-01-13 21:26:40 INFO  TorrentBroadcast:54 - Started reading broadcast variable 1
2019-01-13 21:26:40 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.0.104:50432 after 5 ms (0 ms spent in bootstraps)
2019-01-13 21:26:41 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 93.3 MB)
2019-01-13 21:26:41 INFO  TorrentBroadcast:54 - Reading broadcast variable 1 took 178 ms
2019-01-13 21:26:41 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 93.3 MB)
2019-01-13 21:26:41 INFO  HadoopRDD:54 - Input split: hdfs://localhost:9000/user_data/worldcount.txt:0+46
2019-01-13 21:26:41 INFO  TorrentBroadcast:54 - Started reading broadcast variable 0
2019-01-13 21:26:41 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.4 KB, free 93.3 MB)
2019-01-13 21:26:41 INFO  TorrentBroadcast:54 - Reading broadcast variable 0 took 21 ms
2019-01-13 21:26:41 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 328.2 KB, free 92.9 MB)
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 1193 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 1
2019-01-13 21:26:43 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2019-01-13 21:26:43 INFO  HadoopRDD:54 - Input split: hdfs://localhost:9000/user_data/worldcount.txt:46+46
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 1107 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 2
2019-01-13 21:26:43 INFO  Executor:54 - Running task 0.0 in stage 1.0 (TID 2)
2019-01-13 21:26:43 INFO  MapOutputTrackerWorker:54 - Updating epoch to 1 and clearing cache
2019-01-13 21:26:43 INFO  TorrentBroadcast:54 - Started reading broadcast variable 2
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 2027.0 B, free 92.9 MB)
2019-01-13 21:26:43 INFO  TorrentBroadcast:54 - Reading broadcast variable 2 took 18 ms
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 3.2 KB, free 92.9 MB)
2019-01-13 21:26:43 INFO  MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 0, fetching them
2019-01-13 21:26:43 INFO  MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:50430)
2019-01-13 21:26:43 INFO  MapOutputTrackerWorker:54 - Got the output locations
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 19 ms
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 0.0 in stage 1.0 (TID 2). 1422 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 3
2019-01-13 21:26:43 INFO  Executor:54 - Running task 1.0 in stage 1.0 (TID 3)
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 1.0 in stage 1.0 (TID 3). 1327 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 4
2019-01-13 21:26:43 INFO  Executor:54 - Running task 0.0 in stage 3.0 (TID 4)
2019-01-13 21:26:43 INFO  TorrentBroadcast:54 - Started reading broadcast variable 3
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 2010.0 B, free 92.9 MB)
2019-01-13 21:26:43 INFO  TorrentBroadcast:54 - Reading broadcast variable 3 took 19 ms
2019-01-13 21:26:43 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 3.2 KB, free 92.9 MB)
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms
(sunxj,1)
(window,2)
(hello,4)
(linux,3)
(world,2)
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 0.0 in stage 3.0 (TID 4). 1181 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 5
2019-01-13 21:26:43 INFO  Executor:54 - Running task 1.0 in stage 3.0 (TID 5)
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
2019-01-13 21:26:43 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms
(spark,3)
2019-01-13 21:26:43 INFO  Executor:54 - Finished task 1.0 in stage 3.0 (TID 5). 1138 bytes result sent to driver
2019-01-13 21:26:43 INFO  CoarseGrainedExecutorBackend:54 - Driver commanded a shutdown

如下图所示：

47、使用在代码中设置jar的方式，在代码中加入：

sc.addJar("/sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar")

如下图所示：

注意：/sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar的路径是build的位置，也可以自定义位置。

48、然后在Run->Debug即可，如下打印信息：

/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:51236,suspend=y,server=n -javaagent:/Users/sunxiaoju/Library/Caches/IntelliJIdea2018.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/tools.jar:/sunxj/work/sparktest/target/classes:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/srcs/scala-reflect-2.11.12-sources.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/srcs/scala-library-2.11.12-sources.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-core_2.11/2.4.0/spark-core_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/Users/sunxiaoju/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/Users/sunxiaoju/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/sunxiaoju/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-launcher_2.11/2.4.0/spark-launcher_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-kvstore_2.11/2.4.0/spark-kvstore_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-common_2.11/2.4.0/spark-network-common_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.4.0/spark-network-shuffle_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.4.0/spark-unsafe_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/Users/sunxiaoju/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/sunxiaoju/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/sunxiaoju/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/sunxiaoju/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/Users/sunxiaoju/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/sunxiaoju/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/sunxiaoju/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/sunxiaoju/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-jackson_2.11/3.5.3/json4s-jackson_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-core_2.11/3.5.3/json4s-core_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-ast_2.11/3.5.3/json4s-ast_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-scalap_2.11/3.5.3/json4s-scalap_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/Users/sunxiaoju/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/sunxiaoju/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/sunxiaoju/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/sunxiaoju/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/sunxiaoju/.m2/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-tags_2.11/2.4.0/spark-tags_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-client/2.7.7/hadoop-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-common/2.7.7/hadoop-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/sunxiaoju/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/sunxiaoju/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/sunxiaoju/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/sunxiaoju/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/sunxiaoju/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/sunxiaoju/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/sunxiaoju/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/sunxiaoju/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/sunxiaoju/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/sunxiaoju/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.7/hadoop-auth-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/sunxiaoju/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/sunxiaoju/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.7/hadoop-mapreduce-client-app-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.7/hadoop-mapreduce-client-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.7.7/hadoop-yarn-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.7.7/hadoop-yarn-server-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.7/hadoop-mapreduce-client-shuffle-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.7.7/hadoop-yarn-api-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.7/hadoop-mapreduce-client-core-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.7/hadoop-yarn-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/sunxiaoju/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.7/hadoop-mapreduce-client-jobclient-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-annotations/2.7.7/hadoop-annotations-2.7.7.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" com.sunxj.sparktest.WordCount
Connected to the target VM, address: '127.0.0.1:51236', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/13 22:06:17 INFO SparkContext: Running Spark version 2.4.0
19/01/13 22:06:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/13 22:06:18 INFO SparkContext: Submitted application: Spark 学习
19/01/13 22:06:18 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/13 22:06:18 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/13 22:06:18 INFO SecurityManager: Changing view acls groups to: 
19/01/13 22:06:18 INFO SecurityManager: Changing modify acls groups to: 
19/01/13 22:06:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/13 22:06:19 INFO Utils: Successfully started service 'sparkDriver' on port 51240.
19/01/13 22:06:19 INFO SparkEnv: Registering MapOutputTracker
19/01/13 22:06:19 INFO SparkEnv: Registering BlockManagerMaster
19/01/13 22:06:19 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/13 22:06:19 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/13 22:06:19 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-89cc396c-09e9-41ff-93fc-c7a74d734cd8
19/01/13 22:06:19 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/13 22:06:19 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/13 22:06:20 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/13 22:06:20 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/13 22:06:20 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/13 22:06:20 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 56 ms (0 ms spent in bootstraps)
19/01/13 22:06:20 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190113220620-0006
19/01/13 22:06:20 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190113220620-0006/0 on worker-20190113204613-192.168.0.104-49258 (192.168.0.104:49258) with 1 core(s)
19/01/13 22:06:20 INFO StandaloneSchedulerBackend: Granted executor ID app-20190113220620-0006/0 on hostPort 192.168.0.104:49258 with 1 core(s), 512.0 MB RAM
19/01/13 22:06:20 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190113220620-0006/0 is now RUNNING
19/01/13 22:06:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51242.
19/01/13 22:06:20 INFO NettyBlockTransferService: Server created on 192.168.0.104:51242
19/01/13 22:06:20 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/13 22:06:20 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 51242, None)
19/01/13 22:06:20 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:51242 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 51242, None)
19/01/13 22:06:20 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 51242, None)
19/01/13 22:06:20 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 51242, None)
19/01/13 22:06:21 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/13 22:06:21 INFO SparkContext: Added JAR /sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar at spark://192.168.0.104:51240/jars/sparktest.jar with timestamp 1547388381342
19/01/13 22:06:23 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/13 22:06:23 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/13 22:06:23 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:51242 (size: 23.7 KB, free: 912.3 MB)
19/01/13 22:06:23 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:17
19/01/13 22:06:25 INFO FileInputFormat: Total input paths to process : 1
19/01/13 22:06:25 INFO SparkContext: Starting job: collect at WordCount.scala:19
19/01/13 22:06:25 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:18)
19/01/13 22:06:25 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:19) with 2 output partitions
19/01/13 22:06:25 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:19)
19/01/13 22:06:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/13 22:06:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/13 22:06:25 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:18), which has no missing parents
19/01/13 22:06:25 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 912.0 MB)
19/01/13 22:06:25 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/13 22:06:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:51242 (size: 2.9 KB, free: 912.3 MB)
19/01/13 22:06:25 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
19/01/13 22:06:25 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:18) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:06:25 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/13 22:06:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.104:51246) with ID 0
19/01/13 22:06:26 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/13 22:06:26 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:51248 with 93.3 MB RAM, BlockManagerId(0, 192.168.0.104, 51248, None)
19/01/13 22:06:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:51248 (size: 2.9 KB, free: 93.3 MB)
19/01/13 22:06:27 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:51248 (size: 23.7 KB, free: 93.3 MB)
19/01/13 22:06:29 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2895 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 131 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:06:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/01/13 22:06:29 INFO DAGScheduler: ShuffleMapStage 0 (map at WordCount.scala:18) finished in 3.359 s
19/01/13 22:06:29 INFO DAGScheduler: looking for newly runnable stages
19/01/13 22:06:29 INFO DAGScheduler: running: Set()
19/01/13 22:06:29 INFO DAGScheduler: waiting: Set(ResultStage 1)
19/01/13 22:06:29 INFO DAGScheduler: failed: Set()
19/01/13 22:06:29 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:18), which has no missing parents
19/01/13 22:06:29 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.2 KB, free 912.0 MB)
19/01/13 22:06:29 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2027.0 B, free 912.0 MB)
19/01/13 22:06:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.104:51242 (size: 2027.0 B, free: 912.3 MB)
19/01/13 22:06:29 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
19/01/13 22:06:29 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:18) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:06:29 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
19/01/13 22:06:29 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:06:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.104:51248 (size: 2027.0 B, free: 93.3 MB)
19/01/13 22:06:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.0.104:51246
19/01/13 22:06:29 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 222 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 38 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:06:29 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
19/01/13 22:06:29 INFO DAGScheduler: ResultStage 1 (collect at WordCount.scala:19) finished in 0.275 s
19/01/13 22:06:29 INFO DAGScheduler: Job 0 finished: collect at WordCount.scala:19, took 4.252570 s
19/01/13 22:06:29 INFO SparkContext: Starting job: foreach at WordCount.scala:20
19/01/13 22:06:29 INFO DAGScheduler: Got job 1 (foreach at WordCount.scala:20) with 2 output partitions
19/01/13 22:06:29 INFO DAGScheduler: Final stage: ResultStage 3 (foreach at WordCount.scala:20)
19/01/13 22:06:29 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
19/01/13 22:06:29 INFO DAGScheduler: Missing parents: List()
19/01/13 22:06:29 INFO DAGScheduler: Submitting ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:18), which has no missing parents
19/01/13 22:06:29 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.2 KB, free 912.0 MB)
19/01/13 22:06:29 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2010.0 B, free 912.0 MB)
19/01/13 22:06:29 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.104:51242 (size: 2010.0 B, free: 912.3 MB)
19/01/13 22:06:29 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
19/01/13 22:06:29 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:18) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:06:29 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks
19/01/13 22:06:29 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 4, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:06:29 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.104:51248 (size: 2010.0 B, free: 93.3 MB)
19/01/13 22:06:29 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 5, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 4) in 70 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:06:29 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 5) in 30 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:06:29 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
19/01/13 22:06:29 INFO DAGScheduler: ResultStage 3 (foreach at WordCount.scala:20) finished in 0.110 s
19/01/13 22:06:29 INFO DAGScheduler: Job 1 finished: foreach at WordCount.scala:20, took 0.115631 s
19/01/13 22:06:29 INFO SparkContext: Invoking stop() from shutdown hook
19/01/13 22:06:29 INFO SparkUI: Stopped Spark web UI at http://192.168.0.104:4040
19/01/13 22:06:29 INFO StandaloneSchedulerBackend: Shutting down all executors
19/01/13 22:06:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/01/13 22:06:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/01/13 22:06:29 INFO MemoryStore: MemoryStore cleared
19/01/13 22:06:29 INFO BlockManager: BlockManager stopped
19/01/13 22:06:29 INFO BlockManagerMaster: BlockManagerMaster stopped
19/01/13 22:06:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/01/13 22:06:29 INFO SparkContext: Successfully stopped SparkContext
19/01/13 22:06:29 INFO ShutdownHookManager: Shutdown hook called
19/01/13 22:06:29 INFO ShutdownHookManager: Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-a319bc1c-3354-45e2-92a1-8af2272777ef
Disconnected from the target VM, address: '127.0.0.1:51236', transport: 'socket'

Process finished with exit code 0

如下图所示：

49、依旧去web页面查看结果，如下图所示：

50、要想直接使用debug运行spark，必须要设置两点，第一点则是设置从39~42步，第二点则是记录好jar打包的路径以及jar的名称，然后将此路径通过sc.addJar()进行设置如第47步即可直接使用debug运行了，而不需要自己打包了。

51、此时执行的结果只能在web上看，现在修改代码将结果打印到控制台中，修改如下：

package com.sunxj.sparktest

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("Spark 学习")
      .setMaster("spark://sunxj-mac-air.local:7077")
      .set("spark.executor.memory", "512m")
      .set("spark.cores.max", "1")

    val sc = new SparkContext(conf)
    //val line = sc.textFile(args(0))
    sc.addJar("/sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar")
    val file=sc.textFile("hdfs://localhost:9000/user_data/worldcount.txt")
    val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
    rdd.collect()
    rdd.collectAsMap().foreach(println)
  }
}

即将rdd.foreach(println)更改为rdd.collectAsMap().foreach(println)，如下图所示：

52、再次运行即可在控制台中打印出信息，如下信息：

/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:52626,suspend=y,server=n -javaagent:/Users/sunxiaoju/Library/Caches/IntelliJIdea2018.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/tools.jar:/sunxj/work/sparktest/target/classes:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.12.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-reflect/srcs/scala-reflect-2.11.12-sources.jar:/Users/sunxiaoju/.ivy2/cache/org.scala-lang/scala-library/srcs/scala-library-2.11.12-sources.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-core_2.11/2.4.0/spark-core_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/sunxiaoju/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/Users/sunxiaoju/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/Users/sunxiaoju/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/Users/sunxiaoju/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/Users/sunxiaoju/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/sunxiaoju/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/Users/sunxiaoju/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/Users/sunxiaoju/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-launcher_2.11/2.4.0/spark-launcher_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-kvstore_2.11/2.4.0/spark-kvstore_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-common_2.11/2.4.0/spark-network-common_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.4.0/spark-network-shuffle_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.4.0/spark-unsafe_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/Users/sunxiaoju/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/sunxiaoju/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/sunxiaoju/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/sunxiaoju/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/sunxiaoju/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/sunxiaoju/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/Users/sunxiaoju/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/sunxiaoju/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/sunxiaoju/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/sunxiaoju/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-jackson_2.11/3.5.3/json4s-jackson_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-core_2.11/3.5.3/json4s-core_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-ast_2.11/3.5.3/json4s-ast_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/json4s/json4s-scalap_2.11/3.5.3/json4s-scalap_2.11-3.5.3.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/sunxiaoju/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/sunxiaoju/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/sunxiaoju/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/Users/sunxiaoju/.m2/repository/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/Users/sunxiaoju/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/sunxiaoju/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/Users/sunxiaoju/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/sunxiaoju/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/sunxiaoju/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/sunxiaoju/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/sunxiaoju/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/sunxiaoju/.m2/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/spark/spark-tags_2.11/2.4.0/spark-tags_2.11-2.4.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-client/2.7.7/hadoop-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-common/2.7.7/hadoop-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/sunxiaoju/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/sunxiaoju/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/sunxiaoju/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/sunxiaoju/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/sunxiaoju/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/sunxiaoju/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/sunxiaoju/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/sunxiaoju/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/sunxiaoju/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/sunxiaoju/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/sunxiaoju/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.7/hadoop-auth-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/Users/sunxiaoju/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/sunxiaoju/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/sunxiaoju/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/sunxiaoju/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/sunxiaoju/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.7/hadoop-mapreduce-client-app-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.7/hadoop-mapreduce-client-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.7.7/hadoop-yarn-client-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.7.7/hadoop-yarn-server-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.7/hadoop-mapreduce-client-shuffle-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.7.7/hadoop-yarn-api-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.7/hadoop-mapreduce-client-core-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.7/hadoop-yarn-common-2.7.7.jar:/Users/sunxiaoju/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/sunxiaoju/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/sunxiaoju/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/sunxiaoju/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.7/hadoop-mapreduce-client-jobclient-2.7.7.jar:/Users/sunxiaoju/.m2/repository/org/apache/hadoop/hadoop-annotations/2.7.7/hadoop-annotations-2.7.7.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" com.sunxj.sparktest.WordCount
Connected to the target VM, address: '127.0.0.1:52626', transport: 'socket'
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/13 22:58:23 INFO SparkContext: Running Spark version 2.4.0
19/01/13 22:58:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/13 22:58:24 INFO SparkContext: Submitted application: Spark 学习
19/01/13 22:58:24 INFO SecurityManager: Changing view acls to: sunxiaoju
19/01/13 22:58:24 INFO SecurityManager: Changing modify acls to: sunxiaoju
19/01/13 22:58:24 INFO SecurityManager: Changing view acls groups to: 
19/01/13 22:58:24 INFO SecurityManager: Changing modify acls groups to: 
19/01/13 22:58:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sunxiaoju); groups with view permissions: Set(); users  with modify permissions: Set(sunxiaoju); groups with modify permissions: Set()
19/01/13 22:58:25 INFO Utils: Successfully started service 'sparkDriver' on port 52630.
19/01/13 22:58:25 INFO SparkEnv: Registering MapOutputTracker
19/01/13 22:58:25 INFO SparkEnv: Registering BlockManagerMaster
19/01/13 22:58:25 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/13 22:58:25 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/13 22:58:25 INFO DiskBlockManager: Created local directory at /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/blockmgr-c29e8e4a-e7be-40be-9c2b-75fde4d1220a
19/01/13 22:58:25 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/13 22:58:25 INFO SparkEnv: Registering OutputCommitCoordinator
19/01/13 22:58:25 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/01/13 22:58:26 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.104:4040
19/01/13 22:58:26 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://sunxj-mac-air.local:7077...
19/01/13 22:58:26 INFO TransportClientFactory: Successfully created connection to sunxj-mac-air.local/192.168.0.104:7077 after 60 ms (0 ms spent in bootstraps)
19/01/13 22:58:26 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190113225826-0013
19/01/13 22:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190113225826-0013/0 on worker-20190113204613-192.168.0.104-49258 (192.168.0.104:49258) with 1 core(s)
19/01/13 22:58:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20190113225826-0013/0 on hostPort 192.168.0.104:49258 with 1 core(s), 512.0 MB RAM
19/01/13 22:58:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52633.
19/01/13 22:58:26 INFO NettyBlockTransferService: Server created on 192.168.0.104:52633
19/01/13 22:58:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/13 22:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190113225826-0013/0 is now RUNNING
19/01/13 22:58:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.104, 52633, None)
19/01/13 22:58:26 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:52633 with 912.3 MB RAM, BlockManagerId(driver, 192.168.0.104, 52633, None)
19/01/13 22:58:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.104, 52633, None)
19/01/13 22:58:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.104, 52633, None)
19/01/13 22:58:27 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/01/13 22:58:27 INFO SparkContext: Added JAR /sunxj/work/sparktest/out/artifacts/sparktest_jar/sparktest.jar at spark://192.168.0.104:52630/jars/sparktest.jar with timestamp 1547391507239
19/01/13 22:58:29 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 250.2 KB, free 912.1 MB)
19/01/13 22:58:29 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 912.0 MB)
19/01/13 22:58:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:52633 (size: 23.7 KB, free: 912.3 MB)
19/01/13 22:58:29 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:16
19/01/13 22:58:31 INFO FileInputFormat: Total input paths to process : 1
19/01/13 22:58:31 INFO SparkContext: Starting job: collect at WordCount.scala:18
19/01/13 22:58:31 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:17)
19/01/13 22:58:31 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:18) with 2 output partitions
19/01/13 22:58:31 INFO DAGScheduler: Final stage: ResultStage 1 (collect at WordCount.scala:18)
19/01/13 22:58:31 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/01/13 22:58:31 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/01/13 22:58:31 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17), which has no missing parents
19/01/13 22:58:31 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 912.0 MB)
19/01/13 22:58:31 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.9 KB, free 912.0 MB)
19/01/13 22:58:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:52633 (size: 2.9 KB, free: 912.3 MB)
19/01/13 22:58:31 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
19/01/13 22:58:31 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:58:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/01/13 22:58:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.104:52637) with ID 0
19/01/13 22:58:32 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7899 bytes)
19/01/13 22:58:32 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.104:52639 with 93.3 MB RAM, BlockManagerId(0, 192.168.0.104, 52639, None)
19/01/13 22:58:33 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.104:52639 (size: 2.9 KB, free: 93.3 MB)
19/01/13 22:58:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.104:52639 (size: 23.7 KB, free: 93.3 MB)
19/01/13 22:58:35 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7899 bytes)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3219 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 102 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:58:35 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/01/13 22:58:35 INFO DAGScheduler: ShuffleMapStage 0 (map at WordCount.scala:17) finished in 3.713 s
19/01/13 22:58:35 INFO DAGScheduler: looking for newly runnable stages
19/01/13 22:58:35 INFO DAGScheduler: running: Set()
19/01/13 22:58:35 INFO DAGScheduler: waiting: Set(ResultStage 1)
19/01/13 22:58:35 INFO DAGScheduler: failed: Set()
19/01/13 22:58:35 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.2 KB, free 912.0 MB)
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2027.0 B, free 912.0 MB)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.104:52633 (size: 2027.0 B, free: 912.3 MB)
19/01/13 22:58:35 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
19/01/13 22:58:35 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:58:35 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
19/01/13 22:58:35 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.104:52639 (size: 2027.0 B, free: 93.3 MB)
19/01/13 22:58:35 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.0.104:52637
19/01/13 22:58:35 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 147 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 42 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:58:35 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
19/01/13 22:58:35 INFO DAGScheduler: ResultStage 1 (collect at WordCount.scala:18) finished in 0.202 s
19/01/13 22:58:35 INFO DAGScheduler: Job 0 finished: collect at WordCount.scala:18, took 4.486396 s
19/01/13 22:58:35 INFO SparkContext: Starting job: collectAsMap at WordCount.scala:19
19/01/13 22:58:35 INFO DAGScheduler: Got job 1 (collectAsMap at WordCount.scala:19) with 2 output partitions
19/01/13 22:58:35 INFO DAGScheduler: Final stage: ResultStage 3 (collectAsMap at WordCount.scala:19)
19/01/13 22:58:35 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
19/01/13 22:58:35 INFO DAGScheduler: Missing parents: List()
19/01/13 22:58:35 INFO DAGScheduler: Submitting ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.2 KB, free 912.0 MB)
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2027.0 B, free 912.0 MB)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.104:52633 (size: 2027.0 B, free: 912.3 MB)
19/01/13 22:58:35 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
19/01/13 22:58:35 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:58:35 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks
19/01/13 22:58:35 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 4, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.104:52639 (size: 2027.0 B, free: 93.3 MB)
19/01/13 22:58:35 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 5, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 4) in 44 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 5) in 30 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:58:35 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
19/01/13 22:58:35 INFO DAGScheduler: ResultStage 3 (collectAsMap at WordCount.scala:19) finished in 0.083 s
19/01/13 22:58:35 INFO DAGScheduler: Job 1 finished: collectAsMap at WordCount.scala:19, took 0.088787 s
19/01/13 22:58:35 INFO SparkContext: Starting job: foreach at WordCount.scala:21
19/01/13 22:58:35 INFO DAGScheduler: Got job 2 (foreach at WordCount.scala:21) with 2 output partitions
19/01/13 22:58:35 INFO DAGScheduler: Final stage: ResultStage 5 (foreach at WordCount.scala:21)
19/01/13 22:58:35 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
19/01/13 22:58:35 INFO DAGScheduler: Missing parents: List()
19/01/13 22:58:35 INFO DAGScheduler: Submitting ResultStage 5 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.3 KB, free 912.0 MB)
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.0 KB, free 912.0 MB)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.0.104:52633 (size: 2.0 KB, free: 912.3 MB)
19/01/13 22:58:35 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
19/01/13 22:58:35 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:58:35 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks
19/01/13 22:58:35 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 6, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.0.104:52639 (size: 2.0 KB, free: 93.3 MB)
19/01/13 22:58:35 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 7, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 6) in 66 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:58:35 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 7) in 38 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:58:35 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
19/01/13 22:58:35 INFO DAGScheduler: ResultStage 5 (foreach at WordCount.scala:21) finished in 0.113 s
19/01/13 22:58:35 INFO DAGScheduler: Job 2 finished: foreach at WordCount.scala:21, took 0.118885 s
19/01/13 22:58:35 INFO SparkContext: Starting job: collectAsMap at WordCount.scala:24
19/01/13 22:58:35 INFO DAGScheduler: Got job 3 (collectAsMap at WordCount.scala:24) with 2 output partitions
19/01/13 22:58:35 INFO DAGScheduler: Final stage: ResultStage 7 (collectAsMap at WordCount.scala:24)
19/01/13 22:58:35 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
19/01/13 22:58:35 INFO DAGScheduler: Missing parents: List()
19/01/13 22:58:35 INFO DAGScheduler: Submitting ResultStage 7 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17), which has no missing parents
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.2 KB, free 912.0 MB)
19/01/13 22:58:35 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2027.0 B, free 912.0 MB)
19/01/13 22:58:35 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.0.104:52633 (size: 2027.0 B, free: 912.3 MB)
19/01/13 22:58:35 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
19/01/13 22:58:35 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 7 (ShuffledRDD[4] at reduceByKey at WordCount.scala:17) (first 15 tasks are for partitions Vector(0, 1))
19/01/13 22:58:35 INFO TaskSchedulerImpl: Adding task set 7.0 with 2 tasks
19/01/13 22:58:35 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 8, 192.168.0.104, executor 0, partition 0, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:36 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.0.104:52639 (size: 2027.0 B, free: 93.3 MB)
19/01/13 22:58:36 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 9, 192.168.0.104, executor 0, partition 1, NODE_LOCAL, 7666 bytes)
19/01/13 22:58:36 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 8) in 46 ms on 192.168.0.104 (executor 0) (1/2)
19/01/13 22:58:36 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 9) in 29 ms on 192.168.0.104 (executor 0) (2/2)
19/01/13 22:58:36 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
19/01/13 22:58:36 INFO DAGScheduler: ResultStage 7 (collectAsMap at WordCount.scala:24) finished in 0.094 s
19/01/13 22:58:36 INFO DAGScheduler: Job 3 finished: collectAsMap at WordCount.scala:24, took 0.101431 s
(spark,3)
(sunxj,1)
(world,2)
(window,2)
(linux,3)
(hello,4)
19/01/13 22:58:36 INFO SparkContext: Invoking stop() from shutdown hook
19/01/13 22:58:36 INFO SparkUI: Stopped Spark web UI at http://192.168.0.104:4040
19/01/13 22:58:36 INFO StandaloneSchedulerBackend: Shutting down all executors
19/01/13 22:58:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/01/13 22:58:36 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/01/13 22:58:36 INFO MemoryStore: MemoryStore cleared
19/01/13 22:58:36 INFO BlockManager: BlockManager stopped
19/01/13 22:58:36 INFO BlockManagerMaster: BlockManagerMaster stopped
19/01/13 22:58:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/01/13 22:58:36 INFO SparkContext: Successfully stopped SparkContext
19/01/13 22:58:36 INFO ShutdownHookManager: Shutdown hook called
19/01/13 22:58:36 INFO ShutdownHookManager: Deleting directory /private/var/folders/7m/ls3n9dj958g25ktsw8d9cym80000gn/T/spark-162aa2b2-c951-4bdd-b922-65cee3c9c351
Disconnected from the target VM, address: '127.0.0.1:52626', transport: 'socket'

Process finished with exit code 0

如下图所示：

53、然后在查看web页面，如下图所示：

54、到此hadoop+hbase+spark+idea编写spark程序完成，此处hbase暂时没有用到。

mac系统下安装hadoop+hbase+spark单机版+IDEA和scala编写使用spark来计算的helloworld程序

一、启动ssh服务

二、建立ssh无密码登录本机

三、安装hadoop

四、安装hbase

五、安装spark

六、使用IDEA+scala编写使用spark来计算的helloworld程序

你可能感兴趣的:(mac)