spark0.9分布式安装
转载自：http://blog.csdn.net/myboyliu2007/article/details/18990277
原文
spark安装包：spark-0.9.0-incubating-bin-hadoop2.tgz
操作系统： CentOS6.4
jdk版本： jdk1.7.0_21
1. Cluster 模式
1.1 安装 Hadoop
用 VMware Workstation 创建三台 CentOS 虚拟机， hostname 分别设置为master,slaver01, slaver02 ，设置 SSH 无密码登陆，安装 hadoop ，然后启动hadoop 集群。参考我的这篇博客， hadoop-2.2.0分布式安装 .
1.2 Scala
在三台机器上都要安装 Scala 2.9.3, 按照我的博客 SparK安装 的步骤。 JDK 在安装 Hadoop 时已经安装了。进入 master 节点。
$ cd$ scp -r scala-2.10.3 root@slaver01:~$ scp -r scala-2.10.3 root@slaver02:~
1.3 在 master 上安装并配置 Spark
解压
$ tar -zxf spark-0.9.0-incubating-bin-hadoop2.tgz$ mv spark-0.9.0-incubating-bin-hadoop2 spark-0.9在 in conf/spark-env.sh 中设置 SCALA_HOME
$ cd ~/spark-0.9/conf$ mv spark-env.sh.template spark-env.sh$ vi spark-env.sh# add the following lineexport SCALA_HOME=/root/scala-2.10.3export JAVA_HOME=/usr/java/jdk1.7.0_21export SPARK_MASTER_IP=192.168.159.129
export SPARK_MASTER_IP=192.168.159.129
export SPARK_WORKER_MEMORY=1000m
# save and exit
在 conf/slaves , 添加 Sparkworker 的 hostname, 一行一个。
$ vim slavesslaver01slaver02master# save and exit（可选）设置 SPARK_HOME 环境变量，并将 SPARK_HOME/bin 加入 PATH
$ vim /etc/profile# add the following lines at the endexport SPARK_HOME=$HOME/spark-0.9export PATH=$PATH:$SPARK_HOME/bin# save and exit vim#make the bash profile take effect immediately$ source /etc/profile1.4 在所有 worker 上安装并配置 Spark
既然 master 上的这个文件件已经配置好了，把它拷贝到所有的 worker 即可。 注意，三台机器 spark 所在目录必须一致，因为 master 会登陆到 worker 上执行命令， master 认为 worker 的 spark 路径与自己一样。
$ cd$ scp -r spark-0.9 root@slaver01:~$ scp -r spark-0.9 root@slaver02:~1.5 启动 Spark 集群
在 master 上执行
$ cd ~/spark-0.9$ ./sbin/start-all.sh检测进程是否启动
[root@master ~]# jps3200 SecondaryNameNode7350 Master7470 Worker3025 NameNode8022 Jps3332 ResourceManager 浏览 master 的 web UI( 默认 http://master:8080 ). 这是你应该可以看到所有的work 节点，以及他们的 CPU 个数和内存等信息。
1.6 运行 Spark 自带的例子
运行 SparkPi
$ cd ~/spark-0.9./bin/run-example org.apache.spark.examples.SparkPi spark://192.168.159.129:7077SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/root/spark-0.9/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/root/spark-0.9/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]0 [spark-akka.actor.default-dispatcher-3] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started89 [spark-akka.actor.default-dispatcher-2] INFO Remoting - Starting remoting386 [spark-akka.actor.default-dispatcher-5] INFO Remoting - Remoting started; listening on addresses :[akka.tcp://spark@master:51617]393 [spark-akka.actor.default-dispatcher-3] INFO Remoting - Remoting now listens on addresses: [akka.tcp://spark@master:51617]451 [main] INFO org.apache.spark.SparkEnv - Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties451 [main] INFO org.apache.spark.SparkEnv - Registering BlockManagerMaster519 [main] INFO org.apache.spark.storage.DiskBlockManager - Created local directory at /tmp/spark-local-20140208154038-05cb524 [main] INFO org.apache.spark.storage.MemoryStore - MemoryStore started with capacity 148.5 MB.556 [main] INFO org.apache.spark.network.ConnectionManager - Bound socket to port 55761 with id = ConnectionManagerId(master,55761)563 [main] INFO org.apache.spark.storage.BlockManagerMaster - Trying to register BlockManager568 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Registering block manager master:55761 with 148.5 MB RAM569 [main] INFO org.apache.spark.storage.BlockManagerMaster - Registered BlockManager598 [main] INFO org.apache.spark.HttpServer - Starting HTTP Server680 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT706 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SocketConnector@0.0.0.0:56100707 [main] INFO org.apache.spark.broadcast.HttpBroadcast - Broadcast server started at http://192.168.159.129:56100716 [main] INFO org.apache.spark.SparkEnv - Registering MapOutputTracker723 [main] INFO org.apache.spark.HttpFileServer - HTTP File server directory is /tmp/spark-5f82de63-0061-4d3b-b0a7-156bda3935f9723 [main] INFO org.apache.spark.HttpServer - Starting HTTP Server723 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT726 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SocketConnector@0.0.0.0:47225990 [main] INFO org.eclipse.jetty.server.Server - jetty-7.x.y-SNAPSHOT991 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/storage/rdd,null}991 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/storage,null}991 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/stages/stage,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/stages/pool,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/stages,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/environment,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/executors,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/metrics/json,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/static,null}992 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{/,null}1003 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started SelectChannelConnector@0.0.0.0:40401004 [main] INFO org.apache.spark.ui.SparkUI - Started Spark Web UI at http://master:404014/02/08 15:40:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable1861 [main] INFO org.apache.spark.SparkContext - Added JAR /root/spark-0.9/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar at http://192.168.159.129:47225/jars/spark-examples_2.10-assembly-0.9.0-incubating.jar with timestamp 13918452399032274 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Connecting to master spark://192.168.159.129:7077...2586 [main] INFO org.apache.spark.SparkContext - Starting job: reduce at SparkPi.scala:392628 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Got job 0 (reduce at SparkPi.scala:39) with 2 output partitions (allowLocal=false)2684 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Final stage: Stage 0 (reduce at SparkPi.scala:39)2685 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Parents of final stage: List()2724 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Missing parents: List()2742 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Submitting Stage 0 (MappedRDD[1] at map at SparkPi.scala:35), which has no missing parents3013 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Submitting 2 missing tasks from Stage 0 (MappedRDD[1] at map at SparkPi.scala:35)3015 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 0.0 with 2 tasks3391 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Connected to Spark cluster with app ID app-20140208154041-00004273 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208154041-0000/0 on worker-20140208153258-master-37524 (master:37524) with 1 cores4274 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208154041-0000/0 on hostPort master:37524 with 1 cores, 512.0 MB RAM4277 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208154041-0000/1 on worker-20140208153257-slaver01-49982 (slaver01:49982) with 1 cores4277 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208154041-0000/1 on hostPort slaver01:49982 with 1 cores, 512.0 MB RAM4282 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208154041-0000/2 on worker-20140208153257-slaver02-53193 (slaver02:53193) with 1 cores4283 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208154041-0000/2 on hostPort slaver02:53193 with 1 cores, 512.0 MB RAM4526 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/0 is now RUNNING6497 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/1 is now RUNNING6666 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/2 is now RUNNING18070 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory27259 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Registered executor: Actor[akka.tcp://sparkExecutor@master:43274/user/Executor#-668520616] with ID 027279 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 0.0:0 as TID 0 on executor 0: master (PROCESS_LOCAL)27287 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 0.0:0 as 1426 bytes in 7 ms35953 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Registering block manager master:47550 with 297.0 MB RAM57587 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/1 is now FAILED (Command exited with code 1)57774 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Executor app-20140208154041-0000/1 removed: Command exited with code 158001 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208154041-0000/3 on worker-20140208153257-slaver01-49982 (slaver01:49982) with 1 cores58002 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208154041-0000/3 on hostPort slaver01:49982 with 1 cores, 512.0 MB RAM58002 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/3 is now RUNNING59570 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/2 is now FAILED (Command exited with code 1)59570 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Executor app-20140208154041-0000/2 removed: Command exited with code 159573 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208154041-0000/4 on worker-20140208153257-slaver02-53193 (slaver02:53193) with 1 cores59573 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208154041-0000/4 on hostPort slaver02:53193 with 1 cores, 512.0 MB RAM59699 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208154041-0000/4 is now RUNNING63430 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 0.0:1 as TID 1 on executor 0: master (PROCESS_LOCAL)63431 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 0.0:1 as 1426 bytes in 1 ms63506 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 0 in 36193 ms on master (progress: 0/2)63599 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(0, 0)63655 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 1 in 224 ms on master (progress: 1/2)63655 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(0, 1)63745 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.DAGScheduler - Stage 0 (reduce at SparkPi.scala:39) finished in 60.628 s63749 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 0.0 from pool63831 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkPi.scala:39, took 61.24527294 sPi is roughly 3.14464266 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/,null}64266 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/static,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/metrics/json,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/executors,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/environment,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/stages,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/stages/pool,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/stages/stage,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/storage,null}64267 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - stopped o.e.j.s.h.ContextHandler{/storage/rdd,null}64480 [main] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Shutting down all executors64484 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Asking each executor to shut down65576 [spark-akka.actor.default-dispatcher-16] ERROR akka.remote.EndpointWriter - AssociationError [akka.tcp://spark@master:51617] <- [akka.tcp://sparkExecutor@master:43274]: Error [Shut down address: akka.tcp://sparkExecutor@master:43274] [akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@master:43274Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.]65853 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.MapOutputTrackerMasterActor - MapOutputTrackerActor stopped!65947 [connection-manager-thread] INFO org.apache.spark.network.ConnectionManager - Selector thread was interrupted!65989 [main] INFO org.apache.spark.network.ConnectionManager - ConnectionManager stopped66015 [main] INFO org.apache.spark.storage.MemoryStore - MemoryStore cleared66017 [main] INFO org.apache.spark.storage.BlockManager - BlockManager stopped66017 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.storage.BlockManagerMasterActor - Stopping BlockManagerMaster66019 [main] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped66024 [spark-akka.actor.default-dispatcher-14] INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.66024 [spark-akka.actor.default-dispatcher-14] INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.66033 [main] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext [root@master spark-0.9]# 运行SparkLR
#Logistic Regression
[root@master spark-0.9]# ./bin/run-example org.apache.spark.examples.SparkLR spark://192.168.159.129:7077SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/root/spark-0.9/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/root/spark-0.9/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]0 [spark-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started95 [spark-akka.actor.default-dispatcher-4] INFO Remoting - Starting remoting441 [spark-akka.actor.default-dispatcher-2] INFO Remoting - Remoting started; listening on addresses :[akka.tcp://spark@master:59496]441 [spark-akka.actor.default-dispatcher-2] INFO Remoting - Remoting now listens on addresses: [akka.tcp://spark@master:59496]494 [main] INFO org.apache.spark.SparkEnv - Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties494 [main] INFO org.apache.spark.SparkEnv - Registering BlockManagerMaster565 [main] INFO org.apache.spark.storage.DiskBlockManager - Created local directory at /tmp/spark-local-20140208155527-140e569 [main] INFO org.apache.spark.storage.MemoryStore - MemoryStore started with capacity 148.5 MB.607 [main] INFO org.apache.spark.network.ConnectionManager - Bound socket to port 46771 with id = ConnectionManagerId(master,46771)kend - Registered executor: Actor[akka.tcp://sparkExecutor@slaver02:37832/user/Executor#1651743406] with ID 345103 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Executor 1 disconnected, so removing it45103 [spark-akka.actor.default-dispatcher-4] ERROR org.apache.spark.scheduler.TaskSchedulerImpl - Lost executor 1 on slaver01: remote Akka client disassociated45103 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.TaskSetManager - Re-queueing tasks for 1 from TaskSet 0.045104 [spark-akka.actor.default-dispatcher-4] WARN org.apache.spark.scheduler.TaskSetManager - Lost TID 1 (task 0.0:0)45104 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 0.0:0 as TID 3 on executor 3: slaver02 (PROCESS_LOCAL)45105 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.DAGScheduler - Executor lost: 1 (epoch 1)45105 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.storage.BlockManagerMasterActor - Trying to remove executor 1 from BlockManagerMaster.45106 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.storage.BlockManagerMaster - Removed 1 successfully in removeExecutor45433 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 0.0:0 as 551899 bytes in 329 ms45453 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208155019-0001/1 is now FAILED (Command exited with code 1)45453 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Executor app-20140208155019-0001/1 removed: Command exited with code 145468 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor added: app-20140208155019-0001/5 on worker-20140208153257-slaver01-49982 (slaver01:49982) with 1 cores45469 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Granted executor ID app-20140208155019-0001/5 on hostPort slaver01:49982 with 1 cores, 512.0 MB RAM45472 [spark-akka.actor.default-dispatcher-5] ERROR akka.remote.EndpointWriter - AssociationError [akka.tcp://spark@master:52897] -> [akka.tcp://sparkExecutor@slaver01:57904]: Error [Association failed with [akka.tcp://sparkExecutor@slaver01:57904]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@slaver01:57904]Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: slaver01/192.168.159.130:57904]45483 [spark-akka.actor.default-dispatcher-4] ERROR akka.remote.EndpointWriter - AssociationError [akka.tcp://spark@master:52897] -> [akka.tcp://sparkExecutor@slaver01:57904]: Error [Association failed with [akka.tcp://sparkExecutor@slaver01:57904]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@slaver01:57904]Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: slaver01/192.168.159.130:57904]45780 [spark-akka.actor.default-dispatcher-5] ERROR akka.remote.EndpointWriter - AssociationError [akka.tcp://spark@master:52897] -> [akka.tcp://sparkExecutor@slaver01:57904]: Error [Association failed with [akka.tcp://sparkExecutor@slaver01:57904]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@slaver01:57904]Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: slaver01/192.168.159.130:57904]45781 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.deploy.client.AppClient$ClientActor - Executor updated: app-20140208155019-0001/5 is now RUNNING46589 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Added rdd_0_1 in memory on master:44542 (size: 717.5 KB, free: 296.3 MB)46762 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 2 in 20510 ms on master (progress: 0/2)46767 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(0, 1)50080 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend - Registered executor: Actor[akka.tcp://sparkExecutor@slaver01:60056/user/Executor#1166039681] with ID 554828 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Registering block manager slaver01:43329 with 297.0 MB RAM62254 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Registering block manager slaver02:55569 with 297.0 MB RAM89782 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo - Added rdd_0_0 in memory on slaver02:55569 (size: 717.5 KB, free: 296.3 MB)90043 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 3 in 44939 ms on slaver02 (progress: 1/2)90043 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(0, 0)90061 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Stage 0 (reduce at SparkLR.scala:64) finished in 84.257 s90066 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 0.0 from pool90075 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkLR.scala:64, took 84.881569465 sOn iteration 290100 [main] INFO org.apache.spark.SparkContext - Starting job: reduce at SparkLR.scala:6490100 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Got job 1 (reduce at SparkLR.scala:64) with 2 output partitions (allowLocal=false)90100 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Final stage: Stage 1 (reduce at SparkLR.scala:64)90101 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Parents of final stage: List()90103 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Missing parents: List()90104 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Submitting Stage 1 (MappedRDD[2] at map at SparkLR.scala:62), which has no missing parents90164 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Submitting 2 missing tasks from Stage 1 (MappedRDD[2] at map at SparkLR.scala:62)90165 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 1.0 with 2 tasks90169 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 1.0:1 as TID 4 on executor 4: master (PROCESS_LOCAL)90191 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 1.0:1 as 551895 bytes in 21 ms90194 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 1.0:0 as TID 5 on executor 3: slaver02 (PROCESS_LOCAL)90220 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 1.0:0 as 551895 bytes in 25 ms91222 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 4 in 1053 ms on master (progress: 0/2)91224 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 1)91609 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 5 in 1415 ms on slaver02 (progress: 1/2)91610 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 0)91610 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.DAGScheduler - Stage 1 (reduce at SparkLR.scala:64) finished in 1.437 s91611 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkLR.scala:64, took 1.510467278 sOn iteration 391616 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 1.0 from pool91622 [main] INFO org.apache.spark.SparkContext - Starting job: reduce at SparkLR.scala:6491623 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Got job 2 (reduce at SparkLR.scala:64) with 2 output partitions (allowLocal=false)91623 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Final stage: Stage 2 (reduce at SparkLR.scala:64)91623 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Parents of final stage: List()91624 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Missing parents: List()91624 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Submitting Stage 2 (MappedRDD[3] at map at SparkLR.scala:62), which has no missing parents91777 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Submitting 2 missing tasks from Stage 2 (MappedRDD[3] at map at SparkLR.scala:62)91777 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 2.0 with 2 tasks91779 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 2.0:1 as TID 6 on executor 4: master (PROCESS_LOCAL)91899 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 2.0:1 as 551896 bytes in 119 ms91899 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 2.0:0 as TID 7 on executor 3: slaver02 (PROCESS_LOCAL)91922 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 2.0:0 as 551896 bytes in 23 ms92290 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 6 in 511 ms on master (progress: 0/2)92291 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(2, 1)92694 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 7 in 794 ms on slaver02 (progress: 1/2)92694 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 2.0 from pool92694 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(2, 0)92695 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Stage 2 (reduce at SparkLR.scala:64) finished in 0.913 s92695 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkLR.scala:64, took 1.072671482 sOn iteration 492704 [main] INFO org.apache.spark.SparkContext - Starting job: reduce at SparkLR.scala:6492704 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Got job 3 (reduce at SparkLR.scala:64) with 2 output partitions (allowLocal=false)92704 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Final stage: Stage 3 (reduce at SparkLR.scala:64)92704 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Parents of final stage: List()92707 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Missing parents: List()92707 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Submitting Stage 3 (MappedRDD[4] at map at SparkLR.scala:62), which has no missing parents92734 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.DAGScheduler - Submitting 2 missing tasks from Stage 3 (MappedRDD[4] at map at SparkLR.scala:62)92734 [spark-akka.actor.default-dispatcher-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 3.0 with 2 tasks92736 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 3.0:1 as TID 8 on executor 4: master (PROCESS_LOCAL)92759 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 3.0:1 as 551899 bytes in 23 ms92760 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 3.0:0 as TID 9 on executor 3: slaver02 (PROCESS_LOCAL)92789 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 3.0:0 as 551899 bytes in 28 ms93091 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 8 in 356 ms on master (progress: 0/2)93092 [spark-akka.actor.default-dispatcher-13] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(3, 1)96638 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 9 in 3878 ms on slaver02 (progress: 1/2)96638 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 3.0 from pool96639 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(3, 0)96639 [spark-akka.actor.default-dispatcher-2] INFO org.apache.spark.scheduler.DAGScheduler - Stage 3 (reduce at SparkLR.scala:64) finished in 3.899 s96639 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkLR.scala:64, took 3.935444196 sOn iteration 596646 [main] INFO org.apache.spark.SparkContext - Starting job: reduce at SparkLR.scala:6496646 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Got job 4 (reduce at SparkLR.scala:64) with 2 output partitions (allowLocal=false)96646 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Final stage: Stage 4 (reduce at SparkLR.scala:64)96646 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Parents of final stage: List()96649 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Missing parents: List()96649 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Submitting Stage 4 (MappedRDD[5] at map at SparkLR.scala:62), which has no missing parents96677 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Submitting 2 missing tasks from Stage 4 (MappedRDD[5] at map at SparkLR.scala:62)96677 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 4.0 with 2 tasks96678 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 4.0:1 as TID 10 on executor 4: master (PROCESS_LOCAL)96702 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 4.0:1 as 551896 bytes in 24 ms96703 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 4.0:0 as TID 11 on executor 3: slaver02 (PROCESS_LOCAL)96726 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 4.0:0 as 551896 bytes in 23 ms97554 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 10 in 876 ms on master (progress: 0/2)97555 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(4, 1)97810 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 11 in 1108 ms on slaver02 (progress: 1/2)97811 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Remove TaskSet 4.0 from pool97811 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(4, 0)97811 [spark-akka.actor.default-dispatcher-14] INFO org.apache.spark.scheduler.DAGScheduler - Stage 4 (reduce at SparkLR.scala:64) finished in 1.130 s97811 [main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkLR.scala:64, took 1.165505777 sFinal w: (5816.075967498865, 5222.008066011391, 5754.751978607454, 3853.1772062206846, 5593.565827145932, 5282.387874201054, 3662.9216051953435, 4890.78210340607, 4223.371512250292, 5767.368579668863)[root@master spark-0.9]# [root@master spark-0.9]#1.6 从 HDFS 读取文件并运行 WordCount
$ cd ~/spark-0.9$ MASTER=spark:// 192.168.159.129:7077 ./spark-shell[root@master spark-0.9]# MASTER=spark://192.168.159.129:7077 bin/spark-shell14/02/08 16:17:57 INFO HttpServer: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties14/02/08 16:17:57 INFO HttpServer: Starting HTTP ServerWelcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 0.9.0 /_/ Using Scala version 2.10.3 (Java HotSpot(TM) Client VM, Java 1.7.0_21)Type in expressions to have them evaluated.Type :help for more information.14/02/08 16:18:04 INFO Slf4jLogger: Slf4jLogger started14/02/08 16:18:04 INFO Remoting: Starting remoting14/02/08 16:18:04 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@master:57338]14/02/08 16:18:04 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@master:57338]14/02/08 16:18:04 INFO SparkEnv: Registering BlockManagerMaster14/02/08 16:18:04 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140208161804-d96e14/02/08 16:18:04 INFO MemoryStore: MemoryStore started with capacity 297.0 MB.14/02/08 16:18:04 INFO ConnectionManager: Bound socket to port 54967 with id = ConnectionManagerId(master,54967)14/02/08 16:18:04 INFO BlockManagerMaster: Trying to register BlockManager14/02/08 16:18:04 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager master:54967 with 297.0 MB RAM14/02/08 16:18:04 INFO BlockManagerMaster: Registered BlockManager14/02/08 16:18:04 INFO HttpServer: Starting HTTP Server14/02/08 16:18:04 INFO HttpBroadcast: Broadcast server started at http://192.168.159.129:5119314/02/08 16:18:04 INFO SparkEnv: Registering MapOutputTracker14/02/08 16:18:04 INFO HttpFileServer: HTTP File server directory is /tmp/spark-a63d283e-90dd-4a09-b2de-d9feda31f60714/02/08 16:18:04 INFO HttpServer: Starting HTTP Server14/02/08 16:18:05 INFO SparkUI: Started Spark Web UI at http://master:404014/02/08 16:18:05 INFO AppClient$ClientActor: Connecting to master spark://192.168.159.129:7077...14/02/08 16:18:06 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140208161806-000714/02/08 16:18:06 INFO AppClient$ClientActor: Executor added: app-20140208161806-0007/0 on worker-20140208153258-master-37524 (master:37524) with 1 cores14/02/08 16:18:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140208161806-0007/0 on hostPort master:37524 with 1 cores, 512.0 MB RAM14/02/08 16:18:06 INFO AppClient$ClientActor: Executor added: app-20140208161806-0007/1 on worker-20140208153257-slaver01-49982 (slaver01:49982) with 1 cores14/02/08 16:18:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140208161806-0007/1 on hostPort slaver01:49982 with 1 cores, 512.0 MB RAM14/02/08 16:18:06 INFO AppClient$ClientActor: Executor added: app-20140208161806-0007/2 on worker-20140208153257-slaver02-53193 (slaver02:53193) with 1 cores14/02/08 16:18:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140208161806-0007/2 on hostPort slaver02:53193 with 1 cores, 512.0 MB RAM14/02/08 16:18:06 INFO AppClient$ClientActor: Executor updated: app-20140208161806-0007/1 is now RUNNING14/02/08 16:18:06 INFO AppClient$ClientActor: Executor updated: app-20140208161806-0007/0 is now RUNNING14/02/08 16:18:06 INFO AppClient$ClientActor: Executor updated: app-20140208161806-0007/2 is now RUNNING14/02/08 16:18:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableCreated spark context..Spark context available as sc.scala> val file = sc.textFile("hdfs://192.168.159.129:9000//tmp-output101/part-r-00000")14/02/08 17:16:29 INFO MemoryStore: ensureFreeSpace(132668) called with curMem=132636, maxMem=31138775014/02/08 17:16:29 INFO MemoryStore: Block broadcast_1 stored as values to memory (estimated size 129.6 KB, free 296.7 MB)file: org.apache.spark.rdd.RDD[String] = MappedRDD[3] at textFile at <console>:12 scala> val count = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)14/02/08 17:16:42 INFO FileInputFormat: Total input paths to process : 1count: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[8] at reduceByKey at <console>:14 scala> count.collect()14/02/08 17:16:49 INFO SparkContext: Starting job: collect at <console>:1714/02/08 17:16:49 INFO DAGScheduler: Registering RDD 6 (reduceByKey at <console>:14)14/02/08 17:16:49 INFO DAGScheduler: Got job 0 (collect at <console>:17) with 2 output partitions (allowLocal=false)14/02/08 17:16:49 INFO DAGScheduler: Final stage: Stage 0 (collect at <console>:17)14/02/08 17:16:49 INFO DAGScheduler: Parents of final stage: List(Stage 1)14/02/08 17:16:49 INFO DAGScheduler: Missing parents: List(Stage 1)14/02/08 17:16:49 INFO DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[6] at reduceByKey at <console>:14), which has no missing parents14/02/08 17:16:49 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MapPartitionsRDD[6] at reduceByKey at <console>:14)14/02/08 17:16:49 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks14/02/08 17:16:49 INFO TaskSetManager: Starting task 1.0:0 as TID 0 on executor 2: slaver02 (NODE_LOCAL)14/02/08 17:16:49 INFO TaskSetManager: Serialized task 1.0:0 as 1955 bytes in 47 ms14/02/08 17:16:49 INFO TaskSetManager: Starting task 1.0:1 as TID 1 on executor 1: slaver01 (NODE_LOCAL)14/02/08 17:16:49 INFO TaskSetManager: Serialized task 1.0:1 as 1955 bytes in 0 ms14/02/08 17:17:02 INFO TaskSetManager: Finished TID 1 in 13049 ms on slaver01 (progress: 0/2)14/02/08 17:17:02 INFO DAGScheduler: Completed ShuffleMapTask(1, 1)14/02/08 17:17:05 INFO TaskSetManager: Finished TID 0 in 15876 ms on slaver02 (progress: 1/2)14/02/08 17:17:05 INFO DAGScheduler: Completed ShuffleMapTask(1, 0)14/02/08 17:17:05 INFO DAGScheduler: Stage 1 (reduceByKey at <console>:14) finished in 15.877 s14/02/08 17:17:05 INFO DAGScheduler: looking for newly runnable stages14/02/08 17:17:05 INFO DAGScheduler: running: Set()14/02/08 17:17:05 INFO DAGScheduler: waiting: Set(Stage 0)14/02/08 17:17:05 INFO DAGScheduler: failed: Set()14/02/08 17:17:05 INFO DAGScheduler: Missing parents for Stage 0: List()14/02/08 17:17:05 INFO DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[8] at reduceByKey at <console>:14), which is now runnable14/02/08 17:17:05 INFO TaskSchedulerImpl: Remove TaskSet 1.0 from pool14/02/08 17:17:05 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MapPartitionsRDD[8] at reduceByKey at <console>:14)14/02/08 17:17:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks14/02/08 17:17:05 INFO TaskSetManager: Starting task 0.0:0 as TID 2 on executor 2: slaver02 (PROCESS_LOCAL)14/02/08 17:17:05 INFO TaskSetManager: Serialized task 0.0:0 as 1807 bytes in 1 ms14/02/08 17:17:05 INFO TaskSetManager: Starting task 0.0:1 as TID 3 on executor 1: slaver01 (PROCESS_LOCAL)14/02/08 17:17:05 INFO TaskSetManager: Serialized task 0.0:1 as 1807 bytes in 0 ms14/02/08 17:17:05 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to spark@slaver01:4304814/02/08 17:17:05 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 147 bytes14/02/08 17:17:05 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to spark@slaver02:4239314/02/08 17:17:07 INFO TaskSetManager: Finished TID 2 in 1741 ms on slaver02 (progress: 0/2)14/02/08 17:17:07 INFO DAGScheduler: Completed ResultTask(0, 0)14/02/08 17:17:07 INFO TaskSetManager: Finished TID 3 in 2311 ms on slaver01 (progress: 1/2)14/02/08 17:17:07 INFO DAGScheduler: Completed ResultTask(0, 1)14/02/08 17:17:07 INFO DAGScheduler: Stage 0 (collect at <console>:17) finished in 2.321 s14/02/08 17:17:07 INFO TaskSchedulerImpl: Remove TaskSet 0.0 from pool14/02/08 17:17:07 INFO SparkContext: Job finished: collect at <console>:17, took 18.921360252 sres1: Array[(String, Int)] = Array((Fate 1,1), (wayside. 1,1), (energetic 3,1), (fourth 7,1), (gouging 1,1), (Frances 1,1), (NOVEMBER 1,1), (about. 2,1), (propel 1,1), (beginning. 2,1), (faunas 1,1), (natural 26,1), (calves, 1,1), (tends 5,1), (sea-level. 1,1), (molluscs. 1,1), (supported 3,1), (stoneless 1,1), (Here 9,1), (Planet 1,1), (dwell 1,1), (behaviour--experimenting, 1,1), (Actual 1,1), (Sluggish 1,1), (Looked 1,1), (mysterious 12,1), (fauna, 2,1), (primaries 1,1), (Mercury's 2,1), (_instinctively_ 2,1), (females 2,1), (ELECTRICITY? 1,1), (claws, 1,1), (sedimentary 6,1), (Vertebrates 4,1), ("little-brain" 4,1), (fertilises 3,1), (word. 1,1), (all; 3,1), ("atomism." 1,1), (reactions. 3,1), (away. 10,1), (beginning, 2,1), (groove. 1,1), (Egg-eating 1,1), (depreciate 1,1), (know, ...scala>1.8 停止 Spark 集群
$ cd ~/spark-0.9$ ./sbin/stop-all.sh