Spark 各种部署模式实验
Yarn
cluster
driver 端，在nodemanager 里 AM 进程里
driver 端 stdout
Client 客户端不会立即退出，持续输出统计状态信息，知道 application 结束
# bin/spark-submit --deploy-mode cluster --class org.apache.spark.sql.SparkPi ./tpcds-generate-perf_2.11-1.0.jar 10000
19/08/29 10:00:03 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
19/08/29 10:00:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/29 10:00:05 INFO RMProxy: Connecting to ResourceManager at bdpe101/192.168.26.132:8032
19/08/29 10:00:05 INFO Client: Requesting a new application from cluster with 4 NodeManagers
19/08/29 10:00:05 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (235520 MB per container)
19/08/29 10:00:05 INFO Client: Will allocate AM container, with 112640 MB memory including 10240 MB overhead
19/08/29 10:00:05 INFO Client: Setting up container launch context for our AM
19/08/29 10:00:05 INFO Client: Setting up the launch environment for our AM container
19/08/29 10:00:05 INFO Client: Preparing resources for our AM container
19/08/29 10:00:06 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/08/29 10:00:08 INFO Client: Uploading resource file:/tmp/spark-7820a16c-a85e-4457-b603-0d06fc0d9a39/__spark_libs__4293830885543743379.zip -> hdfs://bdpe101:9000/user/root/.sparkStaging/application_1563424469628_1277/__spark_libs__4293830885543743379.zip
19/08/29 10:00:10 INFO Client: Uploading resource file:/home/shen/spark-2.3.2-bin-hadoop2.7-cache/tpcds-generate-perf_2.11-1.0.jar -> hdfs://bdpe101:9000/user/root/.sparkStaging/application_1563424469628_1277/tpcds-generate-perf_2.11-1.0.jar
19/08/29 10:00:10 INFO Client: Uploading resource file:/home/oap/oap_jars/oap.jar -> hdfs://bdpe101:9000/user/root/.sparkStaging/application_1563424469628_1277/oap.jar
19/08/29 10:00:10 INFO Client: Uploading resource file:/home/oap/oap_jars/spark-sql-perf_2.11-0.4.11-SNAPSHOT.jar -> hdfs://bdpe101:9000/user/root/.sparkStaging/application_1563424469628_1277/spark-sql-perf_2.11-0.4.11-SNAPSHOT.jar
19/08/29 10:00:10 INFO Client: Uploading resource file:/tmp/spark-7820a16c-a85e-4457-b603-0d06fc0d9a39/__spark_conf__1689679907136384086.zip -> hdfs://bdpe101:9000/user/root/.sparkStaging/application_1563424469628_1277/__spark_conf__.zip
19/08/29 10:00:10 INFO SecurityManager: Changing view acls to: root
19/08/29 10:00:10 INFO SecurityManager: Changing modify acls to: root
19/08/29 10:00:10 INFO SecurityManager: Changing view acls groups to:
19/08/29 10:00:10 INFO SecurityManager: Changing modify acls groups to:
19/08/29 10:00:10 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/08/29 10:00:10 INFO Client: Submitting application application_1563424469628_1277 to ResourceManager
19/08/29 10:00:10 INFO YarnClientImpl: Submitted application application_1563424469628_1277
19/08/29 10:00:11 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:11 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.root
start time: 1567044010868
final status: UNDEFINED
tracking URL: http://bdpe101:8088/proxy/application_1563424469628_1277/
user: root
19/08/29 10:00:12 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:13 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:14 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:15 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:16 INFO Client: Application report for application_1563424469628_1277 (state: ACCEPTED)
19/08/29 10:00:17 INFO Client: Application report for application_1563424469628_1277 (state: RUNNING)
19/08/29 10:00:17 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.26.32
ApplicationMaster RPC port: 0
queue: root.root
start time: 1567044010868
final status: UNDEFINED
tracking URL: http://bdpe101:8088/proxy/application_1563424469628_1277/
user: root
19/08/29 10:00:18 INFO Client: Application report for application_1563424469628_1277 (state: RUNNING)
...
19/08/29 10:02:14 INFO Client: Application report for application_1563424469628_1277 (state: RUNNING)
19/08/29 10:02:15 INFO Client: Application report for application_1563424469628_1277 (state: RUNNING)
19/08/29 10:02:16 INFO Client: Application report for application_1563424469628_1277 (state: FINISHED)
19/08/29 10:02:16 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.26.32
ApplicationMaster RPC port: 0
queue: root.root
start time: 1567044010868
final status: SUCCEEDED
tracking URL: http://bdpe101:8088/proxy/application_1563424469628_1277/
user: root
19/08/29 10:02:16 INFO ShutdownHookManager: Shutdown hook called
19/08/29 10:02:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-4516a113-4302-40fa-afa7-f912784aa554
19/08/29 10:02:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-7820a16c-a85e-4457-b603-0d06fc0d9a39
当然手动退出 客户端进程，spark application 依然会运行至结束
Stanalone
cluster
首先Client 客户端 提交应用会立即退出
bin/spark-submit \
--master spark://bdpe101:7077 \
--deploy-mode cluster\
--conf spark.executor.memory=150g \
--conf spark.driver.memory=100g \
--class org.apache.spark.sql.SparkPi \
./tpcds-generate-perf_2.11-1.0.jar 10000
若遇到以下错误
19/08/28 16:20:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/28 16:20:59 ERROR deploy.ClientEndpoint: Exception from cluster was: java.nio.file.NoSuchFileException: /opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/./tpcds-generate-perf_2.11-1.0.jar
java.nio.file.NoSuchFileException: /opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/./tpcds-generate-perf_2.11-1.0.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
at java.nio.file.Files.copy(Files.java:1274)
at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:664)
at org.apache.spark.util.Utils$.copyFile(Utils.scala:635)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:719)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:509)
at org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)
at org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)
at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)
需要手动部署tpcds-generate-perf_2.11-1.0.jar 到 worker 节点。
（注意，Client 模式可以总动分发jar）
某个 worker 节点，会生成 DriverWrapper 进程
# jps
94036 CoarseGrainedExecutorBackend
41767 DataNode
35287 Worker
93734 DriverWrapper
42200 NodeManager
94508 Jps
[root@bdpe102 ~]# ps aux | grep 93734
root 93734 189 0.3 130082740 932040 ? Sl 10:48 0:43 /opt/Beaver/jdk/bin/java -cp /opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/conf/:/opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/jars/*:/opt/Beaver/hadoop/etc/hadoop/ -Xmx102400M -Dspark.master=spark://bdpe101:7077 -Dspark.local.dir=/opt/spark/shuffle/ -Dspark.submit.deployMode=cluster -Dspark.sql.oap.parquet.data.cache.enable=true -Dspark.memory.offHeap.enabled=true -Dspark.sql.oap.fiberCache.persistent.memory.initial.size=60g -Dspark.jars=file:/opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/./examples/jars/spark-examples_2.11-2.4.1.jar -Dspark.memory.offHeap.size=50g -Dspark.executor.memory=150g -Dspark.yarn.numa.enabled=true -Dspark.driver.supervise=false -Dspark.app.name=org.apache.spark.examples.SparkPi -Dspark.sql.oap.fiberCache.persistent.memory.reserved.size=10g -Dspark.driver.memory=100g -Dspark.rpc.askTimeout=10s org.apache.spark.deploy.worker.DriverWrapper spark://[email protected]:49433 /opt/spark-2.4.1-oap-aep-executor-centos-tpcds-kit-k8spatch-standalone-minio/work/driver-20190828104115-0001/spark-examples_2.11-2.4.1.jar org.apache.spark.examples.SparkPi 10000
root 94658 0.0 0.0 112660 968 pts/0 S+ 10:48 0:00 grep --color=auto 93734
$SPARK_HOME/work/driver-20190828162236-0008/
[root@bdpe102 driver-20190828162236-0008]# cat stdout
Main first PI
Pi is roughly 3.1436757183785917
Main last PI
补充一些日志
客户端
// 客户端进程会立即退出
//spark-examples_2.11-2.4.1.jar 为 worker 中能感知的jar 路径
$ $SPARK_HOME/bin/spark-submit --master spark://compute-1-7.jf1.dmp-pmss.intel.com:32100 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar 10000
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/10/18 01:57:06 INFO SecurityManager: Changing view acls to: xshen
19/10/18 01:57:06 INFO SecurityManager: Changing modify acls to: xshen
19/10/18 01:57:06 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:06 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xshen); groups with view permissions: Set(); users with modify permissions: Set(xshen); groups with modify permissions: Set()
19/10/18 01:57:06 INFO Utils: Successfully started service 'driverClient' on port 37221.
19/10/18 01:57:06 INFO TransportClientFactory: Successfully created connection to compute-1-7.jf1.dmp-pmss.intel.com/10.20.1.7:32100 after 69 ms (0 ms spent in bootstraps)
19/10/18 01:57:06 INFO ClientEndpoint: Driver successfully submitted as driver-20191018015706-0001
19/10/18 01:57:06 INFO ClientEndpoint: ... waiting before polling master for driver state
19/10/18 01:57:11 INFO ClientEndpoint: ... polling master for driver state
19/10/18 01:57:11 INFO ClientEndpoint: State of driver-20191018015706-0001 is RUNNING
19/10/18 01:57:11 INFO ClientEndpoint: Driver running on 10.60.219.144:32965 (worker-20191018015517-10.60.219.144-32965)
19/10/18 01:57:11 INFO ShutdownHookManager: Shutdown hook called
19/10/18 01:57:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-f4fcc525-50e1-46fa-9c69-7cc18c89a505
worker 日志，只运行executor
19/10/18 01:57:08 INFO Worker: Asked to launch executor app-20191018015708-0001/0 for Spark Pi
19/10/18 01:57:08 INFO SecurityManager: Changing view acls to: root
19/10/18 01:57:08 INFO SecurityManager: Changing modify acls to: root
19/10/18 01:57:08 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:08 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/10/18 01:57:08 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.rpc.askTimeout=10s" "-Dspark.driver.port=38835" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@spark-worker-dcpmm-statefulset-aep1-1.worker-dcpmm-service.spark-project.svc.cluster.local:38835" "--executor-id" "0" "--hostname" "10.60.171.234" "--cores" "30" "--app-id" "app-20191018015708-0001" "--worker-url" "spark://[email protected]:34163"
19/10/18 01:59:06 INFO Worker: Asked to kill executor app-20191018015708-0001/0
19/10/18 01:59:06 INFO ExecutorRunner: Runner thread for executor app-20191018015708-0001/0 interrupted
19/10/18 01:59:06 INFO ExecutorRunner: Killing process!
19/10/18 01:59:06 INFO Worker: Executor app-20191018015708-0001/0 finished with state KILLED exitStatus 143
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Clean up non-shuffle files associated with the finished executor 0
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20191018015708-0001, execId=0)
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Application app-20191018015708-0001 removed, cleanupLocalDirs = true
19/10/18 01:59:06 INFO Worker: Cleaning up local directories for application app-20191018015708-0001
worker 进程日志，该worker 也运行了 driver
19/10/18 01:57:06 INFO Worker: Asked to launch driver driver-20191018015706-0001
19/10/18 01:57:06 INFO DriverRunner: Copying user jar file:/opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar to /opt/spark/work/driver-20191018015706-0001/spark-examples_2.11-2.4.1.jar
19/10/18 01:57:06 INFO Utils: Copying /opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar to /opt/spark/work/driver-20191018015706-0001/spark-examples_2.11-2.4.1.jar
19/10/18 01:57:06 INFO DriverRunner: Launch Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/home/xshen/spark-2.4.1-bin-hadoop2.7//conf:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.app.name=spark-pi" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.rpc.askTimeout=10s" "-Dspark.master=spark://compute-1-7.jf1.dmp-pmss.intel.com:32100" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:32965" "/opt/spark/work/driver-20191018015706-0001/spark-examples_2.11-2.4.1.jar" "org.apache.spark.examples.SparkPi" "10000"
19/10/18 01:57:08 INFO Worker: Asked to launch executor app-20191018015708-0001/2 for Spark Pi
19/10/18 01:57:08 INFO SecurityManager: Changing view acls to: root
19/10/18 01:57:08 INFO SecurityManager: Changing modify acls to: root
19/10/18 01:57:08 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:08 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/10/18 01:57:08 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.rpc.askTimeout=10s" "-Dspark.driver.port=38835" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@spark-worker-dcpmm-statefulset-aep1-1.worker-dcpmm-service.spark-project.svc.cluster.local:38835" "--executor-id" "2" "--hostname" "10.60.219.144" "--cores" "29" "--app-id" "app-20191018015708-0001" "--worker-url" "spark://[email protected]:32965"
19/10/18 01:59:06 INFO Worker: Asked to kill executor app-20191018015708-0001/2
19/10/18 01:59:06 INFO ExecutorRunner: Runner thread for executor app-20191018015708-0001/2 interrupted
19/10/18 01:59:06 INFO ExecutorRunner: Killing process!
19/10/18 01:59:06 INFO Worker: Executor app-20191018015708-0001/2 finished with state KILLED exitStatus 143
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Clean up non-shuffle files associated with the finished executor 2
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20191018015708-0001, execId=2)
19/10/18 01:59:06 INFO ExternalShuffleBlockResolver: Application app-20191018015708-0001 removed, cleanupLocalDirs = true
19/10/18 01:59:06 INFO Worker: Cleaning up local directories for application app-20191018015708-0001
19/10/18 01:59:06 INFO Worker: Driver driver-20191018015706-0001 exited successfully
root@spark-worker-dcpmm-statefulset-aep1-1:/opt/spark/work/driver-20191018015706-0001# ls -l
total 4856
-rwxr-xr-x. 1 root root 2023914 Oct 18 01:57 spark-examples_2.11-2.4.1.jar
-rw-r--r--. 1 root root 2937445 Oct 18 01:59 stderr
-rw-r--r--. 1 root root 33 Oct 18 01:59 stdout
driver 日志
// head -50 stderr
Launch Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/home/xshen/spark-2.4.1-bin-hadoop2.7//conf:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.app.name=spark-pi" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.rpc.askTimeout=10s" "-Dspark.master=spark://compute-1-7.jf1.dmp-pmss.intel.com:32100" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:32965" "/opt/spark/work/driver-20191018015706-0001/spark-examples_2.11-2.4.1.jar" "org.apache.spark.examples.SparkPi" "10000"
========================================
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/10/18 01:57:07 INFO SecurityManager: Changing view acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/10/18 01:57:07 INFO Utils: Successfully started service 'Driver' on port 42391.
19/10/18 01:57:07 INFO DriverWrapper: Driver address: 10.60.219.144:42391
19/10/18 01:57:07 INFO WorkerWatcher: Connecting to worker spark://[email protected]:32965
19/10/18 01:57:07 INFO SecurityManager: Changing view acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/10/18 01:57:07 INFO TransportClientFactory: Successfully created connection to /10.60.219.144:32965 after 23 ms (0 ms spent in bootstraps)
19/10/18 01:57:07 INFO WorkerWatcher: Successfully connected to spark://[email protected]:32965
19/10/18 01:57:07 INFO SparkContext: Running Spark version 2.4.1
19/10/18 01:57:07 INFO SparkContext: Submitted application: Spark Pi
19/10/18 01:57:07 INFO SecurityManager: Changing view acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls to: root
19/10/18 01:57:07 INFO SecurityManager: Changing view acls groups to:
19/10/18 01:57:07 INFO SecurityManager: Changing modify acls groups to:
19/10/18 01:57:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/10/18 01:57:07 INFO Utils: Successfully started service 'sparkDriver' on port 38835.
19/10/18 01:57:08 INFO SparkEnv: Registering MapOutputTracker
19/10/18 01:57:08 INFO SparkEnv: Registering BlockManagerMaster
19/10/18 01:57:08 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/10/18 01:57:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/10/18 01:57:08 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-15a0a05f-acb2-4829-bc47-79875462809a
19/10/18 01:57:08 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/10/18 01:57:08 INFO SparkEnv: Registering OutputCommitCoordinator
19/10/18 01:57:08 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/10/18 01:57:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-worker-dcpmm-statefulset-aep1-1.worker-dcpmm-service.spark-project.svc.cluster.local:4040
19/10/18 01:57:08 INFO SparkContext: Added JAR file:/opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar at spark://spark-worker-dcpmm-statefulset-aep1-1.worker-dcpmm-service.spark-project.svc.cluster.local:38835/jars/spark-examples_2.11-2.4.1.jar with timestamp 1571363828188
19/10/18 01:57:08 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://compute-1-7.jf1.dmp-pmss.intel.com:32100...
19/10/18 01:57:08 INFO TransportClientFactory: Successfully created connection to compute-1-7.jf1.dmp-pmss.intel.com/10.20.1.7:32100 after 1 ms (0 ms spent in bootstraps)
19/10/18 01:57:08 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20191018015708-0001
// tail
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9968.0 in stage 0.0 (TID 9968) in 832 ms on 10.60.222.181 (executor 1) (9995/10000)
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9994.0 in stage 0.0 (TID 9994) in 573 ms on 10.60.222.181 (executor 1) (9996/10000)
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9985.0 in stage 0.0 (TID 9985) in 699 ms on 10.60.222.181 (executor 1) (9997/10000)
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9986.0 in stage 0.0 (TID 9986) in 683 ms on 10.60.222.181 (executor 1) (9998/10000)
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9992.0 in stage 0.0 (TID 9992) in 599 ms on 10.60.222.181 (executor 1) (9999/10000)
19/10/18 01:59:06 INFO TaskSetManager: Finished task 9993.0 in stage 0.0 (TID 9993) in 592 ms on 10.60.222.181 (executor 1) (10000/10000)
19/10/18 01:59:06 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/10/18 01:59:06 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 117.469 s
19/10/18 01:59:06 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 117.560139 s
19/10/18 01:59:06 INFO SparkUI: Stopped Spark web UI at http://spark-worker-dcpmm-statefulset-aep1-1.worker-dcpmm-service.spark-project.svc.cluster.local:4040
19/10/18 01:59:06 INFO StandaloneSchedulerBackend: Shutting down all executors
19/10/18 01:59:06 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/10/18 01:59:06 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
executor 日志
19/10/18 01:59:05 INFO Executor: Running task 9999.0 in stage 0.0 (TID 9999)
19/10/18 01:59:05 INFO Executor: Finished task 9936.0 in stage 0.0 (TID 9936). 781 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9942.0 in stage 0.0 (TID 9942). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9885.0 in stage 0.0 (TID 9885). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9893.0 in stage 0.0 (TID 9893). 867 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9952.0 in stage 0.0 (TID 9952). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9947.0 in stage 0.0 (TID 9947). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9950.0 in stage 0.0 (TID 9950). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9953.0 in stage 0.0 (TID 9953). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9907.0 in stage 0.0 (TID 9907). 824 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9961.0 in stage 0.0 (TID 9961). 867 bytes result sent to driver
19/10/18 01:59:05 INFO Executor: Finished task 9904.0 in stage 0.0 (TID 9904). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9981.0 in stage 0.0 (TID 9981). 867 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9983.0 in stage 0.0 (TID 9983). 867 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9935.0 in stage 0.0 (TID 9935). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9927.0 in stage 0.0 (TID 9927). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9987.0 in stage 0.0 (TID 9987). 867 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9925.0 in stage 0.0 (TID 9925). 867 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9988.0 in stage 0.0 (TID 9988). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9991.0 in stage 0.0 (TID 9991). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9926.0 in stage 0.0 (TID 9926). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9996.0 in stage 0.0 (TID 9996). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9997.0 in stage 0.0 (TID 9997). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9934.0 in stage 0.0 (TID 9934). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9956.0 in stage 0.0 (TID 9956). 867 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9957.0 in stage 0.0 (TID 9957). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9958.0 in stage 0.0 (TID 9958). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9977.0 in stage 0.0 (TID 9977). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9984.0 in stage 0.0 (TID 9984). 824 bytes result sent to driver
19/10/18 01:59:06 INFO Executor: Finished task 9999.0 in stage 0.0 (TID 9999). 867 bytes result sent to driver
19/10/18 01:59:06 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
19/10/18 01:59:06 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown```
## K8S
Client 模式下，也会分发jar
## 其他
#### oap.jar
standalone + client 模式下
--jars可以分发但是不能覆盖 spark 相关类。
#### --packages
Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.