在Kubernetes 集群使用 SPARK

增加权限

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default --namespace=default

Cluster 模式 测试

开启代理

# kubectl proxy
Starting to serve on 127.0.0.1:8001

spark-pi

bin/spark-submit \
    --master k8s://http://127.0.0.1:8001 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=1 \
    --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1 \
    /opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar

注意是http 不是 https
如果是https 会报一下错误

Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

response

19/05/14 10:32:29 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-1557801148384-driver
	 namespace: default
	 labels: spark-app-selector -> spark-1285a340b30f42349cca621b3015768f, spark-role -> driver
	 pod uid: 84e3af6b-75f0-11e9-ba83-001e67d8acca
	 creation time: 2019-05-14T02:32:29Z
	 service account name: default
	 volumes: spark-local-dir-1, spark-conf-volume, default-token-qqr9h
	 node name: sr535
	 start time: 2019-05-14T02:26:46Z
	 container images: zhixingheyitian/spark:spark2.4.1
	 phase: Pending
	 status: [ContainerStatus(containerID=null, image=zhixingheyitian/spark:spark2.4.1, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
19/05/14 10:32:29 INFO Client: Waiting for application spark-pi to finish...
19/05/14 10:32:31 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-1557801148384-driver
	 namespace: default
	 labels: spark-app-selector -> spark-1285a340b30f42349cca621b3015768f, spark-role -> driver
	 pod uid: 84e3af6b-75f0-11e9-ba83-001e67d8acca
	 creation time: 2019-05-14T02:32:29Z
	 service account name: default
	 volumes: spark-local-dir-1, spark-conf-volume, default-token-qqr9h
	 node name: sr535
	 start time: 2019-05-14T02:26:46Z
	 container images: zhixingheyitian/spark:spark2.4.1
	 phase: Running
	 status: [ContainerStatus(containerID=docker://5850465eae76f274593987938f680a1d80eb2abf5fabd534f10d919adf01602c, image=zhixingheyitian/spark:spark2.4.1, imageID=docker-pullable://zhixingheyitian/spark@sha256:8f08f90c68a5c4806f6ba57d1eb675c239d6fd96ca7c83cec1b6fe0d1ff25d06, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=2019-05-14T02:26:48Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})]
19/05/14 10:32:40 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-1557801148384-driver
	 namespace: default
	 labels: spark-app-selector -> spark-1285a340b30f42349cca621b3015768f, spark-role -> driver
	 pod uid: 84e3af6b-75f0-11e9-ba83-001e67d8acca
	 creation time: 2019-05-14T02:32:29Z
	 service account name: default
	 volumes: spark-local-dir-1, spark-conf-volume, default-token-qqr9h
	 node name: sr535
	 start time: 2019-05-14T02:26:46Z
	 container images: zhixingheyitian/spark:spark2.4.1
	 phase: Succeeded
	 status: [ContainerStatus(containerID=docker://5850465eae76f274593987938f680a1d80eb2abf5fabd534f10d919adf01602c, image=zhixingheyitian/spark:spark2.4.1, imageID=docker-pullable://zhixingheyitian/spark@sha256:8f08f90c68a5c4806f6ba57d1eb675c239d6fd96ca7c83cec1b6fe0d1ff25d06, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://5850465eae76f274593987938f680a1d80eb2abf5fabd534f10d919adf01602c, exitCode=0, finishedAt=2019-05-14T02:26:57Z, message=null, reason=Completed, signal=null, startedAt=2019-05-14T02:26:48Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
19/05/14 10:32:40 INFO LoggingPodStatusWatcherImpl: Container final statuses:


	 Container name: spark-kubernetes-driver
	 Container image: zhixingheyitian/spark:spark2.4.1
	 Container state: Terminated
	 Exit code: 0

不用代理，直接提交spark-pi

cluster-info

# kubectl cluster-info
Kubernetes master is running at https://10.0.2.131:6443
KubeDNS is running at https://10.0.2.131:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

run

bin/spark-submit \
    --master k8s://https://10.0.2.131:6443 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=1 \
    --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1 \
    /opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar

倘若此时，选用 http，则会报以下错误

19/05/14 10:40:43 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS.
19/05/14 10:40:44 WARN WatchConnectionManager: Exec Failure: HTTP 400, Status: 400 - Client sent an HTTP request to an HTTPS server.

java.net.ProtocolException: Expected HTTP 101 response but was '400 Bad Request'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

查看日志

# kubectl logs -f spark-pi-1557801409338-driver
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.32.0.8 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal
19/05/14 02:31:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/05/14 02:31:11 INFO SparkContext: Running Spark version 2.4.1
19/05/14 02:31:11 INFO SparkContext: Submitted application: Spark Pi
19/05/14 02:31:11 INFO SecurityManager: Changing view acls to: root
19/05/14 02:31:11 INFO SecurityManager: Changing modify acls to: root
19/05/14 02:31:11 INFO SecurityManager: Changing view acls groups to: 
19/05/14 02:31:11 INFO SecurityManager: Changing modify acls groups to: 
19/05/14 02:31:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/05/14 02:31:11 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
19/05/14 02:31:11 INFO SparkEnv: Registering MapOutputTracker
19/05/14 02:31:11 INFO SparkEnv: Registering BlockManagerMaster
19/05/14 02:31:11 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/05/14 02:31:11 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/05/14 02:31:11 INFO DiskBlockManager: Created local directory at /var/data/spark-eabbba32-543b-41da-8b32-4d77d0b053ff/blockmgr-8b6d4829-7558-4e9f-92cd-34bc24063f99
19/05/14 02:31:11 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
19/05/14 02:31:11 INFO SparkEnv: Registering OutputCommitCoordinator
19/05/14 02:31:12 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/05/14 02:31:12 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-pi-1557801409338-driver-svc.default.svc:4040
19/05/14 02:31:12 INFO SparkContext: Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar at spark://spark-pi-1557801409338-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.1.jar with timestamp 1557801072208
19/05/14 02:31:13 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes.
19/05/14 02:31:13 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
19/05/14 02:31:13 INFO NettyBlockTransferService: Server created on spark-pi-1557801409338-driver-svc.default.svc:7079
19/05/14 02:31:13 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/05/14 02:31:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-pi-1557801409338-driver-svc.default.svc, 7079, None)
19/05/14 02:31:13 INFO BlockManagerMasterEndpoint: Registering block manager spark-pi-1557801409338-driver-svc.default.svc:7079 with 413.9 MB RAM, BlockManagerId(driver, spark-pi-1557801409338-driver-svc.default.svc, 7079, None)
19/05/14 02:31:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-pi-1557801409338-driver-svc.default.svc, 7079, None)
19/05/14 02:31:13 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-pi-1557801409338-driver-svc.default.svc, 7079, None)
19/05/14 02:31:17 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.32.0.10:55182) with ID 1
19/05/14 02:31:17 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
19/05/14 02:31:17 INFO BlockManagerMasterEndpoint: Registering block manager 10.32.0.10:35701 with 413.9 MB RAM, BlockManagerId(1, 10.32.0.10, 35701, None)
19/05/14 02:31:17 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
19/05/14 02:31:17 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
19/05/14 02:31:17 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
19/05/14 02:31:17 INFO DAGScheduler: Parents of final stage: List()
19/05/14 02:31:17 INFO DAGScheduler: Missing parents: List()
19/05/14 02:31:17 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
19/05/14 02:31:18 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 413.9 MB)
19/05/14 02:31:18 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 413.9 MB)
19/05/14 02:31:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-pi-1557801409338-driver-svc.default.svc:7079 (size: 1256.0 B, free: 413.9 MB)
19/05/14 02:31:18 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/05/14 02:31:18 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
19/05/14 02:31:18 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/05/14 02:31:18 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 10.32.0.10, executor 1, partition 0, PROCESS_LOCAL, 7885 bytes)
19/05/14 02:31:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.32.0.10:35701 (size: 1256.0 B, free: 413.9 MB)
19/05/14 02:31:18 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 10.32.0.10, executor 1, partition 1, PROCESS_LOCAL, 7885 bytes)
19/05/14 02:31:18 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 790 ms on 10.32.0.10 (executor 1) (1/2)
19/05/14 02:31:19 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 42 ms on 10.32.0.10 (executor 1) (2/2)
19/05/14 02:31:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/05/14 02:31:19 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.062 s
19/05/14 02:31:19 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.139292 s
Pi is roughly 3.142155710778554
19/05/14 02:31:19 INFO SparkUI: Stopped Spark web UI at http://spark-pi-1557801409338-driver-svc.default.svc:4040
19/05/14 02:31:19 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
19/05/14 02:31:19 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
19/05/14 02:31:19 WARN ExecutorPodsWatchSnap

非主节点提交spark 任务，需要配置 token

# bin/spark-submit \
>     --master k8s://https://10.0.2.131:6443 \
>     --deploy-mode cluster \
>     --name spark-pi \
>     --class org.apache.spark.examples.SparkPi \
>     --conf spark.executor.instances=1 \
>     --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1 \
>     /opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar
19/05/14 10:38:52 WARN client.Config: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
19/05/14 10:38:53 WARN internal.WatchConnectionManager: Exec Failure
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

Client 模式

排除环境变量的干扰

unset HADOOP_HOME
unset HADOOP_CONF_DIR

run

bin/spark-sql \
    --master k8s://https://10.0.2.131:6443 \
    --name spark-sql \
    --conf spark.executor.instances=1 \
    --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1-oap-aep-executor-centos

response

19/05/14 15:10:22 INFO OapIndexInfo: 
host 10.32.0.8 executor id: 1
partition file: hdfs://bdpe101:9000/tpcds_parquet_gzip/tpcds_1g_parquet/store_sales/part-00000-a0ceba57-9659-45cd-a1ad-687011e9c91e-c000.gz.parquet use OAP index: false
19/05/14 15:10:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 12409 ms on 10.32.0.8 (executor 1) (1/1)
19/05/14 15:10:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/05/14 15:10:29 INFO DAGScheduler: ResultStage 0 (processCmd at CliDriver.java:376) finished in 12.564 s
19/05/14 15:10:29 INFO DAGScheduler: Job 0 finished: processCmd at CliDriver.java:376, took 12.636331 s
2451897	50904	5695	38271	442082	2693	46104	4	26	1	91	47.2	65.13	31.26	0.0	2844.66	4295.2	5926.83	170.67	0.0	2844.66	3015.33	-1450.54
2451897	50904	1871	38271	442082	2693	46104	4	263	1	75	83.79	93.84	3.75	0.0	281.25	6284.25	7038.0	14.06	0.0	281.25	295.31	-6003.0
2451897	50904	3533	38271	442082	2693	46104	4	179	1	15	90.79	128.01	24.32	240.76	364.8	1361.85	1920.15	8.68	240.76	124.04	132.72	-1237.81
2451897	50904	14989	38271	442082	2693	46104	4	89	1	33	28.98	40.57	23.53	0.0	776.49	956.34	1338.81	7.76	0.0	776.49	784.25	-179.85
2451897	50904	7928	38271	442082	2693	46104	4	30	1	14	87.61	145.43	84.34	0.0	1180.76	1226.54	2036.02	11.8	0.0	1180.76	1192.56	-45.78
2451897	50904	15233	38271	442082	2693	46104	4	72	1	70	96.43	109.93	47.26	0.0	3308.2	6750.1	7695.1	165.41	0.0	3308.2	3473.61	-3441.9
2451897	NULL	9497	38271	NULL	NULL	NULL	NULL	NULL	1	NULL	NULL	NULL	21.32	NULL	NULL	2389.86	4492.62	NULL	NULL	527.67	538.22	-1862.19
NULL	50904	15193	38271	442082	NULL	46104	NULL	NULL	1	24	NULL	9.8	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL
2452129	31625	1219	95293	605087	4481	40275	1	53	2	40	4.75	8.02	0.88	0.0	35.2	190.0	320.8	1.4	0.0	35.2	36.6	-154.8
2452129	31625	16745	95293	605087	4481	40275	1	130	2	99	70.57	141.14	77.62	0.0	7684.38	6986.43	13972.86	537.9	0.0	7684.38	8222.28	697.95
Time taken: 16.377 seconds, Fetched 10 row(s)
19/05/14 15:10:29 INFO SparkSQLCLIDriver: Time taken: 16.377 seconds, Fetched 10 row(s)