runzhliu

【Spark】用scala2.11编译打包构建镜像

如果关注 Spark 社区的同学已经知道，从 Spark 3.0 开始，就不再支持用 Scala 2.11 来编译了，Scala 2.11 相关的依赖也被 Owen 去掉了，所以如果要用 Spark 3.0 的同学就需要用 Scala 2.12 去编译打包了。

而如果还在用 Spark 2.x 版本的，至少在 2.4.5 的版本中，已经是将 Scala 2.12 作为默认的 Scala 版本了，如果用户的程序是用 Scala 2.11 写的，也需要将 Spark 用 Scala 2.11 编一次，而 Maven 的仓库里应该只有 Scala 2.12 编出来的 jar 依赖可以供下载，所以如果你的需求是下面那样的：

可以用 Spark 2.4.5，并且用户的代码是用 Scala 2.11 作为依赖的

那么就需要自己用 Scala 2.11 编一次 Spark 的依赖。具体的流程也比较简单，Spark 有提供一个脚本来修改项目 pom.xml 文件关于 Scala 依赖版本的工具，用户尽量不要自己去找 pom.xml 里的 Scala 依赖版本，还挺容易找错找漏的，用脚本 change-scala-version.sh 就可以了。
下面是我从编译打包到构建镜像完整流程的日志，可以供大家参考一下。

➜  spark git:(v2.4.5) dev/change-scala-version.sh --help
Usage: change-scala-version.sh [-h|--help] <version>
where :
  -h| --help Display this help text
  valid version values : 2.11 2.12

➜  spark git:(v2.4.5) dev/change-scala-version.sh 2.11
dev/../hadoop-cloud/pom.xml
dev/../assembly/pom.xml
dev/../repl/pom.xml
dev/../core/pom.xml
dev/../external/spark-ganglia-lgpl/pom.xml
dev/../external/flume/pom.xml
dev/../external/kafka-0-8/pom.xml
dev/../external/kafka-0-10-sql/pom.xml
dev/../external/kafka-0-8-assembly/pom.xml
dev/../external/kinesis-asl-assembly/pom.xml
dev/../external/flume-assembly/pom.xml
dev/../external/kinesis-asl/pom.xml
dev/../external/docker-integration-tests/pom.xml
dev/../external/kafka-0-10-assembly/pom.xml
dev/../external/kafka-0-10/pom.xml
dev/../external/avro/pom.xml
dev/../external/flume-sink/pom.xml
dev/../tools/pom.xml
dev/../streaming/pom.xml
dev/../mllib-local/pom.xml
dev/../launcher/pom.xml
dev/../pom.xml
dev/../examples/pom.xml
dev/../sql/core/pom.xml
dev/../sql/hive-thriftserver/pom.xml
dev/../sql/catalyst/pom.xml
dev/../sql/hive/pom.xml
dev/../common/network-common/pom.xml
dev/../common/tags/pom.xml
dev/../common/kvstore/pom.xml
dev/../common/network-yarn/pom.xml
dev/../common/unsafe/pom.xml
dev/../common/network-shuffle/pom.xml
dev/../common/sketch/pom.xml
dev/../mllib/pom.xml
dev/../resource-managers/kubernetes/core/pom.xml
dev/../resource-managers/kubernetes/integration-tests/pom.xml
dev/../resource-managers/yarn/pom.xml
dev/../resource-managers/mesos/pom.xml
dev/../graphx/pom.xml
dev/../docs/_plugins/copy_api_dirs.rb
➜  spark git:(v2.4.5) dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes
+++ dirname dev/make-distribution.sh
++ cd dev/..
++ pwd
+ SPARK_HOME=/root/tx/spark-all/spark
+ DISTDIR=/root/tx/spark-all/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/root/tx/spark-all/spark/build/mvn
+ ((  3  ))
+ case $1 in
+ MAKE_TGZ=true
+ shift
+ ((  2  ))
+ case $1 in
+ break
+ '[' -z /root/txKona-8.0.0-232 ']'
+ '[' -z /root/txKona-8.0.0-232 ']'
++ command -v git
+ '[' /usr/bin/git ']'
++ git rev-parse --short HEAD
+ GITREV=cee4ecb
+ '[' '!' -z cee4ecb ']'
+ GITREVSTRING=' (git revision cee4ecb)'
+ unset GITREV
++ command -v /root/tx/spark-all/spark/build/mvn
+ '[' '!' /root/tx/spark-all/spark/build/mvn ']'
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ VERSION=2.4.5
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=scala.binary.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SCALA_VERSION=2.11
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=hadoop.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SPARK_HADOOP_VERSION=2.7.3
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.activeProfiles -pl sql/hive -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ fgrep --count 'hive'
++ echo -n
+ SPARK_HIVE=0
+ '[' none == none ']'
+ NAME=2.7.3
+ echo 'Spark version is 2.4.5'
Spark version is 2.4.5
+ '[' true == true ']'
+ echo 'Making spark-2.4.5-bin-2.7.3.tgz'
Making spark-2.4.5-bin-2.7.3.tgz
+ cd /root/tx/spark-all/spark
+ export 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ MAVEN_OPTS='-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
+ echo -e '\nBuilding with...'

Building with...
+ echo -e '$ /root/tx/spark-all/spark/build/mvn' -T 1C clean package -DskipTests -Phadoop-2.7 '-Pkubernetes\n'
$ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes

+ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes
Using `mvn` from path: /root/apache-maven-3.6.2/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM                                           [pom]
[INFO] Spark Project Tags                                                 [jar]
[INFO] Spark Project Sketch                                               [jar]
[INFO] Spark Project Local DB                                             [jar]
[INFO] Spark Project Networking                                           [jar]
[INFO] Spark Project Shuffle Streaming Service                            [jar]
[INFO] Spark Project Unsafe                                               [jar]
[INFO] Spark Project Launcher                                             [jar]
[INFO] Spark Project Core                                                 [jar]
[INFO] Spark Project ML Local Library                                     [jar]
[INFO] Spark Project GraphX                                               [jar]
[INFO] Spark Project Streaming                                            [jar]
[INFO] Spark Project Catalyst                                             [jar]
[INFO] Spark Project SQL                                                  [jar]
[INFO] Spark Project ML Library                                           [jar]
[INFO] Spark Project Tools                                                [jar]
[INFO] Spark Project Hive                                                 [jar]
[INFO] Spark Project REPL                                                 [jar]
[INFO] Spark Project Kubernetes                                           [jar]
[INFO] Spark Project Assembly                                             [pom]
[INFO] Spark Integration for Kafka 0.10                                   [jar]
[INFO] Kafka 0.10+ Source for Structured Streaming                        [jar]
[INFO] Spark Project Examples                                             [jar]
[INFO] Spark Integration for Kafka 0.10 Assembly                          [jar]
[INFO] Spark Avro                                                         [jar]
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
[INFO]
.....省略很多
.....省略很多
.....省略很多
.....省略很多
.....省略很多
[INFO] -----------------< org.apache.spark:spark-parent_2.11 >-----------------
[INFO] Building Spark Project Parent POM 2.4.5                           [1/25]
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-examples_2.11 ---
[INFO] Building jar: /root/tx/spark-all/spark/examples/target/spark-examples_2.11-2.4.5-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.5:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 47.105 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 27.619 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  4.121 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 13.466 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.274 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  3.603 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 13.363 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  7.332 s]
[INFO] Spark Project Core ................................. SUCCESS [03:08 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 20.801 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 17.927 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 34.502 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:58 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:38 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:02 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 26.929 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 50.893 s]
[INFO] Spark Project REPL ................................. SUCCESS [  4.578 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 16.327 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  5.431 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 15.950 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 14.692 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 18.164 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  4.847 s]
[INFO] Spark Avro ......................................... SUCCESS [ 10.145 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:40 min (Wall Clock)
[INFO] Finished at: 2020-07-27T09:11:44+08:00
[INFO] ------------------------------------------------------------------------
+ rm -rf /root/tx/spark-all/spark/dist
+ mkdir -p /root/tx/spark-all/spark/dist/jars
+ echo 'Spark 2.4.5 (git revision cee4ecb) built for Hadoop 2.7.3'
+ echo 'Build flags: -Phadoop-2.7' -Pkubernetes
+ cp /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/activation-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-format-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-memory-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-vector-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/automaton-1.11-8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-ipc-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-mapred-1.8.2-hadoop2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill_2.11-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill-java-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-beanutils-1.9.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-cli-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compiler-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compress-1.8.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-digester-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-io-2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-net-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/compress-lzf-1.0.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/core-1.1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-client-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-framework-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-recipes-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/flatbuffers-1.2.0-3f79e055.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/generex-1.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/gson-2.2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/guava-14.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-annotations-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-auth-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-hdfs-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-app-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-core-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-api-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-server-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-api-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-locator-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-utils-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hppc-0.7.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/htrace-core-3.1.0-incubating.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpclient-4.5.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpcore-4.4.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/ivy-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-databind-2.6.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-dataformat-yaml-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-jaxrs-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-mapper-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-jaxb-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-paranamer-2.7.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-scala_2.11-2.6.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-xc-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/janino-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javassist-3.18.1-GA.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.annotation-api-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.inject-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.servlet-api-3.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.ws.rs-api-2.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jaxb-api-2.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jcl-over-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-client-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-common-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-core-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-guava-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-media-jaxb-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-server-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jetty-util-6.1.26.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/joda-time-2.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-ast_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-core_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-jackson_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-scalap_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsp-api-2.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsr305-1.3.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jtransforms-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jul-to-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kryo-shaded-4.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-client-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-common-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/leveldbjni-all-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/log4j-1.2.17.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/logging-interceptor-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/lz4-java-1.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/machinist_2.11-0.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/macro-compat_2.11-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-core-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-graphite-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-json-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-jvm-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/minlog-1.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-3.9.9.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-all-4.1.42.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/objenesis-2.5.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okhttp-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okio-1.15.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/opencsv-2.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/oro-2.0.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/osgi-resource-locator-1.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-column-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-common-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-encoding-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-format-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-hadoop-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-jackson-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/protobuf-java-2.5.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/py4j-0.10.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/pyrolite-4.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/RoaringBitmap-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-compiler-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-library-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-parser-combinators_2.11-1.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-reflect-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-xml_2.11-1.0.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shapeless_2.11-2.3.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shims-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-api-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-log4j12-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snakeyaml-1.15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snappy-java-1.1.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-catalyst_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-graphx_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kubernetes_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kvstore_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-launcher_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib-local_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-common_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-shuffle_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-repl_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sketch_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sql_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-streaming_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5-tests.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-unsafe_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire-macros_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stax-api-1.0-2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stream-2.7.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/univocity-parsers-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/validation-api-1.1.0.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xbean-asm6-shaded-4.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xercesImpl-2.9.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xmlenc-0.52.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xz-1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zjsonpatch-0.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zookeeper-3.4.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zstd-jni-1.3.2-2.jar /root/tx/spark-all/spark/dist/jars/
+ '[' -f '/root/tx/spark-all/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar' ']'
+ '[' -d /root/tx/spark-all/spark/resource-managers/kubernetes/core/target/ ']'
+ mkdir -p /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/docker/src/main/dockerfiles /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/integration-tests/tests /root/tx/spark-all/spark/dist/kubernetes/
+ mkdir -p /root/tx/spark-all/spark/dist/examples/jars
+ cp /root/tx/spark-all/spark/examples/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/scopt_2.11-3.7.0.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar /root/tx/spark-all/spark/dist/examples/jars
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ name=aircompressor-0.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/aircompressor-0.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ name=commons-codec-1.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-codec-1.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ name=commons-lang-2.6.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-lang-2.6.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ name=orc-core-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-core-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ name=orc-mapreduce-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-mapreduce-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ name=orc-shims-1.5.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-shims-1.5.5.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ name=paranamer-2.8.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/paranamer-2.8.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/scopt_2.11-3.7.0.jar
+ name=scopt_2.11-3.7.0.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/scopt_2.11-3.7.0.jar ']'
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/spark-examples_2.11-2.4.5.jar
+ name=spark-examples_2.11-2.4.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/spark-examples_2.11-2.4.5.jar ']'
+ mkdir -p /root/tx/spark-all/spark/dist/examples/src/main
+ cp -r /root/tx/spark-all/spark/examples/src/main /root/tx/spark-all/spark/dist/examples/src/
+ '[' -e /root/tx/spark-all/spark/LICENSE-binary ']'
+ cp /root/tx/spark-all/spark/LICENSE-binary /root/tx/spark-all/spark/dist/LICENSE
+ cp -r /root/tx/spark-all/spark/licenses-binary /root/tx/spark-all/spark/dist/licenses
+ cp /root/tx/spark-all/spark/NOTICE-binary /root/tx/spark-all/spark/dist/NOTICE
+ '[' -e /root/tx/spark-all/spark/CHANGES.txt ']'
+ cp -r /root/tx/spark-all/spark/data /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ echo 'Skipping building python distribution package'
Skipping building python distribution package
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/conf/docker.properties.template /root/tx/spark-all/spark/conf/fairscheduler.xml.template /root/tx/spark-all/spark/conf/log4j.properties.template /root/tx/spark-all/spark/conf/metrics.properties.template /root/tx/spark-all/spark/conf/slaves.template /root/tx/spark-all/spark/conf/spark-defaults.conf.template /root/tx/spark-all/spark/conf/spark-env.sh.template /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/README.md /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/bin /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/python /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ cp -r /root/tx/spark-all/spark/sbin /root/tx/spark-all/spark/dist
+ '[' -d /root/tx/spark-all/spark/R/lib/SparkR ']'
+ '[' true == true ']'
+ TARDIR_NAME=spark-2.4.5-bin-2.7.3
+ TARDIR=/root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ cp -r /root/tx/spark-all/spark/dist /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ tar czf spark-2.4.5-bin-2.7.3.tgz -C /root/tx/spark-all/spark spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
➜  spark git:(v2.4.5) tar zxvf spark-2.4.5-bin-2.7.3.tgz
...省略很多
...省略很多
...省略很多
➜  spark git:(v2.4.5) ✗ cd spark-2.4.5-bin-2.7.3
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon  201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
 ---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
 ---> Running in 788ed25e1956
Removing intermediate container 788ed25e1956
 ---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
 ---> Running in fd77ef759bed
Removing intermediate container fd77ef759bed
 ---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
 ---> Running in fe4d6784371b
Removing intermediate container fe4d6784371b
 ---> 4ac25b12085b
Step 5/15 : RUN set -ex &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y bash tini libc6 libpam-modules libnss3 &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/*
 ---> Running in ea60b610c182
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
^C
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ vi kubernetes/dockerfiles/spark/Dockerfile
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon  201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
 ---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
 ---> Using cache
 ---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
 ---> Using cache
 ---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
 ---> Using cache
 ---> 4ac25b12085b
Step 5/15 : RUN set -ex &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y bash tini libc6 libpam-modules libnss3 &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/*
 ---> Running in 3e9d1d5ac7c1
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Fetched 8365 kB in 3s (3140 kB/s)
Reading package lists...
+ ln -s /lib /lib64
+ apt install -y bash tini libc6 libpam-modules libnss3

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
bash is already the newest version (5.0-4).
bash set to manually installed.
libc6 is already the newest version (2.28-10).
libc6 set to manually installed.
libpam-modules is already the newest version (1.3.1-5).
libpam-modules set to manually installed.
The following package was automatically installed and is no longer required:
  lsb-base
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libnspr4 libsqlite3-0
The following NEW packages will be installed:
  libnspr4 libnss3 libsqlite3-0 tini
0 upgraded, 4 newly installed, 0 to remove and 15 not upgraded.
Need to get 2159 kB of archives.
After this operation, 6175 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u3 [1159 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libsqlite3-0 amd64 3.27.2-3 [641 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 tini amd64 0.18.0-1 [247 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2159 kB in 1s (3301 kB/s)
Selecting previously unselected package libnspr4:amd64.
(Reading database ... 6888 files and directories currently installed.)
Preparing to unpack .../libnspr4_2%3a4.20-1_amd64.deb ...
Unpacking libnspr4:amd64 (2:4.20-1) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../libsqlite3-0_3.27.2-3_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3) ...
Selecting previously unselected package libnss3:amd64.
Preparing to unpack .../libnss3_2%3a3.42.1-1+deb10u3_amd64.deb ...
Unpacking libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Selecting previously unselected package tini.
Preparing to unpack .../tini_0.18.0-1_amd64.deb ...
Unpacking tini (0.18.0-1) ...
Setting up libsqlite3-0:amd64 (3.27.2-3) ...
Setting up libnspr4:amd64 (2:4.20-1) ...
Setting up tini (0.18.0-1) ...
Setting up libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
+ mkdir -p /opt/spark
+ mkdir -p /opt/spark/work-dir
+ touch /opt/spark/RELEASE
+ rm /bin/sh
+ ln -sv /bin/bash /bin/sh
'/bin/sh' -> '/bin/bash'
+ echo auth required pam_wheel.so use_uid
+ chgrp root /etc/passwd
+ chmod ug+rw /etc/passwd
+ rm -rf /var/cache/apt/archives
Removing intermediate container 3e9d1d5ac7c1
 ---> 1ffbc3c2f99f
Step 6/15 : COPY ${spark_jars} /opt/spark/jars
 ---> 9a4610c4c102
Step 7/15 : COPY bin /opt/spark/bin
 ---> 06c908997a9f
Step 8/15 : COPY sbin /opt/spark/sbin
 ---> b32be7e7dda4
Step 9/15 : COPY ${img_path}/spark/entrypoint.sh /opt/
 ---> 6bee90987e7f
Step 10/15 : COPY examples /opt/spark/examples
 ---> 65775aca9747
Step 11/15 : COPY ${k8s_tests} /opt/spark/tests
 ---> 6768a8d5c69a
Step 12/15 : COPY data /opt/spark/data
 ---> 1520647faf15
Step 13/15 : ENV SPARK_HOME /opt/spark
 ---> Running in 7688890172f1
Removing intermediate container 7688890172f1
 ---> bf2dd6442c38
Step 14/15 : WORKDIR /opt/spark/work-dir
 ---> Running in b42600a07aaf
Removing intermediate container b42600a07aaf
 ---> 96c18bdcae55
Step 15/15 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
 ---> Running in 62490f481000
Removing intermediate container 62490f481000
 ---> c016374ced2b
Successfully built c016374ced2b
Successfully tagged yourhub.com/pulic/spark:2.11_2.4.5
Sending build context to Docker daemon  201.1MB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
 ---> c016374ced2b
Step 3/9 : WORKDIR /
 ---> Running in ebdd964393d8
Removing intermediate container ebdd964393d8
 ---> 76ca1b8a91e3
Step 4/9 : RUN mkdir ${SPARK_HOME}/python
 ---> Running in 3073d6d8a3f7
Removing intermediate container 3073d6d8a3f7
 ---> 176a7fb978a3
Step 5/9 : RUN apt install -y python python-pip &&     apt install -y python3 python3-pip &&     rm -r /usr/lib/python*/ensurepip &&     pip install --upgrade pip setuptools &&     rm -r /root/.cache && rm -rf /var/cache/apt/*
 ---> Running in 4a5fdb7337fa


➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ ls
bin   data      jars        LICENSE   NOTICE  README.md  sbin
conf  examples  kubernetes  licenses  python  RELEASE
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker images | grep -i "2.11_2.4.5"
yourhub.com/pulic/spark-py                                 2.11_2.4.5                                 c7190686b340        About a minute ago   1GB
yourhub.com/pulic/spark                                    2.11_2.4.5                                 c016374ced2b        2 minutes ago        504MB
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark]
7006d88e7365: Pushed
cbcceeb16130: Pushed
d18039eb9e2c: Pushed
2e4b66a0a7ba: Pushed
e94624e3886b: Pushed
ba738a43f5a4: Pushed
2c069a31829d: Pushed
da53a3e76eb8: Pushed
34f9780c8bfb: Mounted from user/dev-spark-operator
e871e75299d2: Mounted from user/dev-spark-operator
109e67eff29c: Mounted from user/dev-spark-operator
556c5fb0d91b: Mounted from user/dev-spark-operator
2.11_2.4.5: digest: sha256:60bccd043dd4d16c7cc2b0879c6de394d1d48561f624082ddd627af2f9c96084 size: 2838
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark-py:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark-py]
fb728a4376ed: Pushed
19442796254b: Pushed
bf5c82549be3: Pushed
7006d88e7365: Mounted from pulic/spark
cbcceeb16130: Mounted from pulic/spark
d18039eb9e2c: Mounted from pulic/spark
2e4b66a0a7ba: Mounted from pulic/spark
e94624e3886b: Mounted from pulic/spark
ba738a43f5a4: Mounted from pulic/spark
2c069a31829d: Mounted from pulic/spark
da53a3e76eb8: Mounted from pulic/spark
34f9780c8bfb: Mounted from pulic/spark
e871e75299d2: Mounted from pulic/spark
109e67eff29c: Mounted from pulic/spark
556c5fb0d91b: Mounted from pulic/spark
2.11_2.4.5: digest: sha256:49c0b17c8addc21bd30735ccd6cdc88c3c57e826f1b138cbedd1b13fa23258f8 size: 3469
➜  spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker run -it yourhub.com/pulic/spark:2.11_2.4.5 bash
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_K8S_CMD=bash
+ case "$SPARK_K8S_CMD" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ exec /usr/bin/tini -s -- bash
root@a3f20a0deb94:/opt/spark/work-dir#
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# cd ..
root@a3f20a0deb94:/opt/spark# ls
RELEASE  bin  data  examples  jars  sbin  tests  work-dir
root@a3f20a0deb94:/opt/spark# ls jars/
RoaringBitmap-0.7.45.jar		     hk2-locator-2.4.0-b34.jar			objenesis-2.5.1.jar
activation-1.1.1.jar			     hk2-utils-2.4.0-b34.jar			okhttp-3.12.0.jar
aircompressor-0.10.jar			     hppc-0.7.2.jar				okio-1.15.0.jar
antlr4-runtime-4.7.jar			     htrace-core-3.1.0-incubating.jar		opencsv-2.3.jar
aopalliance-repackaged-2.4.0-b34.jar	     httpclient-4.5.6.jar			orc-core-1.5.5-nohive.jar
apacheds-i18n-2.0.0-M15.jar		     httpcore-4.4.10.jar			orc-mapreduce-1.5.5-nohive.jar
apacheds-kerberos-codec-2.0.0-M15.jar	     ivy-2.4.0.jar				orc-shims-1.5.5.jar
api-asn1-api-1.0.0-M20.jar		     jackson-annotations-2.6.7.jar		oro-2.0.8.jar
api-util-1.0.0-M20.jar			     jackson-core-2.6.7.jar			osgi-resource-locator-1.0.1.jar
arpack_combined_all-0.1.jar		     jackson-core-asl-1.9.13.jar		paranamer-2.8.jar
arrow-format-0.10.0.jar			     jackson-databind-2.6.7.3.jar		parquet-column-1.10.1.jar
arrow-memory-0.10.0.jar			     jackson-dataformat-yaml-2.6.7.jar		parquet-common-1.10.1.jar
arrow-vector-0.10.0.jar			     jackson-jaxrs-1.9.13.jar			parquet-encoding-1.10.1.jar
automaton-1.11-8.jar			     jackson-mapper-asl-1.9.13.jar		parquet-format-2.4.0.jar
avro-1.8.2.jar				     jackson-module-jaxb-annotations-2.6.7.jar	parquet-hadoop-1.10.1.jar
avro-ipc-1.8.2.jar			     jackson-module-paranamer-2.7.9.jar		parquet-jackson-1.10.1.jar
avro-mapred-1.8.2-hadoop2.jar		     jackson-module-scala_2.11-2.6.7.1.jar	protobuf-java-2.5.0.jar
breeze-macros_2.11-0.13.2.jar		     jackson-xc-1.9.13.jar			py4j-0.10.7.jar
breeze_2.11-0.13.2.jar			     janino-3.0.9.jar				pyrolite-4.13.jar
chill-java-0.9.3.jar			     javassist-3.18.1-GA.jar			scala-compiler-2.11.12.jar
chill_2.11-0.9.3.jar			     javax.annotation-api-1.2.jar		scala-library-2.11.12.jar
commons-beanutils-1.9.4.jar		     javax.inject-2.4.0-b34.jar			scala-parser-combinators_2.11-1.1.0.jar
commons-cli-1.2.jar			     javax.servlet-api-3.1.0.jar		scala-reflect-2.11.12.jar
commons-codec-1.10.jar			     javax.ws.rs-api-2.0.1.jar			scala-xml_2.11-1.0.5.jar
commons-collections-3.2.2.jar		     jaxb-api-2.2.2.jar				shapeless_2.11-2.3.2.jar
commons-compiler-3.0.9.jar		     jcl-over-slf4j-1.7.16.jar			shims-0.7.45.jar
commons-compress-1.8.1.jar		     jersey-client-2.22.2.jar			slf4j-api-1.7.16.jar
commons-configuration-1.6.jar		     jersey-common-2.22.2.jar			slf4j-log4j12-1.7.16.jar
commons-crypto-1.0.0.jar		     jersey-container-servlet-2.22.2.jar	snakeyaml-1.15.jar
commons-digester-1.8.jar		     jersey-container-servlet-core-2.22.2.jar	snappy-java-1.1.7.3.jar
commons-httpclient-3.1.jar		     jersey-guava-2.22.2.jar			spark-catalyst_2.11-2.4.5.jar
commons-io-2.4.jar			     jersey-media-jaxb-2.22.2.jar		spark-core_2.11-2.4.5.jar
commons-lang-2.6.jar			     jersey-server-2.22.2.jar			spark-graphx_2.11-2.4.5.jar
commons-lang3-3.5.jar			     jetty-util-6.1.26.jar			spark-kubernetes_2.11-2.4.5.jar
commons-math3-3.4.1.jar			     joda-time-2.9.3.jar			spark-kvstore_2.11-2.4.5.jar
commons-net-3.1.jar			     json4s-ast_2.11-3.5.3.jar			spark-launcher_2.11-2.4.5.jar
compress-lzf-1.0.3.jar			     json4s-core_2.11-3.5.3.jar			spark-mllib-local_2.11-2.4.5.jar
core-1.1.2.jar				     json4s-jackson_2.11-3.5.3.jar		spark-mllib_2.11-2.4.5.jar
curator-client-2.7.1.jar		     json4s-scalap_2.11-3.5.3.jar		spark-network-common_2.11-2.4.5.jar
curator-framework-2.7.1.jar		     jsp-api-2.1.jar				spark-network-shuffle_2.11-2.4.5.jar
curator-recipes-2.7.1.jar		     jsr305-1.3.9.jar				spark-repl_2.11-2.4.5.jar
flatbuffers-1.2.0-3f79e055.jar		     jtransforms-2.4.0.jar			spark-sketch_2.11-2.4.5.jar
generex-1.0.2.jar			     jul-to-slf4j-1.7.16.jar			spark-sql_2.11-2.4.5.jar
gson-2.2.4.jar				     kryo-shaded-4.0.2.jar			spark-streaming_2.11-2.4.5.jar
guava-14.0.1.jar			     kubernetes-client-4.6.1.jar		spark-tags_2.11-2.4.5-tests.jar
hadoop-annotations-2.7.3.jar		     kubernetes-model-4.6.1.jar			spark-tags_2.11-2.4.5.jar
hadoop-auth-2.7.3.jar			     kubernetes-model-common-4.6.1.jar		spark-unsafe_2.11-2.4.5.jar
hadoop-client-2.7.3.jar			     leveldbjni-all-1.8.jar			spire-macros_2.11-0.13.0.jar
hadoop-common-2.7.3.jar			     log4j-1.2.17.jar				spire_2.11-0.13.0.jar
hadoop-hdfs-2.7.3.jar			     logging-interceptor-3.12.0.jar		stax-api-1.0-2.jar
hadoop-mapreduce-client-app-2.7.3.jar	     lz4-java-1.4.0.jar				stream-2.7.0.jar
hadoop-mapreduce-client-common-2.7.3.jar     machinist_2.11-0.6.1.jar			univocity-parsers-2.7.3.jar
hadoop-mapreduce-client-core-2.7.3.jar	     macro-compat_2.11-1.1.1.jar		validation-api-1.1.0.Final.jar
hadoop-mapreduce-client-jobclient-2.7.3.jar  metrics-core-3.1.5.jar			xbean-asm6-shaded-4.8.jar
hadoop-mapreduce-client-shuffle-2.7.3.jar    metrics-graphite-3.1.5.jar			xercesImpl-2.9.1.jar
hadoop-yarn-api-2.7.3.jar		     metrics-json-3.1.5.jar			xmlenc-0.52.jar
hadoop-yarn-client-2.7.3.jar		     metrics-jvm-3.1.5.jar			xz-1.5.jar
hadoop-yarn-common-2.7.3.jar		     minlog-1.3.0.jar				zjsonpatch-0.3.0.jar
hadoop-yarn-server-common-2.7.3.jar	     netty-3.9.9.Final.jar			zookeeper-3.4.6.jar
hk2-api-2.4.0-b34.jar

你可能感兴趣的:(Kubernetes,Spark)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
react-intl——react国际化使用方案苹果酱0567 面试题汇总与解析 java 开发语言中间件 spring boot 后端
国际化介绍i18n：internationalization国家化简称，首字母+首尾字母间隔的字母个数+尾字母，类似的还有k8s(Kubernetes)React-intl是React中最受欢迎的库。使用步骤安装#usenpmnpminstallreact-intl-D#useyarn项目入口文件配置//index.tsximportReactfrom"react";importReactDOMf
Kubernetes数据持久化看清所苡看轻 kubernetes(k8s)emptyDir HostPath pv pvc kubernetes
在k8s中，Volume（数据卷）存在明确的生命周期（与包含该数据卷的容器组（pod）相同）。因此Volume的生命周期比同一容器组（pod）中任意容器的生命周期要更长，不管容器重启了多少次，数据都被保留下来。当然，如果pod不存在了，数据卷自然退出了。此时，根据pod所使用的数据卷类型不同，数据可能随着数据卷的退出而删除，也可能被真正持久化，并在下次容器组重启时仍然可以使用。从根本上来说，一个数
Kubernetes部署MySQL数据持久化沫殇-MS Kubernetes MySQL数据库 kubernetes mysql 容器
一、安装配置NFS服务端1、安装nfs-kernel-server：sudoapt-yinstallnfs-kernel-server2、服务端创建共享目录#列出所有可用块设备的信息lsblk#格式化磁盘sudomkfs-text4/dev/sdb#创建一个目录：sudomkdir-p/data/nfs/mysql#更改目录权限：sudochown-Rnobody:nogroup/data/nfs
Kubernetes的3种数据持久化方式 Seal^_^ 【云原生】容器化与编排技术持续集成 #Kubernetes kubernetes 容器云原生 EmptyDir 面试 HostPath
Kubernetes的3种数据持久化方式1.EmptyDir2.HostPath3.PersistentVolume(PV)TheBegin点点关注，收藏不迷路Kubernetes提供了几种数据持久化方式，以满足不同场景的需求：1.EmptyDir用途：临时数据存储，Pod内容器间共享。特点：生命周期与Pod相同，Pod删除时数据也删除。2.HostPath用途：访问宿主机特定文件或目录。特点：增
【Kubernetes】常见面试题汇总（十一） summer.335 Kubernetes kubernetes 容器云原生
目录33.简述Kubernetes外部如何访问集群内的服务？34.简述Kubernetesingress？35.简述Kubernetes镜像的下载策略？33.简述Kubernetes外部如何访问集群内的服务？（1）对于Kubernetes，集群外的客户端默认情况，无法通过Pod的IP地址或者Service的虚拟IP地址：虚拟端口号进行访问。（2）通常可以通过以下方式进行访问Kubernetes集群
k8s中Service暴露的种类以及用法听说唐僧不吃肉 K8S kubernetes 容器云原生
一、说明在Kubernetes中，有几种不同的方式可以将服务（Service）暴露给外部流量。这些方式通过定义服务的spec.type字段来确定。二、详解1.ClusterIP定义：默认类型，服务只能在集群内部访问。作用：通过集群内部IP地址暴露服务。示例：spec:type:ClusterIPports:-port:80targetPo
Kubernetes 自定义控制器开发 IT回忆录 Kubenetes kubernetes
目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
用kubedam搭建的k8s证书过期处理方法我滴鬼鬼呀wks k8s 1024程序员节
kubeadm部署的k8s证书过期1、查看证书过期时间kubeadmalphacertscheck-expiration若证书已经过期无法试用kubectl命令建议修改服务器时间到未过期的时间段2、配置kube-controller-manager.yaml文件cat/etc/kubernetes/manifests/kube-controller-manager.yamlapiVersion:v
k8s证书过期问题处理 olina_qin kubernetes 容器云原生
k8s证书过期问题处理opensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-dateskubeadmcertsrenewallsystemctlrestartkubeleopensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-text|grep"NotAfter"cp/etc/kubernet
Kubernetes Ingress 控制器（Nginx）安装与使用教程农优影
KubernetesIngress控制器（Nginx）安装与使用教程kubernetes-ingressNGINXandNGINXPlusIngressControllersforKubernetes项目地址:https://gitcode.com/gh_mirrors/ku/kubernetes-ingress1.项目目录结构及介绍在nginxinc/kubernetes-ingress仓库中，
【K8s】专题十一：Kubernetes 集群证书过期处理方法行者Sun1989 Kubernetes kubernetes 云原生容器
本文内容均来自个人笔记并重新梳理，如有错误欢迎指正！如果对您有帮助，烦请点赞、关注、转发、订阅专栏！专栏订阅入口Linux专栏|Docker专栏|Kubernetes专栏往期精彩文章【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法（续）【Docker】MySQL源码构建Docker镜
分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据 python hadoop 计算机毕业设计选题计算机毕业设计源码数据分析 spark毕设
作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
Docker学习十一：Kubernetes概述爱打羽球的程序猿 Docker学习系列 docker kubernetes 学习
一、Kubernetes简介2006年，Google提出了云计算的概念，当时的云计算领域还是以虚拟机为代表的云平台。2013年，Docker横空出世，Docker提出了镜像、仓库等核心概念，规范了服务的交付标准，使得复杂服务的落地变得更加简单，之后Docker又定义了OCI标准，Docker在容器领域称为事实的标准。但是，Docker诞生只是帮助定义了开发和交付标准，如果想要在生产环境中大批量的使
Cloud Native Weekly | 华为云抢先发布Redis5.0，红帽宣布收购混合云提供商 weixin_34302561 数据库 devops 大数据
1——华为云抢先发布Redis5.02——DigitalOceanK8s服务正式上线3——红帽宣布收购混合云提供商NooBaa4——微软发布多项AzureKubernetes服务更新1华为云抢先发布Redis5.012月17日，华为云在DCS2.0的基础上，快人一步，抢先推出了新的Redis5.0产品，这是一个崭新的突破。目前国内在缓存领域的发展普遍停留在Redis4.0阶段，华为云率先发布了Re
（k8s）Kubernetes 从0到1容器编排之旅道不贱卖，法不轻传 kubernets kubernetes 容器云原生
一、引言在当今数字化的浪潮中，Kubernetes如同一艘强大的航船，引领着容器化应用的部署与管理。它以其卓越的灵活性、可扩展性和可靠性，成为众多企业和开发者的首选。然而，要真正发挥Kubernetes的强大威力，仅仅掌握基本操作是远远不够的。本文将带你深入探索Kubernetes使用过程中的奇技妙法，为你开启一段优雅的容器编排之旅。二、高级资源管理之精妙艺术1.资源配额与限制：雕琢资源之美•Ku
Spark 组件 GraphX、Streaming 叶域大数据 spark spark 大数据分布式
Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
【K8S】kubernetes集群架构与组件奇奇怪怪^ 云 Linux IT 运维服务器 linux
文章目录【K8S】kubernetes集群架构与组件kubernetes组件**master组件**node组件整体流程POD终止过程【K8S】kubernetes集群架构与组件kubernetes组件K8S是属于主从设备模型(Master-slave架构)，即有Master节点负责集群的调度、管理和运维，Slave节点是集群中的运算工作负载节点在K8S中，主节点一般被称为Master节点，而从节
K8S学习笔记02——K8S组件沉淅尘 #Docker #K8S kubernetes
Kubernetes组件一、控制平面组件（ControlPlaneComponents）(1)kube-apiserver(2)etcd(3)kube-scheduler(4)kube-controller-manager(5)cloud-controller-manager二、Node组件1.kubelet2.kube-proxy3.容器运行时（ContainerRuntime）三、插件（Add
Kubernetes——组件窒息う Kubernetes kubernetes 容器
文章目录K8S的优势核心架构角色与功能集群图例K8S的优势能管理大量跨主机容器快速部署应用快速扩展应用无缝对接新的应用节省资源，优化硬件资源的使用核心架构master（管理节点）node（计算节点）images（镜像节点）角色与功能Master功能提供集群的控制对集群进行全局决策检测和响应集群事件Master节点核心组件APIServer是整个系统的对外接口，提供客户端和其他组件调用后端元数据存储
Kubernetes组件汉只只网络 docker 大数据分布式 hadoop
Kubernetes核心组件Kubernetes定义了一组构建块，它们可以共同提供部署、维护和扩展应用程序的机制。组成Kubernetes的组件设计为松耦合和可扩展的，这样可以满足多种不同的工作负载。可扩展性在很大程度上由KubernetesAPI提供——它被作为扩展的内部组件以及Kubernetes上运行的容器等使用。Kubernetes主要由以下几个核心组件组成：etcd保存了整个集群的状态；
【Linux 从基础到进阶】Kubernetes 集群搭建与管理爱技术的小伙子 Linux从基础到进阶 linux kubernetes 运维
Kubernetes集群搭建与管理Kubernetes（简称K8s）是一个用于自动化部署、扩展和管理容器化应用程序的开源平台。它提供了容器编排功能，能够管理大量的容器实例，并支持应用的自动扩展、高可用性和自愈能力。本文将详细介绍如何在CentOS和Ubuntu系统上安装和配置Kubernetes集群，并讲解Kubernetes的基本概念和管理操作。1.Kubernetes基础概念在了解如何搭建Ku
Spark集群的三种模式 MelodyYN #Spark spark hadoop big data
文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
【Kubernetes】常见面试题汇总（十三） summer.335 Kubernetes kubernetes 容器云原生
目录39.简述KubernetesScheduler使用哪两种算法将Pod绑定到worker节点？40.简述Kuberneteskubelet的作用？41.简述Kuberneteskubelet监控Worker节点资源是使用什么组件来实现的？39.简述KubernetesScheduler使用哪两种算法将Pod绑定到worker节点？KubernetesScheduler根据如下两种调度算法将Po
Java中的大数据处理框架对比分析省赚客app开发者 java 开发语言
Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
Ansible自动化部署kubernetes集群 theo.wu kubernetes ansible 自动化
机器环境介绍1.1.机器信息介绍IPhostnameapplicationCPUMemory192.168.204.129k8s-master01etcd，kube-apiserver，kube-controller-manager，kube-scheduler,kubelet,kube-proxy,containerd2C4G192.168.204.130k8s-worker01etcd，kub
写出渗透测试信息收集详细流程卿酌南烛_b805
一、扫描域名漏洞：域名漏洞扫描工具有AWVS、APPSCAN、Netspark、WebInspect、Nmap、Nessus、天镜、明鉴、WVSS、RSAS等。二、子域名探测：1、dns域传送漏洞2、搜索引擎查找（通过Google、bing、搜索c段）3、通过ssl证书查询网站：https://myssl.com/ssl.html和https://www.chinassl.net/ssltools
K8S - Emptydir - 取代ELK 使用fluentd 构建logging saidcar nvd11 K8S kubernetes
由于k8s的无状态service通常部署在多个POD中，实现多实例面向高并发。但是k8s本身并没有提供集中查询多个pod的日志的功能其中1个常见方案就是ELK.本文的方案是利用fluentdsidecar和emptydir把多个pod的日志导向到bigquery的table中。Emptydir的简介Kubernetes中的EmptyDir是一种用于容器之间共享临时存储的空目录卷类型。EmptyDi
Spark MLlib模型训练—推荐算法 ALS(Alternative Least Squares) 不二人生 Spark ML 实战 spark-ml 推荐算法算法
SparkMLlib模型训练—推荐算法ALS(AlternativeLeastSquares)如果你平时爱刷抖音，或者热衷看电影，不知道有没有过这样的体验：这类影视App你用得越久，它就好像会读心术一样，总能给你推荐对胃口的内容。其实这种迎合用户喜好的推荐，离不开机器学习中的推荐算法。在今天这一讲，我们就结合两个有趣的电影推荐场景，为你讲解SparkMLlib支持的协同过滤与频繁项集算法电影推荐场
Java实现的简单双向Map，支持重复Value superlxw1234 java 双向map
关键字：Java双向Map、DualHashBidiMap 有个需求，需要根据即时修改Map结构中的Value值，比如，将Map中所有value=V1的记录改成value=V2，key保持不变。数据量比较大，遍历Map性能太差，这就需要根据Value先找到Key，然后去修改。即：既要根据Key找Value，又要根据Value
PL/SQL触发器基础及例子百合不是茶 oracle数据库触发器 PL/SQL编程
触发器的简介; 触发器的定义就是说某个条件成立的时候，触发器里面所定义的语句就会被自动的执行。因此触发器不需要人为的去调用，也不能调用。触发器和过程函数类似过程函数必须要调用, 一个表中最多只能有12个触发器类型的,触发器和过程函数相似触发器不需要调用直接执行, 触发时间：指明触发器何时执行，该值可取： before：表示在数据库动作之前触发
[时空与探索]穿越时空的一些问题 comsci 问题
我们还没有进行过任何数学形式上的证明,仅仅是一个猜想..... 这个猜想就是; 任何有质量的物体(哪怕只有一微克)都不可能穿越时空,该物体强行穿越时空的时候,物体的质量会与时空粒子产生反应,物体会变成暗物质,也就是说,任何物体穿越时空会变成暗物质..(暗物质就我的理
easy ui datagrid上移下移一行商人shang js 上移下移 easyui datagrid
/** * 向上移动一行 * * @param dg * @param row */ function moveupRow(dg, row) { var datagrid = $(dg); var index = datagrid.datagrid("getRowIndex", row); if (isFirstRow(dg, row)) {
Java反射 oloz 反射
本人菜鸟，今天恰好有时间，写写博客，总结复习一下java反射方面的知识，欢迎大家探讨交流学习指教首先看看java中的Class package demo; public class ClassTest { /*先了解java中的Class*/ public static void main(String[] args) { //任何一个类都
springMVC 使用JSR-303 Validation验证杨白白 spring mvc
JSR-303是一个数据验证的规范，但是spring并没有对其进行实现，Hibernate Validator是实现了这一规范的，通过此这个实现来讲SpringMVC对JSR-303的支持。 JSR-303的校验是基于注解的，首先要把这些注解标记在需要验证的实体类的属性上或是其对应的get方法上。登录需要验证类 public class Login { @NotEmpty
log4j 香水浓 log4j
log4j.rootCategory=DEBUG, STDOUT, DAILYFILE, HTML, DATABASE #log4j.rootCategory=DEBUG, STDOUT, DAILYFILE, ROLLINGFILE, HTML #console log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender log4
使用ajax和history.pushState无刷新改变页面URL agevs jquery 框架 Ajax html5 chrome
表现如果你使用chrome或者firefox等浏览器访问本博客、github.com、plus.google.com等网站时，细心的你会发现页面之间的点击是通过ajax异步请求的，同时页面的URL发生了了改变。并且能够很好的支持浏览器前进和后退。是什么有这么强大的功能呢？ HTML5里引用了新的API，history.pushState和history.replaceState，就是通过
centos中文乱码 AILIKES centos OS ssh
一、CentOS系统访问 g.cn ，发现中文乱码。于是用以前的方式：yum -y install fonts-chinese CentOS系统安装后，还是不能显示中文字体。我使用 gedit 编辑源码，其中文注释也为乱码。后来，终于找到以下方法可以解决，需要两个中文支持的包： fonts-chinese-3.02-12.
触发器 baalwolf 触发器
触发器(trigger)：监视某种情况，并触发某种操作。触发器创建语法四要素：1.监视地点(table) 2.监视事件(insert/update/delete) 3.触发时间(after/before) 4.触发事件(insert/update/delete) 语法： create trigger triggerName after/before
JS正则表达式的i m g bijian1013 JavaScript 正则表达式
g:表示全局（global)模式，即模式将被应用于所有字符串，而非在发现第一个匹配项时立即停止。 i:表示不区分大小写（case-insensitive）模式，即在确定匹配项时忽略模式与字符串的大小写。 m:表示
HTML5模式和Hashbang模式 bijian1013 JavaScript AngularJS Hashbang模式 HTML5模式
我们可以用$locationProvider来配置$location服务（可以采用注入的方式，就像AngularJS中其他所有东西一样）。这里provider的两个参数很有意思，介绍如下。 html5Mode 一个布尔值，标识$location服务是否运行在HTML5模式下。 ha
[Maven学习笔记六]Maven生命周期 bit1129 maven
从mvn test的输出开始说起当我们在user-core中执行mvn test时，执行的输出如下： /software/devsoftware/jdk1.7.0_55/bin/java -Dmaven.home=/software/devsoftware/apache-maven-3.2.1 -Dclassworlds.conf=/software/devs
【Hadoop七】基于Yarn的Hadoop Map Reduce容错 bit1129 hadoop
运行于Yarn的Map Reduce作业，可能发生失败的点包括 Task Failure Application Master Failure Node Manager Failure Resource Manager Failure 1. Task Failure 任务执行过程中产生的异常和JVM的意外终止会汇报给Application Master。僵死的任务也会被A
记一次数据推送的异常解决端口解决 ronin47 记一次数据推送的异常解决
　　需求：从db获取数据然后推送到B 程序开发完成，上jboss,刚开始报了很多错，逐一解决，可最后显示连接不到数据库。机房的同事说可以ping 通。　　自已画了个图，逐一排除，把linux 防火墙　和　setenforce　设置最低。　　　service iptables stop
巧用视错觉-UI更有趣 brotherlamp UI ui视频 ui教程 ui自学 ui资料
我们每个人在生活中都曾感受过视错觉（optical illusion）的魅力。视错觉现象是双眼跟我们开的一个玩笑，而我们往往还心甘情愿地接受我们看到的假象。其实不止如此，视觉错现象的背后还有一个重要的科学原理——格式塔原理。格式塔原理解释了人们如何以视觉方式感觉物体，以及图像的结构，视角，大小等要素是如何影响我们的视觉的。在下面这篇文章中，我们首先会简单介绍一下格式塔原理中的基本概念，
线段树-poj1177-N个矩形求边长（离散化+扫描线） bylijinnan 数据结构算法线段树
package com.ljn.base; import java.util.Arrays; import java.util.Comparator; import java.util.Set; import java.util.TreeSet; /** * POJ 1177 (线段树+离散化+扫描线)，题目链接为http://poj.org/problem?id=1177
HTTP协议详解 chicony http协议
引言
Scala设计模式 chenchao051 设计模式 scala
Scala设计模式我的话：在国外网站上看到一篇文章，里面详细描述了很多设计模式，并且用Java及Scala两种语言描述，清晰的让我们看到各种常规的设计模式，在Scala中是如何在语言特性层面直接支持的。基于文章很nice，我利用今天的空闲时间将其翻译，希望大家能一起学习，讨论。翻译
安装mysql daizj mysql 安装
安装mysql (1)删除linux上已经安装的mysql相关库信息。rpm -e xxxxxxx --nodeps (强制删除) 执行命令rpm -qa |grep mysql 检查是否删除干净 (2)执行命令 rpm -i MySQL-server-5.5.31-2.el
HTTP状态码大全 dcj3sjt126com http状态码
完整的 HTTP 1.1规范说明书来自于RFC 2616，你可以在http://www.talentdigger.cn/home/link.php?url=d3d3LnJmYy1lZGl0b3Iub3JnLw%3D%3D在线查阅。HTTP 1.1的状态码被标记为新特性，因为许多浏览器只支持 HTTP 1.0。你应只把状态码发送给支持 HTTP 1.1的客户端，支持协议版本可以通过调用request
asihttprequest上传图片 dcj3sjt126com ASIHTTPRequest
NSURL *url =@"yourURL"; ASIFormDataRequest*currentRequest =[ASIFormDataRequest requestWithURL:url]; [currentRequest setPostFormat:ASIMultipartFormDataPostFormat];[currentRequest se
C语言中，关键字static的作用 e200702084 C++c C#
在C语言中，关键字static有三个明显的作用： 1)在函数体，局部的static变量。生存期为程序的整个生命周期，（它存活多长时间）；作用域却在函数体内（它在什么地方能被访问（空间））。一个被声明为静态的变量在这一函数被调用过程中维持其值不变。因为它分配在静态存储区，函数调用结束后并不释放单元，但是在其它的作用域的无法访问。当再次调用这个函数时，这个局部的静态变量还存活，而且用在它的访
win7/8使用curl geeksun win7
1. WIN7/8下要使用curl，需要下载curl-7.20.0-win64-ssl-sspi.zip和Win64OpenSSL_Light-1_0_2d.exe。下载地址： http://curl.haxx.se/download.html 请选择不带SSL的版本，否则还需要安装SSL的支持包 2. 可以给Windows增加c
Creating a Shared Repository; Users Sharing The Repository hongtoushizi git
转载自： http://www.gitguys.com/topics/creating-a-shared-repository-users-sharing-the-repository/ Commands discussed in this section: git init –bare git clone git remote git pull git p
Java实现字符串反转的8种或9种方法 Josh_Persistence 异或反转递归反转二分交换反转 java字符串反转栈反转
注：对于第7种使用异或的方式来实现字符串的反转，如果不太看得明白的，可以参照另一篇博客： http://josh-persistence.iteye.com/blog/2205768 /** * */ package com.wsheng.aggregator.algorithm.string; import java.util.Stack; /**
代码实现任意容量倒水问题 home198979 PHP 算法倒水
形象化设计模式实战 HELLO!架构 redis命令源码解析倒水问题：有两个杯子，一个A升，一个B升，水有无限多，现要求利用这两杯子装C
Druid datasource zhb8015 druid
推荐大家使用数据库连接池 DruidDataSource. http://code.alibabatech.com/wiki/display/Druid/DruidDataSource DruidDataSource经过阿里巴巴数百个应用一年多生产环境运行验证，稳定可靠。它最重要的特点是：监控、扩展和性能。下载和Maven配置看这里： http
两种启动监听器ApplicationListener和ServletContextListener spjich java spring 框架
引言:有时候需要在项目初始化的时候进行一系列工作，比如初始化一个线程池，初始化配置文件，初始化缓存等等，这时候就需要用到启动监听器，下面分别介绍一下两种常用的项目启动监听器 ServletContextListener 特点: 依赖于sevlet容器，需要配置web.xml 使用方法: public class StartListener implements
JavaScript Rounding Methods of the Math object 何不笑 JavaScript Math
The next group of methods has to do with rounding decimal values into integers. Three methods — Math.ceil(), Math.floor(), and Math.round() — handle rounding in differen