cdh5.14.2版flink1.10编译过程

 

一、编译环境

1、Java（至少1.8以上）

#java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

2、Maven(至少Maven 3以上版本）

# mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /opt/maven-3.6.3
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /opt/jdk1.8.0_181/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-957.5.1.el7.x86_64", arch: "amd64", family: "unix"

3、安装node

1）下载软件：https://nodejs.org/dist/latest-v10.x/node-v10.19.0-linux-x64.tar.gz

2）解压：tar -xvf node-v10.19.0-linux-x64.tar.gz

4、编辑/etc/profile文件

vi /etc/profile

export JAVA_HOME=/opt/jdk1.8.0_181
export MAVEN_HOME=/opt/maven-3.6.3
export SCALA_HOME=/opt/scala-2.11.12
export NODE_HOME=/opt/node-v10.9.0-linux-x64
export PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$NODE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile

5、验证：

java -version
node -v
npm -v

3、最好将\opt\maven-3.6.3\conf\settings.xml中的镜像仓库改成国内的镜像仓库（其中有一些并没有用到，有备无患）

    
        alimaven
        central
        aliyun maven
        http://maven.aliyun.com/nexus/content/repositories/central/
    
    
        alimaven
        aliyun maven
        http://maven.aliyun.com/nexus/content/groups/public/
        central
    
    
        central
        Maven Repository Switchboard
        http://repo1.maven.org/maven2/
        central
    
    
        repo2
        central
        Human Readable Name for this Mirror.
        http://repo2.maven.org/maven2/
    
    
        ibiblio
        central
        Human Readable Name for this Mirror.
        http://mirrors.ibiblio.org/pub/mirrors/maven2/
    
    
        jboss-public-repository-group
        central
        JBoss Public Repository Group
        http://repository.jboss.org/nexus/content/groups/public
    
    
        google-maven-central
        Google Maven Central
        https://maven-central.storage.googleapis.com
        
        central
    
    
        maven.net.cn
        oneof the central mirrors in china
        http://maven.net.cn/content/groups/public/
        central
    

二、下载编译

1、下载源码

Flink1.10：https://github.com/apache/flink/tree/release-1.10

Flink-shaded 9.0： https://github.com/apache/flink-shaded/tree/release-9.0

三、编译源码

1、编译Flink-shaded 9.0

1）修改flink-shaded项目中的pom.xml 这里修改是为了加入cdh 等中央仓库，否则编译对应版本可能找不到cdh 相关的包。把下面内容加入到 里面去。

    vendor-repos
    
        
            vendor-repos
        
    
    
    
        
        
            cloudera-releases
            https://repository.cloudera.com/artifactory/cloudera-repos
            
                true
            
            
                false
            
        
        
        
            HDPReleases
            HDP Releases
            https://repo.hortonworks.com/content/repositories/releases/
            false
            true
        
        
            HortonworksJettyHadoop
            HDP Jetty
            https://repo.hortonworks.com/content/repositories/jetty-hadoop
            false
            true
        
        
        
            mapr-releases
            https://repository.mapr.com/maven/
            false
            true
        
    

2）修改flink-shade项目下面的flink-shaded-hadoop-2-uber中的pom.xml增加依赖

  commons-cli

  commons-cli

  1.3.1

3）执行以下语句

mvn  -T2C clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.14.2 -Dscala-2.11 -Drat.skip=true

 经过漫长的等待后。终于编译成功了

[INFO] Reactor Summary:
[INFO]
[INFO] flink-shaded 9.0 ................................... SUCCESS [06:40 min]
[INFO] flink-shaded-force-shading 9.0 ..................... SUCCESS [ 11.616 s]
[INFO] flink-shaded-asm-7 7.1-9.0 ......................... SUCCESS [01:54 min]
[INFO] flink-shaded-guava-18 18.0-9.0 ..................... SUCCESS [01:56 min]
[INFO] flink-shaded-netty-4 4.1.39.Final-9.0 .............. SUCCESS [02:24 min]
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-9.0 SUCCESS [02:11 min]
[INFO] flink-shaded-jackson-parent 2.10.1-9.0 ............. SUCCESS [  0.030 s]
[INFO] flink-shaded-jackson-2 2.10.1-9.0 .................. SUCCESS [04:40 min]
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.10.1-9.0  SUCCESS [03:21 min]
[INFO] flink-shaded-hadoop-2 2.6.0-cdh5.14.2-9.0 .......... SUCCESS [44:50 min]
[INFO] flink-shaded-hadoop-2-uber 2.6.0-cdh5.14.2-9.0 ..... SUCCESS [17:25 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:10 h (Wall Clock)
[INFO] Finished at: 2020-05-25T04:03:32+01:00
[INFO] ------------------------------------------------------------------------

2、编译Flink1.10

mvn clean install -DskipTests -Dfast -Drat.skip=true -Dhaoop.version=2.6.0-cdh-5.14.2 -Pvendor-repos -Dinclude-hadoop -Dscala-2.11 -T2C
参数的含义：

# -Dfast  #在flink根目录下pom.xml文件中fast配置项目中含快速设置,其中包含了多项构建时的跳过参数. #例如apache的文件头(rat)合法校验，代码风格检查，javadoc生成的跳过等，详细可阅读pom.xml
# install maven的安装命令
# -T2C #支持多处理器或者处理器核数参数,加快构建速度,推荐Maven3.3及以上
# -Pinclude-hadoop  将 hadoop的 jar包，打入到lib/中
# -Pvendor-repos   # 使用cdh、hdp 的hadoop 需要添加该参数
# -Dscala-2.11     # 指定scala的版本为2.11
# -Dhadoop.version=2.6.0-cdh5.14.2  指定 hadoop 的版本
 

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for flink 1.10-SNAPSHOT:
[INFO]
[INFO] force-shading ...................................... SUCCESS [  3.835 s]
[INFO] flink .............................................. SUCCESS [  4.660 s]
[INFO] flink-annotations .................................. SUCCESS [  5.075 s]
[INFO] flink-shaded-curator ............................... SUCCESS [  5.081 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.671 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  5.557 s]
[INFO] flink-metrics ...................................... SUCCESS [  0.458 s]
[INFO] flink-metrics-core ................................. SUCCESS [  2.031 s]
[INFO] flink-core ......................................... SUCCESS [ 32.645 s]
[INFO] flink-java ......................................... SUCCESS [ 16.242 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.276 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  6.292 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.170 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [  6.300 s]
[INFO] flink-runtime ...................................... SUCCESS [01:36 min]
[INFO] flink-scala ........................................ SUCCESS [02:19 min]
[INFO] flink-mapr-fs ...................................... SUCCESS [  1.412 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [ 12.953 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [  4.598 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 17.638 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [ 19.634 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 50.206 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [ 20.926 s]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [ 33.799 s]
[INFO] flink-optimizer .................................... SUCCESS [  7.398 s]
[INFO] flink-clients ...................................... SUCCESS [  4.458 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 33.167 s]
[INFO] flink-test-utils ................................... SUCCESS [  3.432 s]
[INFO] flink-runtime-web .................................. SUCCESS [06:37 min]
[INFO] flink-examples ..................................... SUCCESS [  0.377 s]
[INFO] flink-examples-batch ............................... SUCCESS [01:05 min]
[INFO] flink-connectors ................................... SUCCESS [  0.088 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [ 47.164 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.696 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [  4.058 s]
[INFO] flink-tests ........................................ SUCCESS [ 56.605 s]
[INFO] flink-streaming-scala .............................. SUCCESS [01:30 min]
[INFO] flink-table ........................................ SUCCESS [  0.460 s]
[INFO] flink-table-common ................................. SUCCESS [ 18.391 s]
[INFO] flink-table-api-java ............................... SUCCESS [ 10.280 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  1.852 s]
[INFO] flink-table-api-scala .............................. SUCCESS [ 29.990 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 24.721 s]
[INFO] flink-sql-parser ................................... SUCCESS [ 22.475 s]
[INFO] flink-libraries .................................... SUCCESS [  0.070 s]
[INFO] flink-cep .......................................... SUCCESS [ 14.019 s]
[INFO] flink-table-planner ................................ SUCCESS [05:52 min]
[INFO] flink-table-runtime-blink .......................... SUCCESS [ 51.079 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [08:17 min]
[INFO] flink-jdbc ......................................... SUCCESS [  2.468 s]
[INFO] flink-hbase ........................................ SUCCESS [ 26.271 s]
[INFO] flink-hcatalog ..................................... SUCCESS [ 34.867 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [  3.184 s]
[INFO] flink-formats ...................................... SUCCESS [  0.090 s]
[INFO] flink-json ......................................... SUCCESS [  2.348 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [  3.748 s]
[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [  2.762 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  1.554 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  2.544 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [  8.497 s]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 42.106 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 48.162 s]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [  8.181 s]
[INFO] flink-connector-elasticsearch7 ..................... SUCCESS [02:32 min]
[INFO] flink-orc .......................................... SUCCESS [  4.634 s]
[INFO] flink-csv .......................................... SUCCESS [  1.178 s]
[INFO] flink-connector-hive ............................... SUCCESS [ 25.955 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [  1.064 s]
[INFO] flink-connector-twitter ............................ SUCCESS [  5.725 s]
[INFO] flink-connector-nifi ............................... SUCCESS [  3.060 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [ 19.267 s]
[INFO] flink-avro ......................................... SUCCESS [ 12.074 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [ 10.436 s]
[INFO] flink-connector-kafka .............................. SUCCESS [  2.911 s]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [ 10.280 s]
[INFO] flink-connector-kinesis ............................ SUCCESS [ 22.412 s]
[INFO] flink-sql-connector-elasticsearch7 ................. SUCCESS [ 16.703 s]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [ 18.729 s]
[INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [  1.007 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  1.868 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  1.527 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  1.570 s]
[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [  2.024 s]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [  5.400 s]
[INFO] flink-parquet ...................................... SUCCESS [  5.223 s]
[INFO] flink-sequence-file ................................ SUCCESS [ 15.688 s]
[INFO] flink-compress ..................................... SUCCESS [  1.179 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 47.720 s]
[INFO] flink-examples-table ............................... SUCCESS [ 18.248 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.439 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  2.185 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  1.800 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [ 13.659 s]
[INFO] flink-container .................................... SUCCESS [  1.191 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  3.570 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.634 s]
[INFO] flink-cli-test ..................................... SUCCESS [  0.558 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  1.136 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.568 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  1.702 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  2.537 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  5.170 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  0.497 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  0.542 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  1.067 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.392 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [ 41.072 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  2.722 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [  6.929 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.867 s]
[INFO] flink-elasticsearch2-test .......................... SUCCESS [ 10.753 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [ 11.548 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [  7.820 s]
[INFO] flink-quickstart ................................... SUCCESS [  2.923 s]
[INFO] flink-quickstart-java .............................. SUCCESS [  0.489 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [  0.436 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  1.379 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  5.101 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  8.482 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  2.510 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  1.361 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  3.210 s]
[INFO] flink-rocksdb-state-memory-control-test ............ SUCCESS [  1.952 s]
[INFO] flink-mesos ........................................ SUCCESS [02:07 min]
[INFO] flink-kubernetes ................................... SUCCESS [ 14.511 s]
[INFO] flink-yarn ......................................... SUCCESS [  5.250 s]
[INFO] flink-gelly ........................................ SUCCESS [ 27.020 s]
[INFO] flink-gelly-scala .................................. SUCCESS [01:00 min]
[INFO] flink-gelly-examples ............................... SUCCESS [ 26.265 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [  1.158 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  0.565 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [  4.574 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [  1.652 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [  0.514 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  2.473 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.751 s]
[INFO] flink-cep-scala .................................... SUCCESS [ 27.445 s]
[INFO] flink-table-uber ................................... SUCCESS [  6.999 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  4.717 s]
[INFO] flink-sql-client ................................... SUCCESS [  4.681 s]
[INFO] flink-state-processor-api .......................... SUCCESS [  3.203 s]
[INFO] flink-python ....................................... SUCCESS [ 35.108 s]
[INFO] flink-scala-shell .................................. SUCCESS [ 50.659 s]
[INFO] flink-dist ......................................... SUCCESS [ 59.308 s]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  3.746 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.568 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.558 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [ 18.821 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  4.708 s]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.836 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [ 24.037 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [ 17.476 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [ 17.467 s]
[INFO] flink-plugins-test ................................. SUCCESS [  0.182 s]
[INFO] dummy-fs ........................................... SUCCESS [  0.569 s]
[INFO] another-dummy-fs ................................... SUCCESS [  0.825 s]
[INFO] flink-tpch-test .................................... SUCCESS [  5.672 s]
[INFO] flink-streaming-kinesis-test ....................... SUCCESS [ 31.441 s]
[INFO] flink-elasticsearch7-test .......................... SUCCESS [  6.095 s]
[INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [  4.853 s]
[INFO] flink-tpcds-test ................................... SUCCESS [  3.026 s]
[INFO] flink-statebackend-heap-spillable .................. SUCCESS [  1.629 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.154 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  1.695 s]
[INFO] flink-yarn-tests ................................... SUCCESS [  9.679 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  1.964 s]
[INFO] flink-docs ......................................... SUCCESS [  9.697 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.457 s]
[INFO] flink-ml-api ....................................... SUCCESS [  1.627 s]
[INFO] flink-ml-lib ....................................... SUCCESS [  3.041 s]
[INFO] flink-walkthroughs ................................. SUCCESS [  1.846 s]
[INFO] flink-walkthrough-common ........................... SUCCESS [  3.381 s]
[INFO] flink-walkthrough-table-java ....................... SUCCESS [  4.436 s]
[INFO] flink-walkthrough-table-scala ...................... SUCCESS [  1.016 s]
[INFO] flink-walkthrough-datastream-java .................. SUCCESS [  0.518 s]
[INFO] flink-walkthrough-datastream-scala ................. SUCCESS [  0.526 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  24:35 min (Wall Clock)
[INFO] Finished at: 2020-05-25T09:15:34+01:00
[INFO] ------------------------------------------------------------------------

编译成功之后，

在/opt/flink-release-1.10/flink-dist/target中找到编译后的jar包

[root@bigdata1 target]# ll
total 119164
drwxr-xr-x 2 root root        27 May 25 09:47 antrun
drwxr-xr-x 2 root root         6 May 25 09:47 archive-tmp
-rw-r--r-- 1 root root    501600 May 25 09:47 bash-java-utils.jar
drwxr-xr-x 3 root root        77 May 25 09:47 classes
drwxr-xr-x 3 root root        32 May 25 09:47 flink-1.10-SNAPSHOT-bin
-rw-r--r-- 1 root root 121305094 May 25 09:47 flink-dist_2.11-1.10-SNAPSHOT.jar
drwxr-xr-x 3 root root        29 May 25 09:47 generated-test-sources
drwxr-xr-x 2 root root        27 May 25 09:47 maven-archiver
drwxr-xr-x 3 root root        21 May 25 09:47 maven-shared-archive-resources
-rw-r--r-- 1 root root    211354 May 25 09:47 original-flink-dist_2.11-1.10-SNAPSHOT.jar
drwxr-xr-x 2 root root        68 May 25 09:47 temporary
drwxr-xr-x 4 root root        59 May 25 09:47 test-classes