编译Flink 1.9.0

闲来无事，编个Flink 1.9 玩玩

1、下载flink、flink-shaded 源码、解压

　　flink 　　flink-shaded 7.0

[venn@venn release]$ ll
total 19880
drwxrwxr-x. 44 venn venn     4096 Sep 12 14:00 flink-release-1.9.0
drwxrwxr-x. 13 venn venn     4096 Sep 11 11:05 flink-shaded-release-7.0
-rw-rw-r--.  1 venn venn 20318507 Sep 12 09:24 release-1.9.0.tar.gz
-rw-rw-r--.  1 venn venn    28196 Sep 11 11:05 release-7.0.tar.gz

 

最开始直接拿flink 的 releast-1.9分支，发现都带了SNAPSHOT，遂放弃

flink-shaded 包含flink 的很多依赖，比如 flink-shaded-hadoop-2，中央仓库只提供了几个hadoop 版本的包，可能没有与自己hadoop 对应的 flink-shaded-hadoop-2 的包。flink1.9 对应的flink-shaded 版本是 7.0

            org.apache.flink
            flink-shaded-hadoop-2
            ${hadoop.version}-${flink.shaded.version}
            true
        

 

我用的hadoop 版本是2.7.7，然而并没有（2.7.5应该可以用）。

2、编译flink-shaded

指定hadoop版本2.7.7， 把编译好的包打到本地maven仓库，后面编译flink 用

[venn@venn flink-shaded-release-7.0]$ mvn clean install -DskipTest -Dhadoop.version=2.7.7


[INFO] Reactor Summary:
[INFO] 
[INFO] flink-shaded 7.0 ................................... SUCCESS [  3.483 s]
[INFO] flink-shaded-force-shading 7.0 ..................... SUCCESS [  0.597 s]
[INFO] flink-shaded-asm-6 6.2.1-7.0 ....................... SUCCESS [  0.627 s]
[INFO] flink-shaded-guava-18 18.0-7.0 ..................... SUCCESS [  0.994 s]
[INFO] flink-shaded-netty-4 4.1.32.Final-7.0 .............. SUCCESS [  3.469 s]
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-7.0 SUCCESS [  0.572 s]
[INFO] flink-shaded-jackson-parent 2.9.8-7.0 .............. SUCCESS [  0.035 s]
[INFO] flink-shaded-jackson-2 2.9.8-7.0 ................... SUCCESS [  1.479 s]
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-7.0 . SUCCESS [  0.857 s]
[INFO] flink-shaded-hadoop-2 2.7.7-7.0 .................... SUCCESS [  9.132 s]
[INFO] flink-shaded-hadoop-2-uber 2.7.7-7.0 ............... SUCCESS [ 17.173 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  38.612 s
[INFO] Finished at: 2019-09-12T15:14:42+08:00
[INFO] ------------------------------------------------------------------------

3、编译 flink

指定hadoop 版本 2.7.7

 

[venn@venn flink-release-1.9.0]$ mvn clean package -DskipTests -Dhadoop.version=2.7.7

[INFO] Reactor Summary for flink 1.9.0:
[INFO] 
[INFO] force-shading ...................................... SUCCESS [  1.281 s]
[INFO] flink .............................................. SUCCESS [  1.829 s]
[INFO] flink-annotations .................................. SUCCESS [  1.280 s]
[INFO] flink-shaded-curator ............................... SUCCESS [  1.501 s]
[INFO] flink-metrics ...................................... SUCCESS [  0.188 s]
[INFO] flink-metrics-core ................................. SUCCESS [  0.736 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.157 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  0.928 s]
[INFO] flink-core ......................................... SUCCESS [ 33.771 s]
[INFO] flink-java ......................................... SUCCESS [  4.827 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.100 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  0.643 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.082 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [  1.168 s]
[INFO] flink-runtime ...................................... SUCCESS [01:58 min]
[INFO] flink-scala ........................................ SUCCESS [ 29.960 s]
[INFO] flink-mapr-fs ...................................... SUCCESS [  0.394 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [  2.916 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [  5.740 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [  8.235 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [ 12.510 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 13.618 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [  4.483 s]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [  5.729 s]
[INFO] flink-optimizer .................................... SUCCESS [ 11.589 s]
[INFO] flink-clients ...................................... SUCCESS [  1.179 s]
[INFO] flink-streaming-java ............................... SUCCESS [  8.882 s]
[INFO] flink-test-utils ................................... SUCCESS [  1.453 s]
[INFO] flink-runtime-web .................................. SUCCESS [02:06 min]
[INFO] flink-examples ..................................... SUCCESS [  0.112 s]
[INFO] flink-examples-batch ............................... SUCCESS [ 10.577 s]
[INFO] flink-connectors ................................... SUCCESS [  0.096 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [  4.225 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.070 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [  1.102 s]
[INFO] flink-tests ........................................ SUCCESS [ 28.820 s]
[INFO] flink-streaming-scala .............................. SUCCESS [ 26.143 s]
[INFO] flink-table ........................................ SUCCESS [  0.054 s]
[INFO] flink-table-common ................................. SUCCESS [  1.771 s]
[INFO] flink-table-api-java ............................... SUCCESS [  1.416 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  0.570 s]
[INFO] flink-table-api-scala .............................. SUCCESS [  5.054 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [  7.626 s]
[INFO] flink-sql-parser ................................... SUCCESS [  3.073 s]
[INFO] flink-libraries .................................... SUCCESS [  0.056 s]
[INFO] flink-cep .......................................... SUCCESS [  2.437 s]
[INFO] flink-table-planner ................................ SUCCESS [01:37 min]
[INFO] flink-orc .......................................... SUCCESS [  0.579 s]
[INFO] flink-jdbc ......................................... SUCCESS [  0.546 s]
[INFO] flink-table-runtime-blink .......................... SUCCESS [  3.868 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [02:02 min]
[INFO] flink-hbase ........................................ SUCCESS [  1.725 s]
[INFO] flink-hcatalog ..................................... SUCCESS [  4.158 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [  0.272 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [  1.671 s]
[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [  0.768 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  0.547 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  0.692 s]
[INFO] flink-formats ...................................... SUCCESS [  0.054 s]
[INFO] flink-json ......................................... SUCCESS [  0.332 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [  0.810 s]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [  8.239 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [  9.715 s]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [  0.747 s]
[INFO] flink-csv .......................................... SUCCESS [  0.251 s]
[INFO] flink-connector-hive ............................... SUCCESS [  1.942 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [  0.332 s]
[INFO] flink-connector-twitter ............................ SUCCESS [  1.304 s]
[INFO] flink-connector-nifi ............................... SUCCESS [  0.349 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [  2.205 s]
[INFO] flink-avro ......................................... SUCCESS [  1.565 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [  0.861 s]
[INFO] flink-connector-kafka .............................. SUCCESS [  0.973 s]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [  0.582 s]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [  4.583 s]
[INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [  0.256 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  0.312 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  0.413 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  0.641 s]
[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [  0.700 s]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [  0.843 s]
[INFO] flink-parquet ...................................... SUCCESS [  0.726 s]
[INFO] flink-sequence-file ................................ SUCCESS [  0.247 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 10.970 s]
[INFO] flink-examples-table ............................... SUCCESS [  5.554 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.069 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  0.464 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  0.298 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [  3.640 s]
[INFO] flink-container .................................... SUCCESS [  0.304 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  0.555 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.056 s]
[INFO] flink-cli-test ..................................... SUCCESS [  0.151 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  0.154 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.098 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  0.153 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  0.141 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  1.119 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  0.136 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  0.183 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  0.233 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.138 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [  5.724 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  0.636 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [  1.688 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.221 s]
[INFO] flink-elasticsearch2-test .......................... SUCCESS [  3.299 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [  3.969 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [  2.275 s]
[INFO] flink-quickstart ................................... SUCCESS [  0.781 s]
[INFO] flink-quickstart-java .............................. SUCCESS [  0.350 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [  0.145 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  0.323 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  1.202 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  3.172 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  0.671 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  0.221 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  0.822 s]
[INFO] flink-e2e-test-utils ............................... SUCCESS [  6.170 s]
[INFO] flink-mesos ........................................ SUCCESS [ 17.591 s]
[INFO] flink-yarn ......................................... SUCCESS [  0.915 s]
[INFO] flink-gelly ........................................ SUCCESS [  2.601 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 13.500 s]
[INFO] flink-gelly-examples ............................... SUCCESS [  7.711 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [  0.193 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  0.123 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [  0.594 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [  0.371 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [  0.179 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  0.237 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.157 s]
[INFO] flink-cep-scala .................................... SUCCESS [  9.399 s]
[INFO] flink-table-uber ................................... SUCCESS [  1.794 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  2.050 s]
[INFO] flink-sql-client ................................... SUCCESS [  1.090 s]
[INFO] flink-state-processor-api .......................... SUCCESS [  0.598 s]
[INFO] flink-python ....................................... SUCCESS [  0.621 s]
[INFO] flink-scala-shell .................................. SUCCESS [  8.821 s]
[INFO] flink-dist ......................................... SUCCESS [ 10.765 s]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  0.285 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.153 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.132 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [  6.045 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  0.516 s]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.192 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [  5.923 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [  5.786 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [  5.000 s]
[INFO] flink-plugins-test ................................. SUCCESS [  0.095 s]
[INFO] flink-tpch-test .................................... SUCCESS [  1.210 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.052 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  0.258 s]
[INFO] flink-yarn-tests ................................... SUCCESS [  2.919 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  0.330 s]
[INFO] flink-docs ......................................... SUCCESS [  0.531 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.052 s]
[INFO] flink-ml-api ....................................... SUCCESS [  0.284 s]
[INFO] flink-ml-lib ....................................... SUCCESS [  0.164 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:33 min
[INFO] Finished at: 2019-09-12T17:41:19+08:00
[INFO] ------------------------------------------------------------------------

正常情况耗时15分钟，异常是可能会卡在 flink-runtime和 flink-runtime-web 上面

 

4、遇到的问题

1)、　flink 1.9 中移除了 map-r 的包，所有不用在这上面耗时间，但是我在编译的时候，还是遇到很多包无法下载（阿里云的镜像都有），只好在本地下载，上传上去（centos 的虚拟机里编译的，虚拟机下不下的包，本地都可以正常下载，就把本地的仓库覆盖上去了）

2)、 flink-runtime/flink-runtime-web 有时候会编译很久

3)、编译flink-table-api-java 的时候编译不过

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project flink-table-api-java: Compilation failure
[ERROR] /home/venn/git/release/flink-release-1.9.0/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ut
ils/factories/AggregateOperationFactory.java:[550,53] unreported exception X; must be caught or declared to be thrown
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :flink-table-api-java

是低版本jdk的bug，下面这种用法有点问题

List aliases = children.subList(1, children.size())
                .stream()
                .map(alias -> ExpressionUtils.extractValue(alias, String.class)
                    .orElseThrow(() -> new ValidationException("Unexpected alias: " + alias)))
                .collect(toList());

解决方法： 升级jdk 1.8 的小版本， 从jdk1.8.0_91 升到 jdk1.8.0_111 解决了

 4)、可以预装个 gcc、gcc-c++，中间会有报错，不过应该不影响

转载于:https://www.cnblogs.com/Springmoon-venn/p/11512136.html