Flink源码编译(Flink1.7.0+Hadoop2.6.0+CDH5.16.1)

Flink源码编译

一、概述

1. 源码编译文档
   . https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html
2. 源码下载地址
   . https://github.com/apache/flink/releases
3. 前置条件
   . Maven 3
   . JDK8
4. 编译命令(hadoop和cdh换成自己使用版本)
   . mvn clean install -DskipTests -Pvendor-repos -Dfast -Dhadoop.version=2.6.0-cdh5.16.1

二、编译

1. 打开window dos，进入flink的根目录
   . E:\wsd\res\flink\flink-release-1.7.0>
2. 执行编译命令
   . mvn clean install -DskipTests -Pvendor-repos -Dfast -Dhadoop.version=2.6.0-cdh5.16.1

三、问题

3.1 缺少kafka-schema-registry-client-3.3.1.jar包

[ERROR] Failed to execute goal on project flink-avro-confluent-registry: Could not resolve dependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.7.0: Could not transfer artifact io.confluent:kafka-schema-registry-client:jar:3.3.1 from/to confluent (http://packages.confluent.io/maven/): GET request of: io/confluent/kafka-schema-registry-client/3.3.1/kafka-schema-registry-client-3.3.1.jar from confluent failed: Read timed out -> [Help 1]

• 下载kafka-schema-registry-client-3.3.1.jar
• http://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/3.3.1/kafka-schema-registry-client-3.3.1.jar
• 安装到本地
• mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=3.3.1 -Dpackaging=jar -Dfile=E://wsd/res/bg/jar/kafka-schema-registry-client-3.3.1.jar
• 重新编译
• mvn clean install -DskipTests -Pvendor-repos -Dfast -Dhadoop.version=2.6.0-cdh5.16.1

3.2 hadoop-mapreduce-client-core-2.6.0-cdh5.16.1.jar(不能下载)

[ERROR] Failed to execute goal on project flink-shaded-hadoop2: Could not resolve dependencies for project org.apache.flink:flink-shaded-hadoop2:jar:1.7.0: Could not transfer artifact org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.16.1 from/to cloudera-releases (https://repository.cloudera.com/artifactory/cloudera-repos): GET request of: org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0-cdh5.16.1/hadoop-mapreduce-client-core-2.6.0-cdh5.16.1.jar from cloudera-releases failed: SSL peer shut down incorrectly -> [Help 1]

• 下载地址
• https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0-cdh5.16.1/
• 安装到本地
• mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-mapreduce-client-core -Dversion=2.6.0-cdh5.16.1 -Dpackaging=jar -Dfile=E://wsd/res/bg/jar/hadoop-mapreduce-client-core-2.6.0-cdh5.16.1.jar
• 重新编译
• mvn clean install -DskipTests -Pvendor-repos -Dfast -Dhadoop.version=2.6.0-cdh5.16.1

3.3 org.apache.hadoop:hadoop-yarn-server-tests:jar:2.6.0-cdh5.16.1

• mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-yarn-server-tests -Dversion=2.6.0-cdh5.16.1 -Dpackaging=jar -Dfile=E://wsd/res/bg/jar/hadoop-yarn-server-tests-2.6.0-cdh5.16.1.jar

3.4 hadoop-yarn-server-nodemanager

• mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-yarn-server-nodemanager -Dversion=2.6.0-cdh5.16.1 -Dpackaging=jar -Dfile=E://wsd/res/bg/jar/hadoop-yarn-server-nodemanager-2.6.0-cdh5.16.1.jar

3.5 org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13-cloudera.1

• mvn install:install-file -DgroupId=org.codehaus.jackson -DartifactId=jackson-mapper-asl -Dversion=1.9.13-cloudera.1 -Dpackaging=jar -Dfile=E://wsd/res/bg/jar/jackson-mapper-asl-1.8.10-cloudera.1.jar

四、编译成功

1. 输出目录(这个文件就是我们要部署到服务器上用的flink)

• E:\wsd\res\bg\flink-release-1.7.0\flink-dist\target\flink-1.7.0-bin\flink-1.7.0

五、总结

Flink的编译过程，简单来说，就是确保网络畅通的情况下，遇到不能下载的依赖jar包，手动下载下来安装到本地，再重新编译