CDH6.3.2集群环境部署Flink-1.15.0

Flink1.15.0安装部署

1.安装准备

1.1环境介绍

​ 本文档部署环境：CDH6.3.2集群、jdk1.8

1.2 Flink安装包

• CDH集群已经安装
• Flink1.15.0安装包
• Flink on yarn需要的Hadoopjar包
  • 登录Maven Repository下载jar包flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0

2.安装包解压和修改配置

2.1 环境变量配置

[root@cdh02 ~]# vim /etc/profile
#jdk1.8
export JAVA_HOME=/usr/local/java/jdk1.8.0_161
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
#scala
export SCALA_HOME="$PATH:/usr/local/scala/scala-2.12.8"
#Hadoop
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/usr/data/conf/hadoop
export HADOOP_CLASSPATH=$HADOOP_HOME/lib/*.jar
#Flinlk
export FLINK_HOME=/opt/shealphy/flink-1.15.0
#ES
export ES_HOME=/usr/local/elasticsearch-6.8.8

export PATH=$PATH:$SCALA_HOME/bin:$ES_HOME/bin:$FLINK_HOME/bin:$HADOOP_HOME/bin

2.2 Flink的配置

• 本次安装使用的是Flink on Yarn模式，无需配置master和worker，yarn自行进行调度

  • 拷贝flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar到flink的lib下

    [root@cdh02 shealphy]# scp -r ./flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar ./flink-1.15.0/lib/
    

3.启动Flink

3.1 启动yarn-session

[root@cdh02 ~]# cd /opt/shealphy/flink-1.15.0/bin
[root@cdh02 bin]# ./yarn-session.sh -jm 1024 -tm 1024 -s 2 -d

• 启动报错

  1. ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli

     ...
     2022-05-09 18:03:22,001 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: yarn.application-attempts, 10
     2022-05-09 18:03:22,001 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: fs.overwrite-files, true
     2022-05-09 18:03:22,002 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: fs.output.always-create-directory, true
     2022-05-09 18:03:22,004 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Error while running the Flink session.
     java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:230) ~[flink-dist-1.15.0.jar:1.15.0]
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:156) ~[flink-dist-1.15.0.jar:1.15.0]
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851) [flink-dist-1.15.0.jar:1.15.0]
     
     ------------------------------------------------------------
      The program finished with the following exception:
     
     java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:230)
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:156)
             at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851)
     

     • 报错分析：缺少org.apache.commons.cli相关的jar或者jar包版本不一致

       • 解决1,从集群环境中拷贝对应jar包

         # 拷贝CDH6.3.2集群hadoop lib下的commons-cli-1.2.jar包到flink lib下
         # 重新启动flink-session依旧报错
         

       • 解决2，从Maven仓库下载高版本jar包commons-cli-1.4.jar

         # 拷贝commons-cli-1.4.jar到flink lib下
         # 重新启动flink-session,启动成功
         
         2022-05-09 18:44:28,961 WARN  org.apache.hadoop.util.NativeCodeLoader                      [] - Unable to load nativ                          e-hadoop library for your platform... using builtin-java classes where applicable
         2022-05-09 18:44:31,990 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Removing 'localhost'                           Key: 'jobmanager.bind-host' , default: null (fallback keys: []) setting from effective configuration; using '0.0.0.                          0' instead.
         2022-05-09 18:44:31,991 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Removing 'localhost'                           Key: 'taskmanager.bind-host' , default: null (fallback keys: []) setting from effective configuration; using '0.0.0                          .0' instead.
         2022-05-09 18:44:32,032 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from f                          raction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), mi                          n value will be used instead
         2022-05-09 18:44:32,042 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting applicati                          on master application_1651710719941_0001
         2022-05-09 18:44:32,430 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted applicatio                          n application_1651710719941_0001
         2022-05-09 18:44:32,430 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the clus                          ter to be allocated
         2022-05-09 18:44:32,432 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, c                          urrent state ACCEPTED
         2022-05-09 18:44:38,569 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - YARN application has                           been deployed successfully.
         2022-05-09 18:44:38,570 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface                           cdh02:39198 of application 'application_1651710719941_0001'.
         JobManager Web Interface: http://cdh02:39198
         

3.2 flink启动

• 有两种模式：session 和 per-job

  这里选用session模式

3.2.1 session方式启动demo

• 必须在active状态下的namenode上进行

  [root@cdh02 flink-1.15.0]# cd /opt/shealphy/flink-1.15.0
  [root@cdh02 flink-1.15.0]# ./bin/flink run examples/batch/WordCount.jar
  Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
  2022-05-09 18:59:10,026 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
  2022-05-09 18:59:10,026 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
  Executing WordCount example with default input data set.
  Use --input to specify file input.
  Printing result to stdout. Use --output to specify output path.
  2022-05-09 18:59:10,393 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/shealphy/flink-1.15.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
  2022-05-09 18:59:10,436 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at cdh02/172.16.0.46:8032
  2022-05-09 18:59:10,534 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
  2022-05-09 18:59:10,595 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface cdh02:39198 of application 'application_1651710719941_0001'.
  Job has been submitted with JobID 50113f5b0efa88b65f68ad713a7a84fb
  Program execution finished
  Job with JobID 50113f5b0efa88b65f68ad713a7a84fb has finished.
  Job Runtime: 10265 ms
  Accumulator Results:
  - 0d9904e571e7cf2f06480ef5dd4b3c99 (java.util.ArrayList) [170 elements]
  (a,5)
  (action,1)
  (after,1)
  (against,1)
  (all,2)
  (and,12)
  (arms,1)
  (arrows,1)
  (awry,1)
  ...
  

3.2.2 per-job方式启动demo

• 启动命令

  [root@cdh02 flink-1.15.0]# ./bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar