Flink On yarn

1、 下载地址

https://archive.apache.org/dist/flink/

2、 解压flink

[root@master ~]#

tar -xzvf /chinaskills/flink-1.10.2-bin-scala_2.11.tgz -C /usr/local/src

3、重命名

[root@master ~]#

mv /usr/local/src/flink-1.10.2 /usr/local/src/flink

4、配置环境变量(当前用户)

[root@master ~]#

vi /root/.bash_profile

export FLINK_HOME=/usr/local/src/flink
export PATH=$PATH:$FLINK_HOME/bin

5、加载环境变量

[root@master ~]#

source  /root/.bash_profile

6、添加jar包

[root@master ~]#

cp /chinaskills/flink-shaded-hadoop-2-uber-2.7.5-10.0.jar /usr/local/src/flink/lib/

7、配置masters

[root@master ~]#

vi /usr/local/src/flink/conf/masters

配置内容:

master:8081

8、配置slaves

[root@master ~]#

vi /usr/local/src/flink/conf/slaves

配置内容:

master
slave1
slave2

9、配置flink-conf.yaml

[root@master ~]#

vi /usr/local/src/flink/conf/flink-conf.yaml

配置内容:

jobmanager.rpc.address: master

10、分发文件给slave1和slave2

[root@master ~]#

scp -r /usr/local/src/flink slave1:/usr/local/src/ 
scp -r /usr/local/src/flink slave2:/usr/local/src/ 
scp /root/.bash_profile  slave1:/root/
scp /root/.bash_profile  slave2:/root/

11、启动Flink集群(两种模式)

11.1 Session-cluster模式

*Session-Cluster模式需要先启动集群，然后再提交作业，接着会像yarn申请一定空间后，资源永远保持不变。如果资源满了，下一个作业就无法提交，只能等到yarn中的其中一个作业执行完成后，释放资源，下个作业才会正常提交，所有作业共享Dispatcher和ResourseManager;适合规模小执行时间短的作业

1. 启动hadoop集群

2. 启动flink

   [root@master ~]#

   yarn-session.sh -n 1 -s 2 -jm 1024 -tm 1024 -nm session-cluster  -d 
   

   参数	参数说明
   -n	TaskManager的数量
   -s	每个TaskManager 的slot数量；默认一个slot一个core,默认每个taskManager的slot为1
   -jm	JobManager的内存（MB）
   -tm	每个taskManager的内存
   -nm	yarn的appname
   -d	后台执行

   输出信息:

   2021-10-19 17:35:40,759 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Submitting application master application_1634636031663_0001
   2021-10-19 17:35:41,060 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1634636031663_0001
   2021-10-19 17:35:41,060 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Waiting for the cluster to be allocated
   2021-10-19 17:35:41,063 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Deploying cluster, current state ACCEPTED
   2021-10-19 17:35:48,089 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - YARN application has been deployed successfully.
   2021-10-19 17:35:48,090 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Found Web Interface master:34454 of application 'application_1634636031663_0001'.
   JobManager Web Interface: http://master:34454
   

   

3. 执行任务

   打jar包上传的步骤参考:https://blog.csdn.net/su_mingyang/article/details/120846470

   flink run -c socket.SocketWordCountTwo /root/flink_java_demo.jar --host master --port 8888
   

   输出信息:

   2021-10-19 17:45:17,380 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
   2021-10-19 17:45:17,380 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
   2021-10-19 17:45:17,645 WARN  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The configuration directory ('/usr/local/src/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
   2021-10-19 17:45:17,645 WARN  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The configuration directory ('/usr/local/src/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
   2021-10-19 17:45:19,231 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at master/192.168.222.201:8032
   2021-10-19 17:45:19,329 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
   2021-10-19 17:45:19,334 WARN  org.apache.flink.yarn.YarnClusterDescriptor                   - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
   2021-10-19 17:45:19,373 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Found Web Interface master:34454 of application 'application_1634636031663_0001'.
   Job has been submitted with JobID cba695c781def9620905738c607cc0ae
   

4. 通过yarn平台查看任务

   

5. 通过flink web端查看

   

   

11.2 Per-Job-Cluster模式

每次提交都会创建一个新的flink集群，任务之间相互独立，互不影响，方便管理。任务执行之后创建的集群也会消失。

[root@master ~]#

flink run -m yarn-cluster -c socket.SocketWordCountTwo /root/flink_java_demo.jar --host master --port 8888

输出信息

2021-10-19 18:18:12,751 WARN  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The configuration directory ('/usr/local/src/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-10-19 18:18:12,751 WARN  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The configuration directory ('/usr/local/src/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-10-19 18:18:14,491 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at master/192.168.222.201:8032
2021-10-19 18:18:14,641 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-10-19 18:18:14,788 WARN  org.apache.flink.yarn.YarnClusterDescriptor                   - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2021-10-19 18:18:14,818 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1728, slotsPerTaskManager=1}
2021-10-19 18:18:18,901 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Submitting application master application_1634636031663_0002
2021-10-19 18:18:18,927 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1634636031663_0002
2021-10-19 18:18:18,927 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Waiting for the cluster to be allocated
2021-10-19 18:18:18,929 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Deploying cluster, current state ACCEPTED
2021-10-19 18:18:24,041 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - YARN application has been deployed successfully.
2021-10-19 18:18:24,042 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Found Web Interface master:46528 of application 'application_1634636031663_0002'.
Job has been submitted with JobID 8e44b2e68e2b7ed59d0e53d1da59cc8f

11.3 关闭flink程序

[root@master ~]# yarn application -kill application_1634636031663_0003
21/10/19 18:27:17 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.222.201:8032
Killing application application_1634636031663_0003
21/10/19 18:27:18 INFO impl.YarnClientImpl: Killed application application_1634636031663_0003