Flink 的yarn 两种提交方式

Flink 的yarn 两种提交方式

标题第一种

1.yarn-session.sh + flink run(提交任务)

命令：

	优缺点：

特点：需要事先申请资源，启动JobManager和TaskManger
优点：不需要每次递交作业申请资源，而是使用已经申请好的资源，从而提高执行效率
缺点：作业执行完成以后，资源不会被释放，因此一直会占用系统资源
应用场景：适合作业递交比较频繁的场景，小作业比较多的场景

/opt/module/flink-1.12.0/bin/yarn-session.sh -n 2 -tm 800 -s 1 -d

参数说明：
申请2个CPU、1600M内存
 -n 表示申请2个容器，这里指的就是多少个taskmanager
 -tm 表示每个TaskManager的内存大小
 -s 表示每个TaskManager的slots数量
 -d 表示以后台程序方式运行

2. 执行成功后会在yarn 的界面出现：

3. 提交一个任务：

	/opt/module/flink-1.12.0/bin/flink run  /opt/module/flink-1.12.0/examples/batch/WordCount.jar

4.查看执行结果

[root@master01 batch]# /opt/module/flink-1.12.0/bin/flink run  /opt/module/flink-1.12.0/examples/batch/WordCount.jar
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:30:59,306 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
2021-03-07 13:30:59,306 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2021-03-07 13:31:01,314 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:31:02,644 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl    [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:31:02,984 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at master01.pxx.com/192.168.1.151:8050
2021-03-07 13:31:03,320 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-03-07 13:31:03,568 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface master02.pxx.com:39632 of application 'application_1613230903820_0014'.
Job has been submitted with JobID 3b4c527309843dc73b467eb44e733aed
Program execution finished
Job with JobID 3b4c527309843dc73b467eb44e733aed has finished.
Job Runtime: 22853 ms
Accumulator Results: 
- 2106e844138b8bd3e937dde396d9d172 (java.util.ArrayList) [170 elements]
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(fair,1)
(fardels,1)
(flesh,1)
(fly,1)
(for,2)
(fortune,1)
(from,1)
(give,1)
[root@master01 batch]# 

表示执行成功

第二种：

1.采用 Per-Job分离模式

特点

特点：每次递交作业都需要申请一次资源
优点：作业运行完成，资源会立刻被释放，不会一直占用系统资源
缺点：每次递交作业都需要申请资源，会影响执行效率，因为申请资源需要消耗时间
应用场景：适合作业比较少的场景、大作业的场景

操作实例：
命令：

/opt/module/flink-1.12.0/bin/flink run -m yarn-cluster -yjm 1024 -ytm 1024 /opt/module/flink-1.12.0/examples/batch/WordCount.jar
参数解释：
 -m  jobmanager的地址
 -yjm 1024 指定jobmanager的内存信息
 -ytm 1024 指定taskmanager的内存信息

如图：

命令行执行结果：

[root@master01 batch]# /opt/module/flink-1.12.0/bin/flink run -m yarn-cluster -yjm 1024 -ytm 1024 /opt/module/flink-1.12.0/examples/batch/WordCount.jar
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:44:56,723 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
2021-03-07 13:44:56,723 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-root.
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2021-03-07 13:44:57,920 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:44:58,957 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl    [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:44:59,206 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at master01.pxx.com/192.168.1.151:8050
2021-03-07 13:44:59,530 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-03-07 13:44:59,872 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 1024 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (4096 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 3072 MB may not be used by Flink.
2021-03-07 13:44:59,873 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured TaskManager memory is 1024 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (4096 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 3072 MB may not be used by Flink.
2021-03-07 13:44:59,873 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{masterMemoryMB=4096, taskManagerMemoryMB=1024, slotsPerTaskManager=2}
2021-03-07 13:45:01,029 WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory      [] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2021-03-07 13:45:07,915 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl    [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:45:07,918 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1613230903820_0015
2021-03-07 13:45:08,230 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1613230903820_0015
2021-03-07 13:45:08,231 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
2021-03-07 13:45:08,236 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED
2021-03-07 13:45:26,144 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - YARN application has been deployed successfully.
2021-03-07 13:45:26,148 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface master02.pxx.com:44213 of application 'application_1613230903820_0015'.
Job has been submitted with JobID 2e6cfab4814d80e167ea0b539ea71a7e
Program execution finished
Job with JobID 2e6cfab4814d80e167ea0b539ea71a7e has finished.
Job Runtime: 24345 ms
Accumulator Results: 
- 40fb3a5a662876772a8b702eda2f6f80 (java.util.ArrayList) [170 elements]

(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
(bodkin,1)
(bourn,1)
(but,1)
(by,2)
(calamity,1)
(cast,1)
(coil,1)
[root@master01 batch]# 

---

执行成功.

在flink 通过提交yarn 模式的时候 会出现错误

[root@master01 bin]# yarn-session.sh -n 2 -tm 800 -s 1 -d
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:02:22,315 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.address, master01.pxx.com
2021-03-07 13:02:22,325 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-07 13:02:22,325 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.memory.process.size, 2048m
2021-03-07 13:02:22,326 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.memory.process.size, 2048m
2021-03-07 13:02:22,326 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2
2021-03-07 13:02:22,326 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: parallelism.default, 1
2021-03-07 13:02:22,326 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2021-03-07 13:02:22,327 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: web.submit.enable, true
2021-03-07 13:02:22,328 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.archive.fs.dir, hdfs://master01.pxx.com:8020/flink/completed-jobs/
2021-03-07 13:02:22,328 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: historyserver.web.address, master01.pxx.com
2021-03-07 13:02:22,328 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: historyserver.web.port, 8082
2021-03-07 13:02:22,328 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: historyserver.archive.fs.dir, hdfs://master01.pxx.com:8020/flink/completed-jobs/
2021-03-07 13:02:23,178 WARN  org.apache.hadoop.util.NativeCodeLoader                      [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-07 13:02:23,432 INFO  org.apache.flink.runtime.security.modules.HadoopModule       [] - Hadoop user set to root (auth:SIMPLE)
2021-03-07 13:02:23,456 INFO  org.apache.flink.runtime.security.modules.JaasModule         [] - Jaas file will be created as /tmp/jaas-5714068345138708081.conf.
2021-03-07 13:02:23,495 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:02:23,575 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Error while running the Flink session.
java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
	at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_251]
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_251]
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_251]
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_251]
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_251]
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_251]
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_251]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_251]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_251]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) ~[?:1.8.0_251]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_251]
	at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:83) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:63) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:45) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:479) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_251]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730) [flink-dist_2.12-1.12.0.jar:1.12.0]
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.util.FeaturesAndProperties
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_251]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_251]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) ~[?:1.8.0_251]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_251]
	... 26 more

------------------------------------------------------------
 The program finished with the following exception:

java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:83)
	at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:63)
	at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:45)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:479)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.util.FeaturesAndProperties
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 26 more

意思是说缺少jar 包： 将这几个包下载下来放如到flink目录中的lib中就可以了

下载地址：

https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api/2.1

jersey-common-2.27.jar
javax.ws.rs-api-2.1.jar
jersey-core-1.19.jar
flink-hadoop-compatibility_2.12-1.12.0.jar