flink部署模式（二）- yarn三种部署模式

基于flink1.15.0

Session Mode on yarn

开启flink yarn-session集群

./bin/yarn-session.sh -d \
-Dtaskmanager.numberOfTaskSlots=5 \
-Dflink.cluster.name=flink-yarn-session \
-Dyarn.application.name="flink-yarn-session" \
-Dresourcemanager.taskmanager-timeout=30000 \
-Drest.flamegraph.enabled=true 

提交任务

• 方式一，不指定 yarn app-id提交

./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
或者

• 方式二，手动指定flink集群，application_1652928425471_0282是flink集群yarn app-id

./bin/flink run -t yarn-session -d
-Dyarn.application.id=application_1652928425471_0285
./examples/streaming/TopSpeedWindowing.jar

• 方式三，在flink web界面上传jar包提交

进入客户端交互界面

./bin/yarn-session.sh -id application_1652928425471_0252
优雅地停止任务

echo “stop” | ./bin/yarn-session.sh -id application_1652928425471_0245

强制杀掉任务

yarn application -kill application_1652928425471_0266

NOTE： 启动集群只有jobmanager，提交job才去yarn申请资源启动taskmanager，任务完成自动释放taskmanager

Per-Job Mode on yarn

启动perjob集群

./bin/flink run -t yarn-per-job --detached \
-Dyarn.application.name="flink-yarn-perjob" \
-Dflink.cluster.name=flink-yarn-perjob \
-Dtaskmanager.numberOfTaskSlots=5 \
./examples/streaming/TopSpeedWindowing.jar

查看集群上运行的job
./bin/flink list -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252
取消某个job

./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252
./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252 ac02048e90c0ad7ec370542b412e498c

**NOTE：**在perjob模式，取消某个job相当于停止集群

Application Mode on yarn

flink1.11之后引入
启动application集群

./bin/flink run-application \
-t yarn-application \
-Dyarn.application.name="flink-yarn-application" \
-Dflink.cluster.name=flink-yarn-perjob \
-Dtaskmanager.numberOfTaskSlots=5 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=1024m \
-Drest.flamegraph.enabled=true \
./examples/streaming/TopSpeedWindowing.jar

查看集群上运行的job

./bin/flink list -t yarn-application -Dyarn.application.id=application_1652928425471_0268

取消某个job

./bin/flink cancel -t yarn-application -Dyarn.application.id=application_1652928425471_0268
./bin/flink cancel -t yarn-application -Dyarn.application.id=application_1652928425471_0254 17e0ba867df35d98c2d70be8103fff2e

**NOTE：**在perjob模式，取消某个job相当于停止集群

另外可以结合yarn依赖预上传设置，来优化job的提交效果