hadoop 3.x 案例4:Flink启动yarn session问题

一. 问题描述

Flink启动yarn-session的时候报错:

2022-12-02 09:27:05,804 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-02 09:27:06,056 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-02 09:27:06,308 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-02 09:27:06,562 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

image.png

二. 解决方案

在网上找了下，基本都是说因为配置了Flink高可用，zookeeper没有启动造成的，然后我检查了我的zookeeper，是启动状态。

测试Hudi集成Flink的时候，遇到各种各样的问题，后面把Flink的配置文件中关于高可用的部分注释掉就没问题了。

但是Flink的 master在哪个节点，yarn-session也必须运行在哪个节点。

修改Flink配置文件：
注释掉Flink高可用的配置，并增加web运行时监视端口号

jobmanager.rpc.address: hp5
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 4096m
taskmanager.memory.process.size: 4096m
taskmanager.numberOfTaskSlots: 3
parallelism.default: 1
#high-availability: zookeeper
#high-availability.cluster-id: /flink-cluster
#high-availability.storageDir: hdfs://hp5:8020/vmcluster/flink/ha/
#high-availability.zookeeper.quorum: hp5:2181,hp6:2181,hp7:2181
#state.backend: filesystem
#state.checkpoints.dir: hdfs://hp5:8020/vmcluster/flink-checkpoints
#state.savepoints.dir: hdfs://hp5:8020/vmcluster/flink-savepoints
#jobmanager.execution.failover-strategy: region
#jobmanager.archive.fs.dir: hdfs://hp5:8020/vmcluster/completed-jobs/
#historyserver.archive.fs.dir: hdfs://hp5:8020/vmcluster/completed-jobs/
classloader.check-leaked-classloader: false

#  Web 的运行时监视器端口
rest.port: 9081