Flink on yarn遇到的问题

Flink版本：flink-1.9.1 

下载地址：https://flink.apache.org/downloads.html

我是分开部署的，部署了hadoop 3.1.2

所以下载 两个文件 Apache Flink 1.9.1 for Scala 2.12 和Pre-bundled Hadoop 2.8.3 （一个jar包）

官方文档：https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/yarn_setup.html

 

Flink on yarn 部署：

1. 直接解压 flink-1.9.1-bin-scala_2.12.tgz

然后将上面的jar包（flink-shaded-hadoop-2-uber-2.8.3-7.0.jar）放在解压后的flink-1.9.1/lib/ 下

2.hadoop环境配置即hadoop的路径要配置进 /etc/profile

• Test if YARN_CONF_DIR, HADOOP_CONF_DIR or HADOOP_CONF_PATH are set (in that order). If one of these variables is set, it is used to read the configuration.
• If the above strategy fails (this should not be the case in a correct YARN setup), the client is using the HADOOP_HOME environment variable. If it is set, the client tries to access $HADOOP_HOME/etc/hadoop (Hadoop 2) and $HADOOP_HOME/conf (Hadoop 1).

export HADOOP_HOME=/home/hadoop-3.1.2

3.提交任务

flink-1.9.1/bin/flink run -m yarn-cluster  -yjm 2048 -ytm 2048 /home/wordCount.jar

去hadoop集群上查看任务

http://127.0.0.1:8088/

4.查看运行日志  在hadoop的logs中，taskmanager.log

 

常见错误

The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment

明明运行了execute()，这个时候要查日志找到具体原因

日志在各node节点的/hadoop-3.1.2/logs/userlogs/ 下面的jobmanager.log和taskmanager.log中

问题

1.org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 60000ms

flink+kafka 遇到的问题，防火墙读关了，本地正常，服务器上就跑不了，google也查不到解决方式。

原因是kafka的问题

解决：

修改kafka配置 kafka_2.12-2.3.0/config/server.properties

#listeners=PLAINTEXT://host_name:9092

这个注释解开，配上yarn集群所在机器能访问的 ip或者hostname

2.其他很多错误，反正大部分是yarn集群的问题导致的，所以要确认yarn集群正确，并不是跑起来就可以

比如：

Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

 

The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.

 

其他等等，解决这些问题要重新配置好yarn集群，参考我的另一篇博客

https://blog.csdn.net/cgb278653219/article/details/103699317

3.jar包不正确导致的无法运行

打包方式网上搜一下