Flink on yarn集群搭建

前面一篇博客中已经搭建了flink Standalone的集群,需要的可以进去看一下,今天主要来说一下flink on yarn 集群的搭建以及怎么提交任务.这篇博客写的比较详细,内容较多,希望大家耐心看完,干货满满

版本信息:

flink-1.6.0

zookeeper-3.4.10

hadoop-2.9.0

zk和hadoop集群的搭建,今天这里就不说了,主要说一下flink的.

(1). vi flink-conf.yaml (里面具体的参数大小可以根据自己的机器实际情况而定)我配置的比较简单.

env.java.home: /home/jason/bigdata/jdk/jdk1.8.0_11
jobmanager.rpc.address: master
taskmanager.heap.size: 4096
taskmanager.numberOfTaskSlots: 4
#parallelism.default: 8
taskmanager.tmp.dirs: /tmp
jobmanager.heap.size: 1024
jobmanager.web.port: 8085
jobmanager.rpc.port: 6123
state.backend: filesystem
state.backend.fs.checkpointdir: hdfs:///flink/checkpoints
#taskmanager.network.numberOfBuffers: 64000
fs.hdfs.hadoopconf: /home/jason/bigdata/hadoop/hadoop-2.9.0/etc/hadoop
recovery.mode: zookeeper
recovery.zookeeper.quorum: master:2181,storm1: