Flink v1.6.1 《Yarn 模式 高可用分布式集群部署》

目录

1. Flink下载

2. 环境准备

2.1 软件要求

2.2 集群部署规划

3. 解压安装

3.1 解压安装包

3.2 修改配置文件

3.3 发送到其他节点

3.4 配置环境变量

4. 启动

5. 参考

---

Flink高可用集群分为Standalone HA 和 YARN Cluster HA，这里只介绍 YARN Cluster HA。

1. Flink下载

安装包下载地址：https://archive.apache.org/dist/flink/flink-1.6.1/flink-1.6.1-bin-hadoop26-scala_2.11.tgz

选择与Hadoop版本对应的安装包

2. 环境准备

2.1 软件要求

• Java 1.8.x或更高版本
• ssh(运行Flink 脚本管理远程组件必须运行sshd)

2.2 集群部署规划

主机名	Master	Worker	Zookeeper
master			zookeeper
node1		worker
node2		worker
node3		worker	zookeeper
node4		worker	zookeeper

Yarn HA集群只运行一个JobManager ，具体是哪个节点由zookeeper选举决定，所以并不清楚会在哪台上。

3. 解压安装

3.1 解压安装包

tar -zxvf flink-1.6.1-bin-hadoop26-scala_2.11.tgz -C /opt/

3.2 修改配置文件

[root@master conf]# cd /opt/flink-1.6.1/conf/
[root@master conf]# ls
flink-conf.yaml           log4j.properties               logback.xml       slaves
log4j-cli.properties      log4j-yarn-session.properties  logback-yarn.xml  sql-client-defaults.yaml
log4j-console.properties  logback-console.xml            masters           zoo.cfg
[root@master conf]# 

Yarn HA 模式需要配置三个文件：flink-conf.yaml，slaves，zoo.cfg。（master选举产生，不用配置）

flink-conf.yaml

high-availability: zookeeper
high-availability.zookeeper.quorum: master:2181,node3:2181,node4:2181
high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 10

slaves

node1
node2
node3
node4

zoo.cfg

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial  synchronization phase can take
initLimit=10

# The number of ticks that can pass between  sending a request and getting an acknowledgement
syncLimit=5

# The directory where the snapshot is stored.
# dataDir=/tmp/zookeeper

# The port at which the clients will connect
clientPort=2181

# ZooKeeper quorum peers
server.1=master:2888:3888
server.2=node3:2888:3888
server.3=node4:2888:3888

此外，还需要在yarn-site.xml中配置 application masters尝试的最大次数（默认2）。

  yarn.resourcemanager.am.max-attempts
  4
  
    The maximum number of application master execution attempts.
  

3.3 发送到其他节点

[root@master conf]# cd ../../
[root@master opt]# scp -r flink-1.6.1 root@node1:/opt/

每个节点都发一次

3.4 配置环境变量

[root@master opt]# echo 'export HADOOP_CONF_DIR=/etc/hadoop/conf' >> /etc/profile
[root@master opt]# echo 'export FLINK_HOME=/opt/flink-1.6.1' >> /etc/profile
[root@master opt]# echo 'export PATH=$PATH:$FLINK_HOME/bin' >> /etc/profile
[root@master opt]# source /etc/profile

4. 启动

[root@master opt]# yarn-session.sh -n 2

从监控界面可以看到主节点是node3

5. 参考

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/yarn_setup.html

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/jobmanager_high_availability.html