Flink on Yarn集群HA高可用部署 基于flink1.12 hadoop 3.0 CDH6.3.2

1.1 概要介绍


Flink on Yarn的HA高可用模式,首先依赖于Yarn自身的高可用机制(ResourceManager高可用),并通过Yarn对JobManager进行管理,当JobManager失效时,Yarn将重新启动JobManager。其次Flink Job在恢复时,需要依赖Checkpoint进行恢复,而Checkpoint的快照依赖于远端的存储:HDFS,所以HDFS也必须是高可用,同时JobManager的元数据信息也依赖于HDFS的高可用(namenode的高可用,和多副本机制),再者JobManager元数据的指针信息要依赖于Zookeeper的高可用。

 

1.2 Flink on Yarn的优势
相对于 Standalone 模式,在Yarn 模式下有以下几点好处:

1.资源按需使用,提高集群的资源利用率;

2.任务有优先级,根据优先级运行作业;

3.基于 Yarn 调度系统,能够自动化地处理各个角色的 Failover:

      JobManager 进程和 TaskManager 进程都由 Yarn NodeManager 监控;

      如果 JobManager 进程异常退出,则 Yarn ResourceManager 会重新调度 JobManager 到其他机器;

      如果 TaskManager 进程异常退出,JobManager 会收到消息并重新向 Yarn ResourceManager 申请资源,重新启动 TaskManager。
 

第2章 Flink on Yarn模式运行的方式


2.1 Per-Job
Per-Job模式:简答的说就是直接run job,每次提交的任务Yarn都会分配一个JobManager,执行完之后整个资源会释放,包括JobManager和TaskManager。

Per-Job模式适合比较大的任务、执行时间比较长的任务。

2.2 Session
Session模式:在Session模式中, Dispatcher 和 ResourceManager 是可以复用的;当执行完Job之后JobManager并不会释放,Session 模式也称为多线程模式,其特点是资源会一直存在不会释放。使用时先启动yarn-session,然后再提交job,每次提交job,也都会分配一个JobManager。
Session模式适合比较小的任务、执行时间比较短的任务。该模式不用频繁的申请资源和释放资源。

 

所以一般生产情况下我们都会选取 on Yarn 部署 per-job方式运行

 

下载的话直接官网下载

https://www.apache.org/dyn/closer.lua/flink/flink-1.12.1/flink-1.12.1-bin-scala_2.12.tgz

https://mirrors.bfsu.edu.cn/apache/flink/flink-1.12.1/flink-1.12.1-bin-scala_2.12.tgz

下载1.12的

解压后进入目录

然后开始部署

修改Hadoop配置

修改hadoop配置文件/etc/hadoop/yarn-site.xml,设置application master重启时,尝试的最大次数。

cdh上yarn ---> 配置 搜索框 yarn.resourcemanager.am.max-attempts

默认2次 先写个4次吧 按需填写即可 然后重启yarn

修改masters

 修改conf目录下masters文件

hadoop1:8081

端口可以改的 以免和spark的端口冲突

部署Per-Job模式

Flink on yarn将会覆盖掉几个参数:
jobmanager.rpc.address因为jobmanager的在集群的运行位置并不是事先确定的,它就是am的地址;
taskmanager.tmp.dirs使用yarn给定的临时目录;
parallelism.default也会被覆盖掉,如果在命令行里指定了slot数。

提前创建flink在hadoop上的逻辑数据目录


#jobmanager和taskmanager、其他client的RPC通信IP地址,TaskManager用于连接到#JobManager/ResourceManager 。HA模式不用配置此项,在master文件中配置,由zookeeper选出leader与#standby

jobmanager.rpc.address: 192.168.101.1

# jobmanager和taskmanager、其他client的RPC通信端口,TaskManager用于连接到JobManager/ResourceManager 。HA模式不用配置此项,在master文件中配置,由zookeeper选出leader与standby


jobmanager.rpc.port: 6123


# jobmanager JVM heap 内存大小 给个4g吧

jobmanager.memory.process.size: 4096m


# taskmanager JVM heap 内存大小 按需填写

taskmanager.memory.process.size: 16384m

# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280m

# 每个taskmanager提供的任务slots数量
# 并行度等于TM 数量乘以每个TM 的Solts 数量 TM=并行度/Solts数量 如果slots数量大于8 则只会起一个TM
# 听说不要超过5

taskmanager.numberOfTaskSlots: 3


# 并行计算个数 和cpu一致吧

parallelism.default: 4


#==============================================================================
# High Availability
#==============================================================================

# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability: zookeeper


# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
# 
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...) 
#
# JobManager元数据保存位置
high-availability.storageDir: hdfs://nameservice1/flink/ha/


# zookeeper 地址
high-availability.zookeeper.quorum:  master1:2181,master2:2181,core1:2181


# ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes
# It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE)
# The default value is "open" and it can be changed to "creator" if ZK security is enabled
#
# high-availability.zookeeper.client.acl: open

#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================

# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# .
#rocksdb是官方生产环境推荐的存储方式
state.backend: rocksdb

# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#存储检查点的数据文件和元数据的默认目录
state.checkpoints.dir: hdfs://nameservice1/flink/flink-checkpoints

# Default target directory for savepoints, optional.
##存储检查点的数据文件和元数据的默认目录 同上即可
state.savepoints.dir: hdfs://nameservice1/flink/flink-savepoints

# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend). 
#
# state.backend.incremental: false

# The failover strategy, i.e., how the job computation recovers from task failures.
# Only restart tasks that may have been affected by the task failure, which typically includes
# downstream tasks and potentially upstream tasks if their produced data is no longer available for consumption.

jobmanager.execution.failover-strategy: region

#==============================================================================
# Rest & web frontend
#==============================================================================

# The port to which the REST client connects to. If rest.bind-port has
# not been specified, then the server will bind to this port as well.
#
rest.port: 8083

# The address to which the REST client will connect to
#
#rest.address: 0.0.0.0

# Port range for the REST and web server to bind to.
#
#rest.bind-port: 8080-8090

# The address that the REST & web server binds to
#
#rest.bind-address: 0.0.0.0

# Flag to specify whether job submission is enabled from the web-based
# runtime monitor. Uncomment to disable.

#web.submit.enable: false

#==============================================================================
# Advanced
#==============================================================================

# Override the directories for temporary files. If not specified, the
# system-specific Java temporary directory (java.io.tmpdir property) is taken.
#
# For framework setups on Yarn or Mesos, Flink will automatically pick up the
# containers' temp directories without any need for configuration.
#
# Add a delimited list for multiple directories, using the system directory
# delimiter (colon ':' on unix) or a comma, e.g.:
#     /data1/tmp:/data2/tmp:/data3/tmp
#
# Note: Each directory entry is read from and written to by a different I/O
# thread. You can include the same directory multiple times in order to create
# multiple I/O threads against that directory. This is for example relevant for
# high-throughput RAIDs.
#
# io.tmp.dirs: /tmp

# The classloading resolve order. Possible values are 'child-first' (Flink's default)
# and 'parent-first' (Java's default).
#
# Child first classloading allows users to use different dependency/library
# versions in their application than those in the classpath. Switching back
# to 'parent-first' may help with debugging dependency issues.
#
# classloader.resolve-order: child-first

# The amount of memory going to the network stack. These numbers usually need 
# no tuning. Adjusting them may be necessary in case of an "Insufficient number
# of network buffers" error. The default min is 64MB, the default max is 1GB.
# 
# taskmanager.memory.network.fraction: 0.1
# taskmanager.memory.network.min: 64mb
# taskmanager.memory.network.max: 1gb

#==============================================================================
# Flink Cluster Security Configuration
#==============================================================================

# Kerberos authentication for various components - Hadoop, ZooKeeper, and connectors -
# may be enabled in four steps:
# 1. configure the local krb5.conf file
# 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit)
# 3. make the credentials available to various JAAS login contexts
# 4. configure the connector to use JAAS/SASL

# The below configure how Kerberos credentials are provided. A keytab will be used instead of
# a ticket cache if the keytab path and principal are set.

# security.kerberos.login.use-ticket-cache: true
# security.kerberos.login.keytab: /path/to/kerberos/keytab
# security.kerberos.login.principal: flink-user

# The configuration below defines which JAAS login contexts

# security.kerberos.login.contexts: Client,KafkaClient

#==============================================================================
# ZK Security Configuration
#==============================================================================

# Below configurations are applicable if ZK ensemble is configured for security

# Override below configuration to provide custom ZK service name if configured
# zookeeper.sasl.service-name: zookeeper

# The configuration below must match one of the values set in "security.kerberos.login.contexts"
# zookeeper.sasl.login-context-name: Client

#==============================================================================
# HistoryServer
#==============================================================================

# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)

# Directory to upload completed jobs to. Add this directory to the list of
# monitored directories of the HistoryServer as well (see below).
# 历史数据的存储位置
jobmanager.archive.fs.dir: hdfs://nameservice1/flink/log/completed-jobs/

# The address under which the web-based HistoryServer listens.
historyserver.web.address: 192.168.102.1

# The port under which the web-based HistoryServer listens.
historyserver.web.port: 8082

# Comma separated list of directories to monitor for completed jobs.
# 历史数据的存储位置 同上
historyserver.archive.fs.dir: hdfs://nameservice1/flink/log/completed-jobs/

# Interval in milliseconds for refreshing the monitored directories.
historyserver.archive.fs.refresh-interval: 10000

然后就可以运行命令测试了

直接命令行跑这个   

export HADOOP_CLASSPATH=`hadoop classpath` 

我配置了环境变量 如果你没配置 那就bin/flink  再flink安装目录下运行 使用官方example 运行看看

flink run -m yarn-cluster ./examples/batch/WordCount.jar   

没报错 出来一堆

这种 就是安装成功了

可以在yarn的界面和cdh的界面查到flink的任务

注意需要在hadoop的master服务器上运行 ,会去获取yarn的地址 但是没找到配置项 只在本地找

 

至此,Flink on Yarn集群高可用模式的部署及使用就完成了

本篇使用了网上的文档进行编写。实际搭建中我怀疑是不是on yarn模式不需要配置什么东西 官方文档非常简单 下包 --运行环境变量--运行flink  下次搭建的时候试试是不是真的on yarn模式不需要配置什么东西

 

 

Flink on Yarn集群HA高可用部署 基于flink1.12 hadoop 3.0 CDH6.3.2_第1张图片

 

on yarn 模式下web ui地址不固定 会在最后一行指出 复制下来网页查看即可 。不知道有没有大佬告诉我一下怎么弄个常驻的web ui  真的很不方便

 

 

 


 

你可能感兴趣的:(hadoop,flink)