1.安装flink-1.12.2

FLINK on YARN模式

解压安装包:

tar -zvxf  flink-1.12.2-bin-scala_2.11.tgz /opt/

修改yarn配置,设置application master重启时尝试的最大次数(cdh内有此参数,默认是2):


yarn.resourcemanager.am.max-attempts
10

The maximum number of application master execution attempts.

配置flink-conf.yaml文件(有些配置已存在,只需要修改参数)

cd /opt/flink-1.12.2/conf/
vi flink-conf.yaml

taskmanager.memory.process.size: 8192m
taskmanager.memory.managed.fraction: 0.2	
jobmanager.memory.process.size: 2048m
taskmanager.numberOfTaskSlots: 3
cluster.evenly-spread-out-slots: true
parallelism.default: 10

high-availability: zookeeper
high-availability.storageDir: hdfs:///flink/ha
high-availability.zookeeper.quorum: node1:2181,node2:2181,node3:2181
high-availability.zookeeper.path.root: /flink

yarn.application-attempts: 5
rest.port: 8085
web.submit.enable: true
io.tmp.dirs: /tmp

classloader.resolve-order: parent-first
classloader.check-leaked-classloader: false

state.backend: rocksdb
execution.checkpointing.interval: 30000
state.checkpoints.dir: hdfs:///flink/checkpoints
state.checkpoints.num-retained: 10
state.savepoints.dir: hdfs:///flink/savepoints
state.backend.incremental: true

historyserver.web.port: 8082
historyserver.archive.fs.refresh-interval: 10000

jobmanager.execution.failover-strategy: region

修改zoo.cfg:

vi zoo.cfg

#添加zookeeper集群
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

配置环境变量:

vi /etc/profile

export FLINK_HOME=/opt/flink-1.12.2
export PATH=$FLINK_HOME/bin:$PATH
export HADOOP_CLASSPATH=`hadoop classpath`
source /etc/profile

测试

FLINK on YARN示例:
yarn-session模式:

#启动成功后,在yarn集群查看,名字是test
bin/yarn-session.sh -nm test  -d  

#启动flinksql客户端	
bin/sql-client.sh embedded -s test 

你可能感兴趣的:(大数据实战,flink)