Flink1.8 集群搭建完全指南(5):Flink on Yarn

在准备好Hadoop的环境后,可以开始搭建Flink了。

这里我们参考的是这篇教程,其中包括很全面的安装步骤:https://files.alicdn.com/tpsservice/4824447b829149c86bedd19424d05915.pdf

这里我们的服务器列表和之前Hadoop是相同的:
我们现在有3台服务器,服务器列表如下:

hostname ip 作用
master 10.16.195.254 JobManager, NodeManager
slave1 10.16.196.1 NodeManager
slave2 10.16.196.5 NodeManager

1. 下载并配置Flink

1.1 下载Flink

在Flink的官网下载Binary的安装包到master的/data目录下,并解压,下载Hadoop的扩展库:

$ wget http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.8.0/flink-1.8.0-bin-scala_2.12.tgz
$ tar -xvf flink-1.8.0-bin-scala_2.12.tgz
$ cd flink-1.8.0/lib
$ wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
1.2 修改Flink的配置文件

Flink的配置文件在目录:/data/flink-1.8.0/conf

  • 修改masters:
master
  • 修改slaves:
master
slave1
slave2
  • 修改flink-conf.yaml:
jobmanager.rpc.address: master
1.3 拷贝Flink到其他机器

将master的flink目录拷贝到其他机器的相同目录:

scp -r /data/flink-1.8.0 root@slave1:/data
scp -r /data/flink-1.8.0 root@slave2:/data

2. 启动Flink

如果想通过Yarn实现资源管理,需要先在Yarn启动Flink,然后再执行Flink的程序。

2.1 启动Flink Yarn Session
$ ./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
2019-07-01 22:43:04,154 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, ads-data-web-online012-bjdxt9p
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-07-01 22:43:04,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-07-01 22:43:04,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-07-01 22:43:04,607 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-01 22:43:04,719 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to hdfs/[email protected] (auth:KERBEROS)
2019-07-01 22:43:04,775 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at /10.16.195.254:8032
2019-07-01 22:43:04,913 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument n is deprecated in will be ignored.
2019-07-01 22:43:05,099 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:43:05,152 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=4, slotsPerTaskManager=1}
2019-07-01 22:43:05,595 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/data/flink-1.8.0/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-07-01 22:43:07,827 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Adding delegation token to the AM container..
2019-07-01 22:43:07,836 INFO  org.apache.hadoop.hdfs.DFSClient                              - Created HDFS_DELEGATION_TOKEN token 3 for hdfs on 10.16.195.254:9000
2019-07-01 22:43:07,848 INFO  org.apache.hadoop.mapreduce.security.TokenCache               - Got dt for hdfs://10.16.195.254:9000; Kind: HDFS_DELEGATION_TOKEN, Service: 10.16.195.254:9000, Ident: (HDFS_DELEGATION_TOKEN token 3 for hdfs)
2019-07-01 22:43:07,848 INFO  org.apache.flink.yarn.Utils                                   - Attempting to obtain Kerberos security token for HBase
2019-07-01 22:43:07,848 INFO  org.apache.flink.yarn.Utils                                   - HBase is not available (not packaged with this application): ClassNotFoundException : "org.apache.hadoop.hbase.HBaseConfiguration".
2019-07-01 22:43:07,857 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1562035367237_0001
2019-07-01 22:43:08,384 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1562035367237_0001
2019-07-01 22:43:08,384 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-07-01 22:43:08,386 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-07-01 22:43:14,953 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
2019-07-01 22:43:15,564 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
Flink JobManager is now running on master:1315 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://master:1315

启动后,可以打开JobManager的管理页面,地址在日志中。在yarn的管理页面中也可以看到一个运行中的应用:


Flink1.8 集群搭建完全指南(5):Flink on Yarn_第1张图片
yarn
2.2 执行示例程序

执行Flink的examples程序:

$ /bin/flink run examples/streaming/WordCount.jar --input hdfs:///user/hdfs/input_dir --output hdfs:///user/hdfs/output_dir
2019-07-01 22:46:45,084 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,084 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,603 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 4
2019-07-01 22:46:45,603 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 4
YARN properties set default parallelism to 4
2019-07-01 22:46:45,636 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at /10.16.195.254:8032
2019-07-01 22:46:45,774 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,774 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,778 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:46:45,960 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Found application JobManager host name 'ads-data-web-online015-bjdxt9p.qiyi.virtual' and port '1315' from supplied application id 'application_1562035367237_0001'
Starting execution of program
Program execution finished
Job with JobID f1a63f47e44521d457773c3e0acab80b has finished.
Job Runtime: 256 ms

以上,便完成了Flink在Yarn上的安装和部署,更多关于Flink安装的细节,可以参考文章开始的文档,其中的内容讲解很详细。

你可能感兴趣的:(Flink1.8 集群搭建完全指南(5):Flink on Yarn)