在准备好Hadoop的环境后,可以开始搭建Flink了。
这里我们参考的是这篇教程,其中包括很全面的安装步骤:https://files.alicdn.com/tpsservice/4824447b829149c86bedd19424d05915.pdf
这里我们的服务器列表和之前Hadoop是相同的:
我们现在有3台服务器,服务器列表如下:
hostname | ip | 作用 |
---|---|---|
master | 10.16.195.254 | JobManager, NodeManager |
slave1 | 10.16.196.1 | NodeManager |
slave2 | 10.16.196.5 | NodeManager |
1. 下载并配置Flink
1.1 下载Flink
在Flink的官网下载Binary的安装包到master的/data目录下,并解压,下载Hadoop的扩展库:
$ wget http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.8.0/flink-1.8.0-bin-scala_2.12.tgz
$ tar -xvf flink-1.8.0-bin-scala_2.12.tgz
$ cd flink-1.8.0/lib
$ wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
1.2 修改Flink的配置文件
Flink的配置文件在目录:/data/flink-1.8.0/conf
- 修改masters:
master
- 修改slaves:
master
slave1
slave2
- 修改flink-conf.yaml:
jobmanager.rpc.address: master
1.3 拷贝Flink到其他机器
将master的flink目录拷贝到其他机器的相同目录:
scp -r /data/flink-1.8.0 root@slave1:/data
scp -r /data/flink-1.8.0 root@slave2:/data
2. 启动Flink
如果想通过Yarn实现资源管理,需要先在Yarn启动Flink,然后再执行Flink的程序。
2.1 启动Flink Yarn Session
$ ./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
2019-07-01 22:43:04,154 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, ads-data-web-online012-bjdxt9p
2019-07-01 22:43:04,155 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2019-07-01 22:43:04,155 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2019-07-01 22:43:04,155 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2019-07-01 22:43:04,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-07-01 22:43:04,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2019-07-01 22:43:04,607 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-01 22:43:04,719 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hdfs/[email protected] (auth:KERBEROS)
2019-07-01 22:43:04,775 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /10.16.195.254:8032
2019-07-01 22:43:04,913 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - The argument n is deprecated in will be ignored.
2019-07-01 22:43:05,099 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:43:05,152 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=4, slotsPerTaskManager=1}
2019-07-01 22:43:05,595 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - The configuration directory ('/data/flink-1.8.0/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-07-01 22:43:07,827 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Adding delegation token to the AM container..
2019-07-01 22:43:07,836 INFO org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 3 for hdfs on 10.16.195.254:9000
2019-07-01 22:43:07,848 INFO org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs://10.16.195.254:9000; Kind: HDFS_DELEGATION_TOKEN, Service: 10.16.195.254:9000, Ident: (HDFS_DELEGATION_TOKEN token 3 for hdfs)
2019-07-01 22:43:07,848 INFO org.apache.flink.yarn.Utils - Attempting to obtain Kerberos security token for HBase
2019-07-01 22:43:07,848 INFO org.apache.flink.yarn.Utils - HBase is not available (not packaged with this application): ClassNotFoundException : "org.apache.hadoop.hbase.HBaseConfiguration".
2019-07-01 22:43:07,857 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application master application_1562035367237_0001
2019-07-01 22:43:08,384 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1562035367237_0001
2019-07-01 22:43:08,384 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for the cluster to be allocated
2019-07-01 22:43:08,386 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying cluster, current state ACCEPTED
2019-07-01 22:43:14,953 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN application has been deployed successfully.
2019-07-01 22:43:15,564 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
Flink JobManager is now running on master:1315 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://master:1315
启动后,可以打开JobManager的管理页面,地址在日志中。在yarn的管理页面中也可以看到一个运行中的应用:
2.2 执行示例程序
执行Flink的examples程序:
$ /bin/flink run examples/streaming/WordCount.jar --input hdfs:///user/hdfs/input_dir --output hdfs:///user/hdfs/output_dir
2019-07-01 22:46:45,084 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,084 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,603 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - YARN properties set default parallelism to 4
2019-07-01 22:46:45,603 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - YARN properties set default parallelism to 4
YARN properties set default parallelism to 4
2019-07-01 22:46:45,636 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /10.16.195.254:8032
2019-07-01 22:46:45,774 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,774 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,778 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:46:45,960 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Found application JobManager host name 'ads-data-web-online015-bjdxt9p.qiyi.virtual' and port '1315' from supplied application id 'application_1562035367237_0001'
Starting execution of program
Program execution finished
Job with JobID f1a63f47e44521d457773c3e0acab80b has finished.
Job Runtime: 256 ms
以上,便完成了Flink在Yarn上的安装和部署,更多关于Flink安装的细节,可以参考文章开始的文档,其中的内容讲解很详细。