入坑Flink - Flink安装部署(单机模式)

1、Flink单机模式

1、将编译后的包拷贝到安装目录

cp -r flink-dist/target/flink-1.8.2-bin/flink-1.8.2/ /opt/platform/

2、改变文件所属

chown -R dc:dc flink-1.8.2

3、修改默认端口号（conf/flink-conf.yaml ）

# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 16123

4、启动

./bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host hadoop-datanode1.
Starting taskexecutor daemon on host hadoop-datanode1.

5、UI界面查看

6、借助官网例子测试

1、First of all, we use netcat to start local server via

nc -l 9000

2、Submit the Flink program:

./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 10010

3、查看

输出

查看日志

tail -100f log/flink-dc-taskexecutor-2-hadoop-datanode1.out
hello : 1
flink : 1
hello : 1
data : 1

遇见问题

1、端口号冲突

org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
	at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:65)
Caused by: java.net.BindException: Could not start actor system on any port in port range 6123
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:172)

2、flink集群

1、修改 conf/flink-conf.yaml 文件

参考官网：Set the jobmanager.rpc.address key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the jobmanager.heap.mb and taskmanager.heap.mb keys.

jobmanager.rpc.address 			Master地址
jobmanager.heap.mb 				jobmanager可用内存
taskmanager.heap.mb 			taskmanager可用内存
parallelism.default				任务并行度
io.tmp.dirs 					临时存储目录表

2、修改conf/slaves 文件

添加taskManager节点

3、分发到其他节点