Flink on yarn setup refer link
bin/yarn-session.sh
bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
Start a long-running Flink cluster on YARN
This command will show you the following overview:
Usage:
Required
-n,--container Number of YARN container to allocate (=Number of Task Managers)
Optional
-D Dynamic properties
-d,--detached Start detached
-jm,--jobManagerMemory Memory for JobManager Container with optional unit (default: MB)
-nm,--name Set a custom name for the application on YARN
-q,--query Display available YARN resources (memory, cores)
-qu,--queue Specify YARN queue.
-s,--slots Number of slots per TaskManager
-tm,--taskManagerMemory Memory per TaskManager Container with optional unit (default: MB)
-z,--zookeeperNamespace Namespace to create the Zookeeper sub-paths for HA mode
附着到一个已经存在的Yarn Session
Use the following command to start a session
bin/yarn-session.sh
This command will show you the following overview:
Usage:
Required
-id,--applicationId YARN application Id
As already mentioned, YARN_CONF_DIR or HADOOP_CONF_DIR environment variable must be set to read the YARN and HDFS configuration.
Example: application_1463870264508_0029:
./bin/yarn-session.sh -id application_1463870264508_0029
Attaching to a running session uses YARN ResourceManager to determine Job Manager RPC port. Stop the YARN session by stopping the unix process (using CTRL+C) or by entering ‘stop’ into the client.
Submit Job to Flink
Use the following command to submit a Flink program to the YARN cluster:
bin/flink
./bin/flink run ./examples/batch/WordCount.jar \
hdfs:///..../LICENSE-2.0.txt hdfs:///.../wordcount-result.txt
Run a single Flink job on YARN
The documentation above describes how to start a Flink cluster within a Hadoop YARN environment. It is also possible to launch Flink within YARN only for executing a single job.
Please note that the client then expects the -yn value to be set (number of TaskManagers).
Example:
./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar
The command line options of the YARN session are also available with the ./bin/flink tool. They are prefixed with a y or yarn (for the long argument options).