HadoopYarn常用命令、yarn application查看任务、yarn logs查看日志、yarn applicationattempt查看尝试运行的任务、查看容器、Yarn生产环境核心参数
文章目录
- 16. Yarn资源调度器
-
- 16.5 Yarn常用命令
-
- 16.5.1 yarn application查看任务
-
- 16.5.1.1 列出所有Application：
- 16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）
- 16.5.1.3 Kill掉Application：
- 16.5.2 yarn logs查看日志
-
- 16.5.2.1 查询Application日志：
- 16.5.2.2 查询Container日志：
- 16.5.3 yarn applicationattempt查看尝试运行的任务
-
- 16.5.3.1 列出所有Application尝试的列表：
- 16.5.3.2 打印ApplicationAttemp状态：
- 16.5.4 yarn container查看容器
-
- 16.5.4.1 列出所有Application 尝试的列表
- 16.5.4.2 打印Container状态：
- 16.5.5 yarn node查看节点状态
- 16.5.6 yarn rmadmin更新配置
- 16.5.7 yarn queue查看队列
- 16.6 Yarn生产环境核心参数
16. Yarn资源调度器
16.5 Yarn常用命令
Yarn状态的查询，除了可以在hadoop103:8088页面查看外，还可以通过命令操作。常见的命令操作如下所示：
需求：执行WordCount案例，并用Yarn命令查看任务运行情况。
-
[summer@hadoop102 ~]$ myhadoop.sh start
这个是之前写的脚本，想了解的可用看我之前写的文章
https://blog.csdn.net/Redamancy06/article/details/126234179 -
[summer@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /testinput /testoutput/output1
16.5.1 yarn application查看任务
16.5.1.1 列出所有Application：
[summer@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-10-10 16:57:24,620 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
这里因为任务已经执行结束了，所以没有显示出来。
16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）
[summer@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-10-10 17:03:48,178 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
application_1665325770064_0003 word count MAPREDUCE summer default FINISHED SUCCEEDED 100% http://hadoop102:19888/jobhistory/job/job_1665325770064_0003
这里的任务application_1665325770064_0003就是刚刚我们执行的任务
16.5.1.3 Kill掉Application：
[summer@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1665325770064_0003
2022-10-10 17:05:56,667 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application application_1665325770064_0003 has already finished
因为application_1665325770064_0003这个任务已经完成了，所以不能再次被kill了
16.5.2 yarn logs查看日志
16.5.2.1 查询Application日志：
yarn logs -applicationId
[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003
16.5.2.2 查询Container日志：
yarn logs -applicationId -containerId
[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003 -containerId container_1665325770064_0003_01_000001
containerId从16.5.3.1里获取
16.5.3 yarn applicationattempt查看尝试运行的任务
16.5.3.1 列出所有Application尝试的列表：
yarn applicationattempt -list
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0003
2022-10-10 20:01:36,619 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
ApplicationAttempt-Id State AM-Container-Id Tracking-URL
appattempt_1665325770064_0003_000001 FINISHED container_1665325770064_0003_01_000001 http://hadoop103:8088/proxy/application_1665325770064_0003/
16.5.3.2 打印ApplicationAttemp状态：
yarn applicationattempt -status
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1665325770064_0003_000001
2022-10-10 20:11:35,446 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application Attempt Report :
ApplicationAttempt-Id : appattempt_1665325770064_0003_000001
State : FINISHED
AMContainer : container_1665325770064_0003_01_000001
Tracking-URL : http://hadoop103:8088/proxy/application_1665325770064_0003/
RPC Port : 42572
AM Host : hadoop103
Diagnostics :
ApplicationAttemptId从16.5.3.1里获取
16.5.4 yarn container查看容器
16.5.4.1 列出所有Application 尝试的列表
yarn container -list
[summer@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1665325770064_0003_000001
2022-10-10 20:15:25,743 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of containers :0
Container-Id Start Time Finish Time State Host Node Http Address LOG-URL
只有在任务跑的途中才能看到 container 的状态，因为这个任务运行完之后，container容器会立即释放，我这里就没有container容器，因为任务已经运行结束了
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0010
2022-10-10 20:25:08,250 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
ApplicationAttempt-Id State AM-Container-Id Tracking-URL
appattempt_1665325770064_0010_000001 RUNNING container_1665325770064_0010_01_000001 http://hadoop103:8088/proxy/application_1665325770064_0010/
这个靠手速弄出来一个结果，这个就是有容器的时候里面显示的内容
16.5.4.2 打印Container状态：
yarn container -status
[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0003_01_000001
2022-10-10 20:20:13,494 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Container with id ‘container_1665325770064_0003_01_000001’ doesn’t exist in RM or Timeline Server.
注：只有在任务跑的途中才能看到 container 的状态
[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0010_01_000001
2022-10-10 20:25:14,642 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Container Report :
Container-Id : container_1665325770064_0010_01_000001
Start-Time : 1665404700780
Finish-Time : 0
State : RUNNING
Execution-Type : GUARANTEED
LOG-URL : http://hadoop104:8042/node/containerlogs/container_1665325770064_0010_01_000001/summer
Host : hadoop104:36495
NodeHttpAddress : http://hadoop104:8042
Diagnostics : null
这个靠手速弄了一个任务在运行中的显示结果
16.5.5 yarn node查看节点状态
列出所有节点：yarn node -list -all
[summer@hadoop102 hadoop-3.1.3]$ yarn node -list -all
2022-10-10 20:33:40,711 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total Nodes:3
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
hadoop104:36495 RUNNING hadoop104:8042 0
hadoop103:38375 RUNNING hadoop103:8042 0
hadoop102:40891 RUNNING hadoop102:8042 0
16.5.6 yarn rmadmin更新配置
加载队列配置：yarn rmadmin -refreshQueues
[summer@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
2022-10-10 20:39:09,817 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8033
16.5.7 yarn queue查看队列
打印队列信息：yarn queue -status
[summer@hadoop102 hadoop-3.1.3]$ yarn queue -status default
2022-10-10 20:43:10,934 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Queue Information :
Queue Name : default
State : RUNNING
Capacity : 100.0%
Current Capacity : .0%
Maximum Capacity : 100.0%
Default Node Label expression :
Accessible Node Labels : *
Preemption : disabled
Intra-queue Preemption : disabled
这里的值更详细
16.6 Yarn生产环境核心参数
