一、上次课程回顾
二、Hadoop第二次课
三、本次课程作业
1、参考连接网址:
1、拷贝一份做备份:
[hadoop@hadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml
2、vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
3、编辑yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
###### 注意
//如果开放8088端口,一般为了避免机器被挖矿,直接修改yarn的端口号;进入到yarn-default.xml,搜索8088,在yarn-site.xml中增加如下:
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop001:38088</value>
</property>
2、由于我们sbin已经配置了环境变量,所以使用which start-yarn.sh能够直接找到
[hadoop@hadoop001 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-hadoop-resourcemanager-hadoop001.out
hadoop001: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-hadoop-nodemanager-hadoop001.out
[hadoop@hadoop001 hadoop]$ jps
2240 Jps
15074 NameNode
1779 ResourceManager
15206 DataNode
15384 SecondaryNameNode
1881 NodeManager
[hadoop@hadoop001 hadoop]$ netstat -nlp|grep 1779
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 :::8030 :::* LISTEN 1779/java
tcp 0 0 :::8031 :::* LISTEN 1779/java
tcp 0 0 :::8032 :::* LISTEN 1779/java
tcp 0 0 :::8033 :::* LISTEN 1779/java
tcp 0 0 ::ffff:172.17.0.5:38088 :::* LISTEN 1779/java
1、案例演示:
1、find查找hadoop目录下存在的jar包:
[hadoop@hadoop001 hadoop]$ find ./ -name "*example*.jar"
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
2、一步步的使用如下这个命令:
[hadoop@hadoop001 hadoop]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount /wordcount/input/wordcount.log /wordcount/output
3、不通过web界面的话如何知道这个作业是运行成功的呢?
1、查看到这个文件是0字节
[hadoop@hadoop001 data]$ hdfs dfs -ls /wordcount/output
20/03/24 23:32:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 hadoop supergroup 0 2020-03-24 23:27 /wordcount/output/_SUCCESS
4、使用命令查看在hdfs上的输出结果:
[hadoop@hadoop001 data]$ hdfs dfs -cat /wordcount/output/part-r-00000
20/03/24 23:34:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hello 3
john 1
world 2
HDFS:存储 计算的结果又返回存储在HDFS上
MR jar包:计算逻辑
Yarn :资源+作业调度
存储:HDFS(分布式文件系统)、Hive、HBase、Kudu、Cassandra(不算是大数据组件)
计算:MR编程、后面会使用Hivesql进行演示,Spark、Flink
资源+作业调度:Yarn
1、CentOS6中修改:
1、vi /etc/sysconfig/network
# Created by cloud-init on instance boot automatically, do not edit.
#
NETWORKING=yes
HOSTNAME=hadoop001
2、在CentOS中修改:
hostnamectl set-hostname hadoop001
完了之后都需要进行重启。
1、jps进程标识文件在哪儿看?
1、j指的是Java,ps就是获取Java的命令:
[root@hadoop001 ~]# which jps
/usr/java/jdk1.8.0_45/bin/jps
2、jps -l 展示出更详细的信息:
[hadoop@hadoop001 ~]$ jps -l
15074 org.apache.hadoop.hdfs.server.namenode.NameNode
1779 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
15206 org.apache.hadoop.hdfs.server.datanode.DataNode
15384 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
1881 org.apache.hadoop.yarn.server.nodemanager.NodeManager
7738 sun.tools.jps.Jps
3、对应的进程的标识文件在哪儿?
[root@hadoop001 hsperfdata_hadoop]# pwd
/tmp/hsperfdata_hadoop
[root@hadoop001 hsperfdata_hadoop]# ll
total 160
-rw------- 1 hadoop hadoop 32768 Mar 24 23:52 15074
-rw------- 1 hadoop hadoop 32768 Mar 24 23:52 15206
-rw------- 1 hadoop hadoop 32768 Mar 24 23:52 15384
-rw------- 1 hadoop hadoop 32768 Mar 24 23:52 1779
-rw------- 1 hadoop hadoop 32768 Mar 24 23:52 1881
//我们的进程是hadoop用户启动的,会存储在/tmp目录下,hsperfdata_hadoop(hadoop用户)
2、root用户和hadoop用户下使用jps的区别?
1、root用户下:
[root@hadoop001 hsperfdata_hadoop]# jps
15074 -- process information unavailable
1779 -- process information unavailable
8454 Jps
15206 -- process information unavailable
15384 -- process information unavailable
1881 -- process information unavailable
2、hadoop用户下:
[hadoop@hadoop001 hadoop]$ jps
15074 NameNode
1779 ResourceManager
15206 DataNode
15384 SecondaryNameNode
1881 NodeManager
8539 Jps
[root@hadoop001 hsperfdata_hadoop]# ps -ef | grep 1779 | grep -v grep | wc -l
1
[root@hadoop001 hsperfdata_hadoop]# mv 15074 /home/hadoop/data/
//不影响进程和停止,移除走后,jps也查不到15074的记录了;
[hadoop@hadoop001 hadoop]$ jps
1779 ResourceManager
15206 DataNode
15384 SecondaryNameNode
1881 NodeManager
10397 Jps
模拟测试:
1、kill -9 1779
//杀掉resourcemanager进程
2、jps发现1779进程已经没了,进入到这个目录,1779进程二进制文件也没了
[hadoop@hadoop001 tmp]$ cd hsperfdata_hadoop/
[hadoop@hadoop001 hsperfdata_hadoop]$ ll
total 96
-rw------- 1 hadoop hadoop 32768 Mar 25 00:16 15206
-rw------- 1 hadoop hadoop 32768 Mar 25 00:15 15384
-rw------- 1 hadoop hadoop 32768 Mar 25 00:16 1881
3、进入到/tmp目录,发现到这个文件记录的还是1779pid
[hadoop@hadoop001 tmp]$ pwd
/tmp
[hadoop@hadoop001 tmp]$ cat yarn-hadoop-resourcemanager.pid
1779
4、再次启动我们杀掉的yarn进程:
[hadoop@hadoop001 tmp]$ jps
11825 Jps
15206 DataNode
15384 SecondaryNameNode
1881 NodeManager
11501 ResourceManager
[hadoop@hadoop001 tmp]$ cat yarn-hadoop-resourcemanager.pid
11501
//发现已经启动了新的pid
如上实验发现的问题:
举例:我们2G的机器,去安装CDH,然后启动mysql服务,然后启动一瞬间就挂了,原因是系统机制直接把它杀了,所以在log中也找不到相关信息。
以后遇到进程挂了,首先去找到log的位置,定位到error在哪,有error的话具体分析;没有error的话可以去/var/log/messages中查看系统日志文件。cat /var/log/messages | grep oom、cat /var/log/secure | grep oom
补充一共4点:
1、配置完环境变量which一下
2、生产上动配置文件之前cp一下
3、kill人家的进程之前确保服务是不是在用
4、linux oom的kill机制了解清楚
1、搭建yarn的伪分布式部署
2、运行mr wordcount的命令
3、jps的真真假假判别
4、linux的oom机制、定期clean机制