环境准备
| 主机名称 | IP地址 |
|---|---|
| s101 | 192.168.200.101 |
| s102 | 192.168.200.102 |
| s103 | 192.168.200.103 |
| s104 | 192.168.200.104 |
- 操作系统信息
[root@hadoop-master ~]# uname -r
2.6.32-358.el6.x86_64
[root@hadoop-master ~]# uname -m
x86_64
[root@hadoop-master ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
[root@hadoop-master ~]# getenforce
Disabled
- 所有主机/etc/hosts文件添加[s101~s104]
192.168.200.101 s101
192.168.200.102 s102
192.168.200.103 s103
192.168.200.104 s104
配置root免密码
登陆到s101服务器，并切换到root用户，su - root
- 在s101主机上生成密钥对
[root@s101 ~]# ssh-keygen
一直回车
- 配置本机登陆面密码
[root@s101 ~]#cd /root/.ssh/
[root@s101 .ssh]#cp id_rsa.pub authorized_keys
测试本机是否免密码成功，如果直登陆上，则说明配置成功
[root@s101 ~]ssh s101
- 将s101的公钥复制到到s102~104主机上
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s102
输入yes，然后输入s102的root密码，同理s103和s104
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s103
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s104
分别测试s102~s104是否免密码成功，如果直登陆上，则说明配置成功
[root@s101 ~]ssh s102
创建hadoop用户，配置hadoop用户免密码登陆
- 创建用户[s101~s104]
useradd hadoop && echo 123456 | passwd --stdin hadoop
- 配置免密码登陆【原理同root配置，此处简写】
在s101上生成密钥对[s101]
[hadoop@s101 ~]$ ssh-keygen
hadoop@s101 ~]$ cd /home/hadoop/.ssh/
[hadoop@s101 .ssh]$ cp id_rsa.pub authorized_key
[hadoop@s101 .ssh]$ ssh s101
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s102
[hadoop@s101 .ssh]$ ssh s102
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s103
[hadoop@s101 .ssh]$ ssh s103
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s104
[hadoop@s101 .ssh]$ ssh s104
给Hadoop用户安装配置JDK
- 解压安装
[hadoop@s101 app]$ tar zxf /app/jdk-8u144-linux-x64.tar.gz -C /app/
[hadoop@s101 app]$ ln -s /app/jdk1.8.0_144 /app/jdk
- 配置hadoop用户的java环境变量
[hadoop@s101 app]$echo -e '##################JAVA环境变量配置#############\nexport JAVA_HOME=/app/jdk\nexport JRE_HOME=$JAVA_HOME/jre\nexport CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH\nexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH' >> ~/.bash_profile && source ~/.bash_profile&&tail -5 ~/.bash_profile
- 检查s101是否配置成功
[hadoop@s101 app]$ java -version
安装Hadoop
- 准备Hadoop安装目录
[root@s101 ~]# mkdir /app
[root@s101 ~]# chown -R hadoop:hadoop /app
[root@s101 ~]# ll -d /app/
drwxr-xr-x 2 hadoop hadoop 4096 Nov 5 15:42 /app/
- 下载安装，首先在s101上安装，然后发送到其他机器
[root@s101 home]# su - hadoop
[hadoop@s101 ~]$ cd /app/
[hadoop@s101 app]$ wget --no-check-certificate https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz
[hadoop@s101 app]$ tar zxf /app/hadoop-2.9.0.tar.gz -C /app/
[hadoop@s101 app]$ ln -s /app/hadoop-2.9.0 /app/hadoop
- 配置hadoop环境变量
echo -e '######Hadoop环境变量配置######\nexport HADOOP_HOME=/app/hadoop\nexport PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH' >> /home/hadoop/.bash_profile && source /home/hadoop/.bash_profile&&tail -3 /home/hadoop/.bash_profile
配置hadoop集群配置相关文件
- [hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/app/jdk
- [hadoop@s101 hadoop]$ vim /app/hadoop/etc/hadoop/slaves
s102
s103
s104
- [hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/core-site.xml
fs.defaultFS
hdfs://s101:9000
- [hadoop@s101 ~]$vim /app/hadoop/etc/hadoop/hdfs-site.xml
dfs.replication
3
- hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/mapred-site.xml
mapreduce.framework.name
yarn
- [hadoop@s101 hadoop]$ vim /app/hadoop/etc/hadoop/yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.hostname
s101
分发文件给s102-s104
- hadoop和jdk
[hadoop@s101 /]$ rsync -lr app hadoop@s102:/
[hadoop@s101 /]$ rsync -lr app hadoop@s103:/
[hadoop@s101 /]$ rsync -lr app hadoop@s104:/
- hadoop环境变量
[hadoop@s101 /]$ cd /home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s102:/home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s103:/home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s104:/home/hadoop/
- 检查s102-s104服务器上对应文件是否分发成功
[hadoop@s101 /home/hadoop]$ll /app/
[hadoop@s101 /home/hadoop]$cat /home/hadoop/.bash_profile
启动hadoop集群
格式化文件系统
[hadoop@s101 /home/hadoop]$hdfs namenode -format
启动集群
执行start-dfs.sh
[hadoop@s101 /home/hadoop]$start-dfs.sh
Starting namenodes on [s101]
s101: starting namenode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-namenode-s101.out
s103: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s103.out
s104: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s104.out
s102: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s102.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-secondarynamenode-s101.out
- 通过jps查看，s101存在进程NameNode和SecondaryNameNode
[hadoop@s101 /home/hadoop]$jps
24392 NameNode
24589 SecondaryNameNode
- s102~s104存在DataNode
[hadoop@s102 /home/hadoop]$jps
3969 DataNode
[hadoop@s103 /home/hadoop]$jps
3896 DataNode
[hadoop@s104 /home/hadoop]$jps
3936 DataNode
执行start-yarn.sh
[hadoop@s101 /home/hadoop]$start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-resourcemanager-s101.out
s104: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s104.out
s103: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s103.out
s102: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s102.out
- 通过jps查看，s101存在进程NameNode、SecondaryNameNode和ResourceManager（新增）
[hadoop@s101 /home/hadoop]$jps
24392 NameNode
24589 SecondaryNameNode
24797 ResourceManager
- s102~s104存在DataNode和NodeManager（新增）
[hadoop@s102 /home/hadoop]$jps
3969 DataNode
4105 NodeManager
[hadoop@s103 /home/hadoop]$jps
4032 NodeManager
3896 DataNode
[hadoop@s104 /home/hadoop]$jps
3936 DataNode
4072 NodeManager
- 访问 http://192.168.200.101:50070/dfshealth.html#tab-overview
image.png