hadoop搭建[单机模式+伪分布式]

环境准备

主机名称	IP地址
hadoop-master	192.168.200.100

echo "192.168.200.100 hadoop-master">>/etc/hosts

其他信息：

[root@hadoop-master ~]# uname -r
2.6.32-358.el6.x86_64
[root@hadoop-master ~]# uname -m
x86_64
[root@hadoop-master ~]# cat /etc/redhat-release 
CentOS release 6.4 (Final)
[root@hadoop-master ~]# getenforce 
Disabled

一、安装JDK

请参考：Linux下快速安装JDK

二、安装hadoop

• 创建用户

useradd hadoop && echo 123456 | passwd --stdin hadoop

• 下载

wget --no-check-certificate https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz

• 解压安装

mkdir /application && tar zxf /usr/local/src/hadoop-2.9.0.tar.gz -C /application/&&ln -s /application/hadoop-2.9.0 /application/hadoop&&chown -R hadoop:hadoop /application/hadoop*&&ll /application/

• 配置Hadoop环境变量

##################Hadoop环境变量配置#############
export HADOOP_HOME=/application/hadoop
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

• 测试

hadoop version

三、配置Hadoop

3.1 单机模式（standalone）

• 不需要任何配置，不需要启用独立的hadoop进程，常用语开发和调试。

测试

hdfs dfs -ls /

3.2 伪分布模式

• 伪分布模式是在一台机器上模拟完全分布式，只有一个副本

修改配置文件

• vim /application/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/jdk

• vim /application/hadoop/etc/hadoop/core-site.xml

     hadoop.tmp.dir
     file:/application/hadoop/tmp


     fs.defaultFS
     hdfs://hadoop-master:9000

• vim /application/hadoop/etc/hadoop/hdfs-site.xml

        
             dfs.replication
             1
        
        
             dfs.namenode.name.dir
             file:/application/hadoop/tmp/dfs/name
        
        
             dfs.datanode.data.dir
             file:/application/hadoop/tmp/dfs/data
        

• cp /application/hadoop/etc/hadoop/mapred-site.xml.template /application/hadoop/etc/hadoop/mapred-site.xml
• vim /application/hadoop/etc/hadoop/mapred-site.xml

        
             mapreduce.framework.name
             yarn
        

• vim /application/hadoop/etc/hadoop/yarn-site.xml

    
         yarn.nodemanager.aux-services
         mapreduce_shuffle
    

配置SSH免密登陆

• 配置本机免密码

检查是否安装了ssh相关软件包（openssh-server openssh-clients openssh）
yum list installed | grep ssh
检查是否启动了sshd进程
ps -Af | grep sshd
su - hadoop
在客户端生成密钥对
ssh-keygen -t dsa
将公钥追加到服务端的认证库（本机）
cat .ssh/id_dsa.pub >> .ssh/authorized_keys
检查authorized_keys的权限是否为644，不是的话，需要修改为644

• 测试(如果第一次需要密码，第二次不需要即可)
  ssh hadoop-master

格式化文件系统

hdfs namenode -format

成功字样：has been successfully formatted
Existing with status 0

启动

[hadoop@hadoop-master ~]start-dfs.sh
[hadoop@hadoop-master ~]$ jps
8880 Jps
8593 DataNode
8769 SecondaryNameNode
8454 NameNode
[hadoop@hadoop-master ~]$ start-yarn.sh
[hadoop@hadoop-master ~]$ jps
8593 DataNode
9041 NodeManager
8769 SecondaryNameNode
8454 NameNode
9338 Jps
8939 ResourceManager

测试

浏览器访问：
http://192.168.200.100:50070

测试往HDFS写文件：

echo "123456" >>/home/hadoop/1.txt
bin/hdfs dfs -appendToFile /home/hadoop/1.txt  hdfs://hadoop-master:9000/1.txt
bin/hdfs dfs -ls hdfs://hadoop-master:9000/

四、三种模式共存

配置hadoop，让其三种模式共存，方便切换

1. 分别复制${HADOOP_HOME}/etc/hadoop文件夹3次，分别更名为local、pseudo、full
2. 删除${HADOOP_HOME}/etc/hadoop文件夹
3. 创建对应软连接ln -s