hadoop完全分布式安装

安装环境如下

centos6.0 32位操作系统，jdk目前最新版jdk1.7,hadoop为0.20.2

第一步：安装环境准备工作

a、在现有的两台exsi服务器准备实验环境，

b、安装配置三台虚拟机，操作系统为censos6.0操作系统，配置内存1GB，硬盘20GB,安装一台后，另外两台使用克隆的方式进行安装

c、服务器的Ip地址、服务器名称如下，注意三台服务器的hosts文件下面需加上对方的服务器名称和IP的解析，使三台服务器可以相互之前在IP地址和名称间切换

    hadoop-1:172.168.16.61

    hadoop-2:172.168.16.62

    hadoop-3:172.168.16.63

    配置如下：

    [root@hadoop-1 src]#hostname hadoop-1

    [root@hadoop-1 src]# vim /etc/sysconfig/network
     NETWORKING=yes  
     HOSTNAME=hadoop-1

    [root@hadoop-1 src]#vim /etc/hosts

      127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
       ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
      172.168.16.61   hadoop-1
       172.168.16.62   hadoop-2
       172.168.16.63   hadoop-3

c、下载最新版jdk

[root@hadoop-1 src]#wget http://download.oracle.com/otn-pub/java/jdk/7u21-b11/jdk-7u21-linux-i586.tar.gz

[root@hadoop-1 src]# ls
debug  hadoop-0.20.2.tar.gz  jdk-7u21-linux-i586.tar.gz  kernels

[root@hadoop-1 src]#tar xvf  jdk-7u21-linux-i586.tar.gz -C /usr/local/

d、配置环境变量，在/etc/profile最后面加上如下内容，三台服务器的JAVA环境都需这样配置，使java环境能正常运行。

[root@hadoop-1 src]#vim /etc/etcprofile

export JAVA_HOME=/usr/local/jdk1.7.0_21
export JAVR_JRE=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

[root@hadoop-1 src]#source /etc/profile

e、检查java是否正常运行

[root@hadoop-1 src]#echo $JAVA_HOME

/usr/local/jdk1.7.0_21

[root@hadoop-1 src]# java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) Client VM (build 23.21-b01, mixed mode)

实验环境准备工作完成

第一步、配置host文件，已在准备工作中完成

第二步、配置hadoop用户

[root@hadoop-1 src]# useradd hadoop

[root@hadoop-2 ~]# useradd hadoop

[root@hadoop-3 ~]# useradd hadoop

第三步、配置ssh免密码登入

[root@hadoop-1 src]#su hadoop

[hadoop@hadoop-1 root]$ ssh-keygen -t rsa   此步骤一路回车

[hadoop@hadoop-1 root]$ls /home/hadoop/.ssh/
authorized_keys  id_rsa  id_rsa.pub  known_hosts             生成id_rsa和id_rsa.pub文件

[hadoop@hadoop-1 root]$cp /home/hadoop/.ssh/id_rsa.pub /home/hadoop/.ssh/authorized_keys

同样在haddoop-2执行以上步骤

 

[root@hadoop-2]#su hadoop

[hadoop@hadoop-2 root]$ ssh-keygen -t rsa   此步骤一路回车

[hadoop@hadoop-2 root]$ls /home/hadoop/.ssh/
authorized_keys  id_rsa  id_rsa.pub  known_hosts             生成id_rsa和id_rsa.pub文件

[hadoop@hadoop-2 root]$cp /home/hadoop/.ssh/id_rsa.pub /home/hadoop/.ssh/authorized_keys

同样在haddoop-3执行以上步骤

[root@hadoop-3]#su hadoop

[hadoop@hadoop-3 root]$ ssh-keygen -t rsa   此步骤一路回车

[hadoop@hadoop-3 root]$ls /home/hadoop/.ssh/
authorized_keys  id_rsa  id_rsa.pub  known_hosts             生成id_rsa和id_rsa.pub文件

[hadoop@hadoop-3 root]$cp /home/hadoop/.ssh/id_rsa.pub /home/hadoop/.ssh/authorized_keys

 

把hadoop-1中id_rsa.pub中的内容追加到hadoop_2和hadoop_3中的authorized_keys文件中。注意追加的时候要用">>"重定向功能，请不要用直接拷的方式，易有问题

[root@hadoop-1 src]# scp /home/hadoop/.ssh/id_rsa.pub 172.168.16.62:/home/hadoop

[root@hadoop-1 src]# scp /home/hadoop/.ssh/id_rsa.pub 172.168.16.63:/home/hadoop

[hadoop@hadoop-2 ~]$ cat id_rsa.pub >> .ssh/authorized_keys

同样在用上面的方法。把hadoop_2和hadoop_3上面的内容追加到另两台不同的机器。方法在此不再列出，配置后文件内容样式如下

[hadoop@hadoop-2 ~]$ cat .ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqaahAX+SuzUlpBeyUoODd51NvUqQfGZKjrC60lUR76FCrRs3wPMDITES9TF86MK4xFk0bzNuK+WZVleq9ZilnOnxJsyz7NoaqOwwy5ACMjsRDMM0C5dFQ21xAODP6jDQ1LsCve0yHeuW6MlbKVERC94LRE5oTt3RFH7gxSMrDmMIOoIFDJXjEYDmHM6/kN7hmUiEH6X6k5sBwQA1dUaIORjy6zUV/4Sz+QPsQmF558V+Lw/CO2EdGYAgw97CHMxbybIG+b9A5IlCw+47d+zcdrX2vUUF1VGxnTlw4OYZCbfYqhvvpE1F9UY3+0RTCAuayGBCqWIFMd06KV2Np9FYfw== hadoop@hadoop-2
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1jyyWKj8/DgqTa0UZkDSX/12Vky/eQXmHccLmmwNSye1bjfGrotX4p05EFT46lzRsLlixwtWF4iWv2kLg/5bn4JJ83MWBW+ANcrqZLdF/lS97xa928lSq7ry4D00wSgLR9fybqo/wv7midn8mxZeI92jbSzMYE/6I5eyRb5GNySFSpGjnxkO0a9QvRSSvgJDZrQ80JNiw6FGUiRacf6kzP1/6qJwWPJnUgHHso/oQN66cmBtjZuCDy7/OGBwjJ1iHgjO8fnAdI3bmTPn7X3LslEUVPFoAXE1XciVM9Mk0Xh8Ixpc50XMG8jKboh4SdSu0QcGOI0R4Yy7rRDNt2QqcQ== hadoop@hadoop-1
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuvT4KuNQeKwarbdvNiCEpUktNzpocvQsGjWYkwWbsU/M2fxyPYrUzgQqfF/NGXeEvf8BzWVgV7pQH9/Ajg2bOUOafcwubLIiimw+wzraQ4MGQERMYKOdd6Su+w+yR5vpohY/x6S5lMiYgmaBTNVhgitD9GjuVX/N5Mbn0c5sTt/TlWSMfgKOp6hNORWlf01JaTyKcCpap+I9gBtAq4vPD1YppvYyrfv9TeW8NdcVVxswGE6XHxPD2b1/+JyBLYE3zN5XfWWaIfqC8gBxJ4brHNxdBFMp+IQ8LJXRyAklwd882P9qxXNFEE/IqFtwm8PvxlV2Ad4APptfDgdRreyWXQ== hadoop@hadoop-3
[hadoop@hadoop-2 ~]$ 

检验ssh是否名密码验证，同时这样做的目的把第一次连结的提示“yes\no”去掉，一定要进行一次本机的连结，进行自身对自身的验证.

[root@hadoop-1 src]# ssh 172.168.16.61
Last login: Fri Apr 26 02:45:56 2013 from hadoop-1
[root@hadoop-1 ~]#

[root@hadoop-1 src]# ssh 172.168.16.62
Last login: Fri Apr 26 02:46:01 2013 from hadoop-1
[root@hadoop-2 ~]# 
[root@hadoop-1 src]# ssh 172.168.16.63
Last login: Fri Apr 26 01:17:26 2013 from aca81034.ipt.aol.com
[root@hadoop-3 ~]# 
第四步、下载并解压hadoop

[hadoop@hadoop-1 ~]$ wget http://archive.apache.org/dist/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz

[hadoop@hadoop-1 ~]$tar xvf hadoop-0.20.2.tar.gz -C /opt/

[hadoop@hadoop-1 ~]$ls /opt/

[hadoop@hadoop-1 ~]$ ls -al /opt/                                    注意，此步骤是在root权限下完成。完成后需对hadoop-0.20.2目录进行权限更改
total 12
drwxr-xr-x.  3 root   root   4096 Apr 25 12:56 .
dr-xr-xr-x. 22 root   root   4096 Apr 25 03:32 ..
drwxr-xr-x. 15 hadoop hadoop 4096 Apr 25 13:58 hadoop-0.20.2

更改hadoop目录的权限

[hadoop@hadoop-1 ~]#chown -R hadoop:hadoop /opt/hadoop-0.20.2

 

同以上步骤在hadoop-2和hadoop-3上面执行

第五步、修改配置文件

a、修改hadoop-env文件

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/hadoop-env.sh 
# Set Hadoop-specific environment variables here.


# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.


# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/local/jdk1.7.0_21

b、修改core-site.xml，配置namenode节，此配置文件中，注意配置tmp目录，以免重启hadoop服务器，造成服务器不能启动

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/core-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-1:9000</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-0.20.2/tmp</value>
<description>A base for other temporary directory</description>
</property>
</configuration>
c、修改hdfs-site.xml文件，配置数据目录和name目录。存入name节点数据和文件数据，同时配置复制数量为“2”，因这里只有两台datanode

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/hdfs-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>
<property>
<name>dfs.name.dir</name>
<value>/opt/hadoop-0.20.2/hdfs/name</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/opt/hadoop-0.20.2/hdfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>
</configuration>
d、配置jobtacker节点。

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoop-1:9001</value>
<final>true</final>
</property>
</configuration>

e、配置senondarynode节点

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/masters 
hadoop-1
f、配置datanode和tasktracter节点

[hadoop@hadoop-1 ~]$ cat /opt/hadoop-0.20.2/conf/slaves 
hadoop-2
hadoop-3

以上配置文件的修改，需同时在hadoop-2和hadoop-3上面进行，并保证所有内容是一样的。

 

第六步、配置环境变量，为了以后使用上的方便，需把hadoop的目录做为环境变量设置，具体配置后的效果如下

[hadoop@hadoop-1 ~]$ cat /etc/profile

 export JAVA_HOME=/usr/local/jdk1.7.0_21
export JAVR_JRE=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_INSTALL/bin:$PATH
export HADOOP_INSTALL=/opt/hadoop-0.20.2

 

第七步：格式化hadoop

[hadoop@hadoop-1 ~]$

hadoop  namenode -format

 

第七步：启动hadoop

[hadoop@hadoop-1 ~]$start-all.sh

 

第八步：验证hadoop

[hadoop@hadoop-1 ~]$ jps
5948 SecondaryNameNode
6019 JobTracker
5802 NameNode
6784 Jps

[hadoop@hadoop-2 ~]$ jps
4199 TaskTracker
9288 Jps
4111 DataNode
[hadoop@hadoop-2 ~]$ 
[hadoop@hadoop-3 root]$ jps
6673 Jps
1591 TaskTracker
1512 DataNode
[hadoop@hadoop-3 root]$

[hadoop@hadoop-1 ~]$ /opt/hadoop-0.20.2/bin/hadoop dfsadmin -report
Configured Capacity: 37073182720 (34.53 GB)
Present Capacity: 32527679488 (30.29 GB)
DFS Remaining: 32527589376 (30.29 GB)
DFS Used: 90112 (88 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0


-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)


Name: 172.168.16.62:50010
Decommission Status : Normal
Configured Capacity: 18536591360 (17.26 GB)
DFS Used: 45056 (44 KB)
Non DFS Used: 2272829440 (2.12 GB)
DFS Remaining: 16263716864(15.15 GB)
DFS Used%: 0%
DFS Remaining%: 87.74%
Last contact: Fri Apr 26 03:18:45 CST 2013




Name: 172.168.16.63:50010
Decommission Status : Normal
Configured Capacity: 18536591360 (17.26 GB)
DFS Used: 45056 (44 KB)
Non DFS Used: 2272673792 (2.12 GB)
DFS Remaining: 16263872512(15.15 GB)
DFS Used%: 0%
DFS Remaining%: 87.74%
Last contact: Fri Apr 26 03:18:46 CST 2013

 

也可以通过以下验证

http://172.168.16.61:50030/jobtracker.jsp

 

http://172.168.16.61:50070/jobtracker.jsp