Hadoop---集群安装

  在一个全配置的集群上，“运行Hadoop”意味着在网络分布的不同服务器上运行一组daemons。这些daemons包括：

•   NameNode
•   DataNode
•   Secondary NameNode
•   JobTracker
•   TaskTracker

  NameNode位于HDFS的主端，指导从端的DataNode执行底层的I/O任务。

  DataNode将HDFS数据块读取或者写入到本地系统的实际文件中

  Secondary NameNode是一个用于监测HDFS集群状态的辅助daemon。

  JobTracker是应用程序和Hadoop之间的纽带。一旦提交代码到集群上，JobTracker就会确定执行计划，包括决定处理哪些文件、为不同的任务分配节点以及监控所有任务的运行。

  TaskTracker管理各个任务在每个从节点上的执行情况。

  下图为Hadoop集群的典型拓扑图。这是一个主/从架构，其中NameNode和JobTracker为主端，DataNode和TaskTracker为从端。

  

  安装一个Hadoop集群时，需要专门指定一个服务器作为主节点。这个服务器通常会运行NameNode和JobTracker。它也将作为一个基站，负责联络并激活所有从节点上的DataNode和TaskTracker。Hadoop使用SSH协议来通信。

  安装Hadoop集群前，要先定义一个公共账号。这里使用xiaoruoen

  安装Hadoop集群时要先验证SSH是否安装。

  如果没有安装，使用yum install openssh来安装，如下

[xiaoruoen@localhost ~]# yum install openssh
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * extras: mirrors.163.com
 * updates: mirrors.163.com
Setting up Install Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: openssh = 4.3p2-36.el5 for package: openssh-clients
--> Processing Dependency: openssh = 4.3p2-36.el5 for package: openssh-askpass
--> Processing Dependency: openssh = 4.3p2-36.el5 for package: openssh-server
---> Package openssh.i386 0:4.3p2-82.el5 set to be updated
--> Running transaction check
---> Package openssh-askpass.i386 0:4.3p2-82.el5 set to be updated
---> Package openssh-clients.i386 0:4.3p2-82.el5 set to be updated
---> Package openssh-server.i386 0:4.3p2-82.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================
 Package                                  Arch                          Version                                  Repository                     Size
=====================================================================================================================================================
Updating:
 openssh                                  i386                          4.3p2-82.el5                             base                          291 k
Updating for dependencies:
 openssh-askpass                          i386                          4.3p2-82.el5                             base                           42 k
 openssh-clients                          i386                          4.3p2-82.el5                             base                          455 k
 openssh-server                           i386                          4.3p2-82.el5                             base                          275 k

Transaction Summary

  成功安装SSH之后，使用主节点上的ssh-keygen来生成一RSA密钥对。

[xiaoruoen@localhost ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/xiaoruoen/.ssh/id_rsa):
Created directory '/home/xiaoruoen/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/xiaoruoen/.ssh/id_rsa.
Your public key has been saved in /home/xiaoruoen/.ssh/id_rsa.pub.
The key fingerprint is:
33:47:bc:c0:cf:82:92:d2:69:3a:73:f1:3a:14:74:f8 xiaoruoen@localhost.localdomain
[xiaoruoen@localhost ~]$ more /home/xiaoruoen/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAnYAiD7YCrZR3L8Ps1KKqy1FoA3azcnrFP481zJfGGVKC2judapiZJcjca48B99Zqa4jNyRgREAzTQHxbSRKkQqnA1TB5BK7sKrXp6yl3lCQ/E4k05i6aLHU6DImufWJlmbBr
3ILrxG+sNiD/ihcZ66o74/lXUemTHcNomRLEUgJoI8U6X2PiTmD5HqW7VxelseJ2yIZ8cfx9n+0MQbEQBj5Nme8JR7XBFEGWFzySfp2hWR9SfTwqcRRuiVgEczics4ebdk22SsP13lAdltSS/mqJfVG2G6neVBTt+E0+0AI8
27DBza9oY2bMiR0HCCihEU2vuLTaKpAzigna1/h2nQ== xiaoruoen@localhost.localdomain

  将公钥分布到从服务器上

[xiaoruoen@localhost ~]$ scp ~/.ssh/id_rsa.pub [email protected]:~/master_key
The authenticity of host '192.168.0.107 (192.168.0.107)' can't be established.
RSA key fingerprint is d1:ac:46:8f:97:32:0b:88:82:2b:14:60:b0:e6:70:c4.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added '192.168.0.107' (RSA) to the list of known hosts.
[email protected]'s password:
id_rsa.pub                                                                                                                            100%  413     0

  登录到从服务器上进行验证

[xiaoruoen@localhost ~]$ mkdir ~/.ssh
[xiaoruoen@localhost ~]$ chmod 700 ~/.ssh
[xiaoruoen@localhost ~]$ mv ~/master_key ~/.ssh/authorized_keys
[xiaoruoen@localhost ~]$ chmod 600 ~/.ssh/authorized_keys

  尝试从主节点登录到目标节点来验证它的正确性

[xiaoruoen@localhost ~]$ ssh 192.168.0.107
Last login: Thu Apr 12 22:07:21 2012 from 192.168.0.102

  到Hadoop根目录下的conf目录下找到hadoop-env.sh，修改JAVA_HOME

[xiaoruoen@localhost ~]$ cd $HADOOP_HOME/conf
[xiaoruoen@localhost conf]$ ll
total 140
-rw-rw-r-- 1 root root 7457 Mar 25 08:01 capacity-scheduler.xml
-rw-rw-r-- 1 root root  535 Mar 25 08:01 configuration.xsl
-rw-rw-r-- 1 root root  178 Mar 25 08:01 core-site.xml
-rw-rw-r-- 1 root root  327 Mar 25 08:01 fair-scheduler.xml
-rw-rw-r-- 1 root root 2237 Mar 25 08:01 hadoop-env.sh
-rw-rw-r-- 1 root root 1488 Mar 25 08:01 hadoop-metrics2.properties
-rw-rw-r-- 1 root root 4644 Mar 25 08:01 hadoop-policy.xml
-rw-rw-r-- 1 root root  178 Mar 25 08:01 hdfs-site.xml
-rw-rw-r-- 1 root root 4441 Mar 25 08:01 log4j.properties
-rw-rw-r-- 1 root root 2033 Mar 25 08:01 mapred-queue-acls.xml
-rw-rw-r-- 1 root root  178 Mar 25 08:01 mapred-site.xml
-rw-rw-r-- 1 root root   10 Mar 25 08:01 masters
-rw-rw-r-- 1 root root   10 Mar 25 08:01 slaves
-rw-rw-r-- 1 root root 1243 Mar 25 08:01 ssl-client.xml.example
-rw-rw-r-- 1 root root 1195 Mar 25 08:01 ssl-server.xml.example
-rw-rw-r-- 1 root root  382 Mar 25 08:01 taskcontroller.cfg

[xiaoruoen@localhost conf]$ vim hadoop-env.sh

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
export JAVA_HOME=/home/samba/jdk1.6.0_30

  Hadoop的设置主要包含在XML配置文件中。在0.20版本前，它们是hadoop-default.xml和hadoop-site.xml。在版本0.20中，这hadoop-site.xml被分离成3个xml文件：core-site.xml，hdfs-stie.xml与mapred-site.xml。

  分布式运行hadoop，就要修改这三个文件

  在core-site.xml配置NameNode

xml version="1.0"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
<property>
 <name>fs.default.namename>
 <value>hdfs://master:9000value>
property>

configuration>

  在mapred-site.xml中配置JobTracker

xml version="1.0"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
<property>
 <name>mapred.job.trackername>
 <value>hdfs://master:9001value>
property>

configuration>

 在hdfs-site.xml中配置增大HDFS备份参数

xml version="1.0"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
<property>
 <name>dfs.replicationname>
 <value>3value>
property>

configuration>

更新masters和slaves文件来指定共它daemon的位置。

设定主服务器所在位置，也就是NameNode与JobTracker所在位置

vim masters
localhost

设定从服务器所在位置，也就是DataNode与TaskTracker所在位置

vim slaves
192.168.0.108
192.168.0.109

格式化HDFS文件系统

hadoop namenode -format

(如果提示未找到类，可以在~/.bash_profile里设置CLASSPATH为$HADOOP_HOME/hadoop-core-*.jar;$HADOOP_HOME/hadoop-tool-*.jar;)

启动所有的节点

./start-all.sh

 使用JPS来查看节点

jps
30879 JobTracker
30717 NameNode
30965 jps

jps
7101 TaskTracker
7617 Jps
6988 DataNode