Hadoop(一)：CentOS 安装hadoop

我的CSDN博客地址： http://blog.csdn.net/qidong7

为Hadoop集群安装SSH
一、验证是否安装了ssh

[hadoop-user@brain /]# which ssh
/usr/bin/ssh
[hadoop-user@brain /]# which sshd
/usr/sbin/sshd
[hadoop-user@brain /]# which ssh-keygen
/usr/bin/ssh-keygen

如果收到类似这样的错误信息

/usr/bin/which no ssh in (/usr/bin:/bin:/usr/bin...

您可以通过Linux安装包管理器安装OpenSSH(www.openssh.com)或者直接下载源码。

二、生成SSH密钥对
务必要避免输入口令，否则，主节点每次试图访问其他节点时，你都得手动输入这个口令

[hadoop-user@brain /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop-user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop-user/.ssh/id_rsa.
Your public key has been saved in /home/hadoop-user/.ssh/id_rsa.pub.
The key fingerprint is:
2e:e1:91:cf:85:48:f9:92:8f:2d:e5:36:92:15:e0:1a root@brain
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . o         |
|    E + .        |
|     + = o       |
|    . B S .      |
|     . / .       |
|      B X        |
|       = .       |
|                 |
+-----------------+

生成密钥对之后，公钥的形式为：

[hadoop-user@brain /]# 
more  /home/hadoop-user/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0pmwcaf/xgUrDQfK4ZePQEqjXhYPSV2iUsRxunk20RxerLOpw5SNJ+3gb2jVSnuncAlABkS
yi8VCCJVmV4b0y6kCP6ap9jdcRJhxLB33zd5IyHMyo9zjdo8j2xRnFtv1SfaqkhIt8/By1E2+Si3dlIE61DhRGtdxGJNkVv0O4+8AANKcEk
C5LK8Hr+pEBBVlM9m9OHYJ9GdekQcMQQC/CtrZohzHznDm70tJ1EtnidYdUcBz31FuOfeJjuEapbkn6Az9HwDEQD8cemtED0QK/aW9FLjko
zid5nLjJ7ZJwmkmq5V3rLSE2AbqGnGtYSYCezJFFHmUpNyG35vYjpGj....

三、将密钥公布(本人的target为localhost)

[hadoop-user@brain hadoop-1.2.0]# scp ~/.ssh/id_rsa.pub root@localhost:~/master_key

[hadoop-user@brain /]# ssh localhost
Last login: Sat Jun  1 18:12:59 2013 from 192.168.1.102

四、安装hadoop

1、 解压
tar xzvf hadoop-1.2.0.tar.gz

2、配置 Hadoop
2.1 . 配置JDK
hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_17                                  

2.2 . 配置conf/core-site.xml、conf/hdfs-site.xml、conf/mapred-site.xml文件     
core-site.xml

<configuration>     
<property>  
       <name>fs.default.name</name>  
       <value>hdfs://localhost:9000</value>  
</property>  
<property>
       <name>hadoop.tmp.dir</name>
        <value>/usr/app/hadoop-hdfs</value>
        </property>
</configuration>   

     

hdfs-site.xml

<configuration>     
   <property>  
         <name>dfs.replication</name>  
         <value>1</value>  
    </property>  
</configuration>  

mapred-site.xml

<configuration>     
    <property>  
         <name>mapred.job.tracker</name>  
         <value>localhost:9001</value>  
    </property>  
</configuration>  

3、启动hadoop
cd /home/brain/hadoop-1.2.0/bin
1.格式化文件系统
     ./hadoop namenode –format
2. 启动/关闭 hadoop
    启动所有任务start-all.sh/stop-all.sh
    启动关闭HDFS: start-dfs.sh/stop-dfs.sh
    启动关闭MapReduce:  start-mapred.sh/stop-mapred.sh     
3. 用jps命令查看进程,确保有  
    namenode,
    dataNode,
    JobTracker,
    TaskTracker                              

如果没有namenode ,
1、先运行stop-all.sh
2、格式化namdenode，不过在这之前要先删除原目录，即core-site.xml下配置 的<name>hadoop.tmp.dir</name>所指向的目录，然后运行 hadoop namenode -format

3、运行start-all.sh  

4、检查执行
访问

http://192.168.1.77:50030    (MapReduce的Web页面)

http://192.168.1.77:50070    (HDfS的web页面)

本人安装过程中出现没有namenode，格式化后仍然没有，这时可以查看下日志，发现端口被占。
有问题查看日志 HADOOP_HOME/logs/

另外如果访问

http://192.168.1.77:50030 访问不到，可能是防火墙的原因，可以关闭防火墙，再次访问
service iptables stop

如果是，修改防火墙配置，对50030端口不监控；
修改防火墙监控端口
vi /etc/sysconfig/iptables
中加
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 50030 -j ACCEPT
就不对50030端口进行监控了

请关注微博 http://weibo.com/u/3272022684