Hadoop 安装 教程

    Hadoop作为一个开源的分布式计算框架，现在在国内是越来越火了。包括淘宝、百度等大公司，都在大批量的使用hadoop。之前对hadoop稍微了解了一点，今天把Hadoop的分布式安装大致介绍一下。

1. 安装前的准备工作
1. Java JDK 1.6.x
2. ssh 以及 sshd。保证各个服务器之前的免密码访问
2. Hadoop下载：http://labs.renren.com/apache-mirror//hadoop/core/hadoop-0.21.0/hadoop-0.21.0.tar.gz
3. 这个包需要在hadoop集群的所有服务器上下载并解压缩
4. 我们计划将整个集群安装在4台服务器上。服务器名字分别为myna[5-8].其中myna5为Namenode, myna6为TaskTracker，myna[5-6]即为master，其余两台myna[7-8]为slaver。
5. 配置这四台服务器的环境变量
export JAVA_HOME=/path/to/bin/java
export HADOOP_HOME=~/hadoop-0.21.0

6. 配置myna5(NameNode)和myna6(TaskTracker):
1. ～/hadoop-0.21.0/conf/core-site.xml:
< property >
     < name >fs.default.name name >
     < value >hdfs://myna5:54320 value >
     < description >The name of the default file system.    A URI whose
    scheme and authority determine the FileSystem implementation.    The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class.    The uri's authority is used to
    determine the host, port, etc. for a filesystem. description >
property >


2. ～/hadoop-0.21.0/conf/hdfs-site.xml:
< property >
     < name >hadoop.tmp.dir name >
     < value >/home/hrj/hadooptmp/hadoop-${user.name} value >
     < description >A base for other temporary directories. description >
property >
< property >
         < name >dfs.upgrade.permission name >
         < value >777 value >
property >

< property >
         < name >dfs.umask name >
         < value >022 value >
property>


3. ～/hadoop-0.21.0/conf/mapred-site.xml:
< property >
     < name >mapred.job.tracker name >
     < value >myna6:54321 value >
     < description >The host and port that the MapReduce job tracker runs
    at.    If "local", then jobs are run in-process as a single map
    and reduce task.
     description >
property >
< property >
     < name >mapred.compress.map.output name >
     < value >true value >
     < description >Should the outputs of the maps be compressed before being
                             sent across the network. Uses SequenceFile compression.
     description >
property >
< property >
         < name >mapred.child.java.opts name >
         < value >-Xmx1024m value >
property >



4. 只需要配置myna5的masters
myna6


5. 配置myna5和myna6的slavers
myna7
myna8


7. 配置myna7和myna8(Slavers)
1. ～/hadoop-0.21.0/conf/core-site.xml:
< property >
         < name >fs.default.name name >
         < value >hdfs://myna5:54320 value >
         < description >The name of the default file system.        A URI whose
        scheme and authority determine the FileSystem implementation.        The
        uri's scheme determines the config property (fs.SCHEME.impl) naming
        the FileSystem implementation class.        The uri's authority is used to
        determine the host, port, etc. for a filesystem. description >
property >


2. ～/hadoop-0.21.0/conf/hdfs-site.xml:
< property >
         < name >hadoop.tmp.dir name >
         < value >/disk/hadooptmp/hadoop-${user.name} value >
property >
< property >
         < name >dfs.data.dir name >
         < value >/home/hadoopdata,/disk/hadoopdata value >
property >
< property >
         < name >dfs.upgrade.permission name >
         < value >777 value >
property >
< property >
         < name >dfs.umask name >
         < value >022 value >
property>

3. ～/hadoop-0.21.0/conf/mapred-site.xml:
< property >
     < name >mapred.job.tracker name >
     < value >myna6:54321 value >
     < description >The host and port that the MapReduce job tracker runs
    at.    If "local", then jobs are run in-process as a single map
    and reduce task.
     description >
property >
< property >
     < name >mapred.compress.map.output name >
     < value >true value >
     < description >Should the outputs of the maps be compressed before being
                             sent across the network. Uses SequenceFile compression.
     description >
property >
< property >
         < name >mapred.child.java.opts name >
         < value >-Xmx1024m value >
property >
< property >
         < name >mapred.tasktracker.map.tasks.maximum name >
         < value >4 value >
     property >
     < property >
         < name >mapred.tasktracker.reduce.tasks.maximum name >
         < value >2 value >
     property >


8. 编辑myna[5-8]的~/hadoop-0.21.0/conf/hadoop-env.sh,，添加JAVA_HOME环境变量
export JAVA_HOME=/path/to/bin/java
9. 至此，Hadoop cluster配置完成。启动Hadoop：
1. 在myna5上启动HDFS
hrj$ ~/hadoop-0.21.0/bin/hadoop namenode -format  
hrj$ ~/hadoop-0.21.0/bin/start-dfs.sh


2. 在myna6上启动MapRed
hrj$ ~/hadoop-0.21.0/bin/start-mapred.sh


10. 启动HDFS后，你可以通过前端页面访问：
1. 访问HDFS：http://myna5:50070
2. 访问MapRed：http://myna6:50030

更加详细的安装帮助及参数配置信息，请大家访问英文官方网 http://hadoop.apache.org/common/docs/r0.21.0/cluster_setup.html