hadoop2.x安装

hadoop2.x安装

1.搭建虚拟机（使用桥接网络，以便设置ssh连接，注意版本）
2.安装jdk
3.编辑hosts的文件
4.关闭防火墙
5.部署免密码登录
6.下载hadoop2.x并解压
7.修改配置文件
8.并分发hadoop到各个节点
9.启动集群

修改配置文件，涉及到的配置文件有：
hadoop2.2/etc/hadoop/hadoop-env.xml
  修改JAVA_HOME
hadoop2.2/etc/hadoop/yarn-env.xml
  修改JAVA_HOME
hadoop2.2/etc/hadoop/slaves
  写入各个主机的名称
hadoop2.2/etc/hadoop/core-site.xml
    

<property>
       <name>fs.defaultFS</name>
       <value>hdfs://master:9000</value>
   </property>
   <property>
       <name>io.file.buffer.size</name>
       <value>131072</value>
   </property>
   <property>
       <name>hadoop.tmp.dir</name>
       <value>file:/opt/hadoopdfs/tmp</value>
       <description>Abase for other temporary directories.</description>
   </property>
      <property>
       <name>hadoop.proxyuser.hduser.hosts</name>
       <value>*</value>
   </property>
   <property>
       <name>hadoop.proxyuser.hduser.groups</name>
       <value>*</value>
   </property>

hadoop2.2/etc/hadoop/hdfs-site.xml
    

<property>
       <name>dfs.namenode.name.dir</name>
       <value>file:/opt/hadoopdfs/name</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
       <value>file:/opt/hadoopdfs/data</value>
   </property>
   <property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
hadoop2.2/etc/hadoop/mapred-site.xml
   <property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
     <final>true</final>
   </property>
   <property>
     <name>mapred.job.tracker</name>
     <value>master:9001</value>
   </property>

hadoop2.2/etc/hadoop/yarn-site.xml
  

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>

----------------------------------------------------------------------------------
格式化namenode:/bin/hdfs namenode -format
启动hdfs:/sbin/start-dfs.sh
    此时在master上进程有： NameNode，ResourceManager
    datanode上进程有： datanode
启动yarn:/sbin/start-yarn.sh
    此时在master上进程有： NameNode，ResourceManager，SecondaryNameNode
    datanode上进程有： DataNode，NodeManager