hadoop安装配置及问题

废话少说，lets do it
一、安装配置

（1）下载安装包：
wget http://labs.renren.com/apache-mirror//hadoop/core/hadoop-1.0.0/hadoop-1.0.0.tar.gz 

（2）解压：tar xzvf hadoop-1.0.0.tar.gz 

（3）配置文件（集群）(datanode和namenode配置一致)
         1)conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


<property>
<name>dfs.name.dir</name>
<value>/localdisk/hadoop/hdfs/name</value>
<description>  </description>
</property>


<property>
<name>dfs.data.dir</name>
<value>/localdisk/hadoop/hdfs/data</value>
<description> </description>
</property>


<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>


</configuration>


By default, HDFS doesn’t require DataNodes to have any reserved free space. In
practice, most systems have questionable stability when the amount of free space gets
too low. You should set dfs.datanode.du.reserved to reserve 1 GB of free space in a
DataNode. A DataNode will stop accepting block writes when its amount of free space
falls below the reserved amount.

      2)conf/mapred.xml
 
               <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


<property>
<name>mapred.job.tracker</name>
<value>192.168.137.1:9001</value>
</property>


<property>
<name>mapred.child.java.opt</name>
<value>-Xmx2048m</value>
</property>


<property>
  <name>mapred.tasktracker.map.tasks.maximum</name>
  <value>24</value>
  <description>The maximum number of map tasks that will be run
  simultaneously by a task tracker.
  </description>
</property>


<property>
  <name>mapred.tasktracker.reduce.tasks.maximum</name>
  <value>24</value>
  <description>The maximum number of reduce tasks that will be run
  simultaneously by a task tracker.
  </description>
</property>


</configuration>

      3)conf/core-site.xml

             <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


        <property>
                <name>fs.default.name</name>
                <value>hdfs://192.168.137.1:9100</value>
        </property>


</configuration>

4)conf/slaves
#localhost
#mpi001
mpi002
mpi003
mpi004
mpi005
mpi006
mpi007
mpi008
mpi009
mpi010
mpi011
mpi012
mpi013
mpi014
mpi015
mpi016
mpi017
mpi018
mpi019
mpi020
mpi021
mpi022
mpi023
mpi024
mpi025
mpi026
mpi027
mpi028
mpi029
mpi030
mpi031
mpi032

5)conf/env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
export HADOOP_HOME=/opt/hadoop-1.0.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib

二、启动hadoop

1）hadoop namenode -format
2）start-all.sh



三、实例

1）准备好wordcount例子，使用fatjar打包成jar文件

2）准备原始文件
  a. 创建两个本地文本文件，
  b. 在hdfs上创建目录  hadoop fs -mkdir  wordcount/input， hadoop fs -mkdir  wordcount/output1（默认会创建output，为防止冲突叫output1）
  c. 将本地文件上传到hdfs上：
hadoop fs -put input1.txt wordcount/input
hadoop fs -put input2.txt wordcount/input
  d. 运行程序
hadoop jar /root/workspace/WordCount/WordCount_fat.jar wordcount/input wordcount/output1
  e. 查看结果
 hadoop fs -cat  wordcount/output1/part-00000


 

四、问题

（1）杀不掉问题
stop-all.sh 不能够杀掉计算节点上的hdfs进程，用一下命令：ps aux |grep hadoop|awk '{print $2}'|xargs kill -9，为了方便起见，做了一个脚本进行批杀

#!/bin/bash
#Node=( c202 c203 c204 c205 c206 c207 c208 c209 c210 c211 c212 c213 c214 c215 c216)
#Node=(mpi017 mpi018 mpi019 mpi020 mpi021 mpi022 mpi023 mpi024 mpi025 mpi026 mpi027 mpi028 mpi029 mpi030 mpi031 mpi032)
Node=( mpi002 mpi003 mpi004 mpi005 mpi006 mpi007 mpi008 mpi009 mpi011 mpi012 mpi013 mpi014 mpi015 mpi016 mpi017 mpi018 mpi019 mpi020 mpi021 mpi022 mpi023 mpi024 mpi025 mpi026 mpi027 mpi028 mpi029 mpi030 mpi031 mpi032)
for((i=0;i<30;i++))
   do


ssh   ${Node[i]}  'ps aux |grep hadoop|awk "{print $2}"|xargs kill -9'


done


（2）配置冲突问题
我们在hdfs-site.xml和core-site.xml中都配置了“fs.default.name”，但端口号不一样，最终起作用的是hdfs-site中的value，namenode起来后可以先查看一下namenode机器上你设置的端口号是否起作用：比如：lsof -i:54301，再看datanode上的节点日志，确定是否访问的是namenode上你所指定的端口号？
tail -f  /opt/hadoop-1.0.0/logs/hadoop-root-datanode-mpi002.log
如果重新更改了配置文件，可能需要重新启动hadoop，先stop-all 再start-all

（3）Incompatible nameID

创建目录没有问题，一旦有实质性的文件上传，就会抛出错误：

2/12/06 21:14:12 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/root/testlog/input/apache_access.log could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1556)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

查看数据节点的日志：

[root@mpi002 ~]# tail /opt/hadoop-1.0.0/logs/hadoop-root-datanode-mpi002.log 
2012-12-06 21:18:52,310 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /localdisk/hadoop/hdfs/data: namenode namespaceID = 1338088189; datanode namespaceID = 1917596014


这个问题重新犯了好几次，年龄大了记性不好，正解如下，就是删除掉所有datanode的数据目录下的内容，或者参考下面文章中的第二种方案

http://blog.csdn.net/wh62592855/article/details/5752199

为了方便删除，脚本如下（前提，所有节点无密码互通）

#!/bin/bash
#Node=( c202 c203 c204 c205 c206 c207 c208 c209 c210 c211 c212 c213 c214 c215 c216)
#Node=(mpi017 mpi018 mpi019 mpi020 mpi021 mpi022 mpi023 mpi024 mpi025 mpi026 mpi027 mpi028 mpi029 mpi030 mpi031 mpi032)
Node=( mpi002 mpi003 mpi004 mpi005 mpi006 mpi007 mpi008 mpi009 mpi011 mpi012 mpi013 mpi014 mpi015 mpi016 mpi017 mpi018 mpi019 mpi020 mpi021 mpi022 mpi023 mpi024 mpi025 mpi026 mpi027 mpi028 mpi029 mpi030 mpi031 mpi032)
for((i=0;i<30;i++))
   do

ssh   ${Node[i]}  'rm -rf /localdisk/hadoop/hdfs/data'

done

上述脚本中， /localdisk/hadoop/hdfs/data为数据节点的存储数据的目录，其配置文件为 vi  $HADOOP_HOME/conf/hdfs-site.xml

<property>
<name>dfs.data.dir</name>
<value>/localdisk/hadoop/hdfs/data</value>
<description> </description>
</property>

hadoop-env.sh： Bash script Environment variables that are used in the scripts to run Hadoop.

core-site.xml Hadoop configuration XML： Configuration settings for Hadoop Core, such as I/O settings that are common to HDFS and MapReduce.

hdfs-site.xml Hadoop configuration XML： Configuration settings for HDFS daemons: the namenode, the secondary namenode, and the datanodes.

mapred-site.xml Hadoop configuration XML： Configuration settings for MapReduce daemons: the jobtracker, and the tasktrackers.

masters Plain text A list of machines (one per line) that each run a secondary namenode.

slaves Plain text A list of machines (one per line) that each run a datanode and a tasktracker