Hadoop2.2.0伪分布式环境搭建（附：64位下编译Hadoop-2.2.0过程）

Hadoop2.2.0伪分布式环境搭建：

    写在前面：Hadoop2.2.0默认是支持32位的OS，如果想要在64位OS下运行的话，可以通过在64位OS下面编译Hadoop2.2.0来实现，编译的操作步骤在最后面呈现。

1：

操作：下载软件；

文件：Hadoop-2.2.0.tar.gz

步骤：

     到http://mirrors.cnnic.cn/apache/hadoop/common/下面选择对应的版本号下载

     或者使用编译过后的支持64位OS的包

    hadoop-2.2.0.x86_64.tar.gz_免费高速下载|百度云网盘-分享无限制

http://pan.baidu.com/s/1eQAB4rg

2：

操作：设置环境变量；

文件：/etc/profile

步骤：

     sudovim /etc/profile

     向文件尾部添加以下内容：

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

export JRE_HOME=$JAVA_HOME/jre

export HADOOP_HOME2=/home/rocketeer/Hadoop/hadoop-2.2.0

export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME2/bin:$HADOOP_HOME2/sbin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/jre/lib:$CLASSPATH

 

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

 

 

3：

操作：设置主机名（本条必须配置，不然在格式化namenode的时候会报不能识别主机名的错误）

文件：/etc/hostname  

      /etc/hosts

步骤：

     sudo vim /etc/hostname

     打开vim编辑器，把里面的内容修改为：

     rocketeer

 

    sudovim /etc/hosts

     将开头两行改为以下内容

     127.0.0.1  localhost.localdomain localhost

     127.0.0.1  new-hostname.localdomain new-hostname

 

     或者改为

     127.0.0.1  localhost

     192.168.159.148 rocketeer

 

验证：

     然后重启虚拟机即可，重启之后输入命令查看是否修改成功：

     Hostname

     Hostname –f

     输入以上两条命令，看是否都返回最新的主机名rocketeer

 

4：

操作：设置无密码登陆SSH

步骤：

sudo apt-getinstall ssh

安装完成后会在~目录（当前用户主目录，即这里的/home/hduser）下产生一个隐藏文件夹.ssh（ls  -a 可以查看隐藏文件）。如果没有这个文件，自己新建即可： 

mkdir .ssh

cd .ssh

ssh-keygen -t rsa之后一路回车（产生秘钥）

id_rsa.pub 追加到授权的 key 里面去

cat id_rsa.pub >> authorized_keys

重启 SSH服务命令使其生效

service ssh restart

最后执行

ssh localhost

     实现无密码连接localhost

 

5：

操作：hadoop配置文件修改

文件：/home/rocketeer/Hadoop/hadoop-2.2.0/etc/hadoop下面的文件

      core-site.xml  hadoop-env.sh hdfs-site.xml  masters  slaves mapred-site.xml 

      yarn-site.xml

配置文件的添加和修改都在中

步骤：

     cd Hadoop/hadoop-2.2.0/etc/hadoop

（1）通过core-site.xml配置namenode和tmp目录

     sudo vim core-site.xml

     添加以下内容

     

         

           fs.defaultFS

           hdfs://localhost:9000

         

 

         

           hadoop.tmp.dir

            /home/rocketeer/Hadoop/hadoop2_tmp

         

     

 

（2）通过hadoop-env.sh配置java的路径（必须配置）

   

  export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

 

（3）通过hdfs-site.xml配置namenode和 datanode存储路径

     sudo vim hdfs-site.xml

     添加以下内容

     

           dfs.namenode.name.dir

           ${hadoop.tmp.dir}/namenode

         

        

         

          dfs.datanode.data.dir

           ${hadoop.tmp.dir}/datanode

         

 

（4）通过mapred-site.xmlhadoop2.0有了yarn所以原来的mapred配置都转向yarn-site.xml文件中了，这里也就指定yarn。

    mvmapred-site.xml.template mapred-site.xml

    sudo vim mapred-site.xml

    添加以下内容

      

         mapreduce.job.tracker

         http://127.0.0.1:9001

     

 

     

         mapreduce.framework.name

         yarn

     

 

     

         mapreduce.system.dir

         /mapred/system

         true

     

 

     

         mapred.local.dir

         /mapred/local

         true

     

 

（5）yarn-site.xml为了简单，快速做测试，使用默认的即可。

（6）masters其实这里可以不配置，其实是指定secondenamenode

（7）slaves指定datanode和tasknode

（8）由于是伪分布式，所以master和slaves都是localhost，如果是真的分布式，这里只想的主机是不一样的，可以和链接里面的博客对比

 

6：

操作：格式化namenode

步骤：

    Cd Hadoop/hadoop-2.2.0/bin

    在这个目录下面输入命令：

    hdfs namenode –format

    显示下列信息，则说明格式化成功：

   rocketeer@rocketeer:~/Hadoop/hadoop-2.2.0/bin$ hdfs namenode -format

14/07/12 23:02:20 INFOnamenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = rocketeer/202.106.199.36

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.2.0

STARTUP_MSG:   classpath =/home/rocketeer/Hadoop/hadoop-2.2.0/etc/hadoop:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/home/rocketeer/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/contrib/capacity-scheduler/*.jar

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common-r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z

STARTUP_MSG:   java = 1.7.0_55

************************************************************/

14/07/12 23:02:20 INFOnamenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

OpenJDK 64-Bit Server VM warning:You have loaded library/home/rocketeer/Hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which mighthave disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that youfix the library with 'execstack -c ', or link it with '-znoexecstack'.

14/07/12 23:02:48 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable

14/07/12 23:02:49 WARN common.Util:Path $HADOOP_HDFS2/namenode should be specified as a URI in configurationfiles. Please update hdfs configuration.

14/07/12 23:02:49 WARN common.Util:Path $HADOOP_HDFS2/namenode should be specified as a URI in configurationfiles. Please update hdfs configuration.

Formatting using clusterid:CID-e6deb9bd-356a-4dea-ba0e-d3af695a6c52

14/07/12 23:02:49 INFOnamenode.HostFileManager: read includes:

HostSet(

)

14/07/12 23:02:50 INFOnamenode.HostFileManager: read excludes:

HostSet(

)

14/07/12 23:02:50 INFOblockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000

14/07/12 23:02:50 INFO util.GSet:Computing capacity for map BlocksMap

14/07/12 23:02:50 INFO util.GSet:VM type       = 64-bit

14/07/12 23:02:50 INFO util.GSet:2.0% max memory = 966.7 MB

14/07/12 23:02:50 INFO util.GSet:capacity      = 2^21 = 2097152 entries

14/07/12 23:02:50 INFOblockmanagement.BlockManager: dfs.block.access.token.enable=false

14/07/12 23:02:50 INFOblockmanagement.BlockManager: defaultReplication         = 3

14/07/12 23:02:50 INFOblockmanagement.BlockManager: maxReplication             = 512

14/07/12 23:02:50 INFOblockmanagement.BlockManager: minReplication             = 1

14/07/12 23:02:50 INFOblockmanagement.BlockManager: maxReplicationStreams      = 2

14/07/12 23:02:50 INFOblockmanagement.BlockManager: shouldCheckForEnoughRacks  = false

14/07/12 23:02:50 INFOblockmanagement.BlockManager: replicationRecheckInterval = 3000

14/07/12 23:02:50 INFOblockmanagement.BlockManager: encryptDataTransfer        =false

14/07/12 23:02:50 INFOnamenode.FSNamesystem: fsOwner            = rocketeer (auth:SIMPLE)

14/07/12 23:02:50 INFOnamenode.FSNamesystem: supergroup         = supergroup

14/07/12 23:02:50 INFOnamenode.FSNamesystem: isPermissionEnabled = true

14/07/12 23:02:50 INFOnamenode.FSNamesystem: HA Enabled: false

14/07/12 23:02:50 INFOnamenode.FSNamesystem: Append Enabled: true

14/07/12 23:02:51 INFO util.GSet:Computing capacity for map INodeMap

14/07/12 23:02:51 INFO util.GSet:VM type       = 64-bit

14/07/12 23:02:51 INFO util.GSet:1.0% max memory = 966.7 MB

14/07/12 23:02:51 INFO util.GSet:capacity      = 2^20 = 1048576 entries

14/07/12 23:02:51 INFOnamenode.NameNode: Caching file names occuring more than 10 times

14/07/12 23:02:51 INFO namenode.FSNamesystem:dfs.namenode.safemode.threshold-pct = 0.9990000128746033

14/07/12 23:02:51 INFOnamenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0

14/07/12 23:02:51 INFOnamenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000

14/07/12 23:02:51 INFOnamenode.FSNamesystem: Retry cache on namenode is enabled

14/07/12 23:02:51 INFOnamenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cacheentry expiry time is 600000 millis

14/07/12 23:02:51 INFO util.GSet:Computing capacity for map Namenode Retry Cache

14/07/12 23:02:51 INFO util.GSet:VM type       = 64-bit

14/07/12 23:02:51 INFO util.GSet:0.029999999329447746% max memory = 966.7 MB

14/07/12 23:02:51 INFO util.GSet:capacity      = 2^15 = 32768 entries

Re-format filesystem in StorageDirectory /home/rocketeer/Hadoop/hadoop-2.2.0/bin/$HADOOP_HDFS2/namenode ? (Yor N) y

14/07/12 23:03:46 INFOcommon.Storage: Storage directory/home/rocketeer/Hadoop/hadoop-2.2.0/bin/$HADOOP_HDFS2/namenode has beensuccessfully formatted.

14/07/12 23:03:46 INFOnamenode.FSImage: Saving image file/home/rocketeer/Hadoop/hadoop-2.2.0/bin/$HADOOP_HDFS2/namenode/current/fsimage.ckpt_0000000000000000000using no compression

14/07/12 23:03:46 INFOnamenode.FSImage: Image file /home/rocketeer/Hadoop/hadoop-2.2.0/bin/$HADOOP_HDFS2/namenode/current/fsimage.ckpt_0000000000000000000of size 201 bytes saved in 0 seconds.

14/07/12 23:03:46 INFOnamenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

14/07/12 23:03:46 INFO util.ExitUtil:Exiting with status 0

14/07/12 23:03:46 INFOnamenode.NameNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting downNameNode at rocketeer/202.106.199.36

************************************************************/

 

 

7：

（1）启动hadoop-2.2.0集群

Start-all.sh

 

（2）分步启动：

启动namenode

 

1      sbin/hadoop-daemon.sh start namenode

2      sbin/hadoop-daemon.sh start datanode

 

运行测试

 

1      jps

 

出现：

12935NameNode

5309 Jps

13012DataNode

 

证明启动成功，如果没有出现DataNode或者NameNode，证明启动没有成功，可以查看hadoop安装目录下的logs下的日志记录。

可以使用sbin/hadoop-daemon.shstop datanode（或namenode）来关闭。

 

启动Manage管理

 

1      sbin/yarn-daemon.sh start resourcemanager

2      sbin/yarn-daemon.sh start nodemanager

 

启动mapreduce history查看进程

mr-jobhistory-daemon.shstart historyserver

 

运行测试

 

1      jps

 

出现：

13338NodeManager

13111ResourceManager

12935NameNode

5309 Jps

13012DataNode

 

证明启动成功同时也可以使用yarn-daemon.shstop resourcemanager（nodemanager）来关闭。

 

如果没有单独配置yarn-site.xml中的yarn.resourcemanager.webapp.address，默认的端口8088访问

http://127.0.0.1:8088/ 可以访问hadoop管理页面

 

如果没有单独配置hdfs-site.xml中的dfs.namenode.http-address,默认端口50070

http://127.0.0.1:50070可以访问namenode节点信息。

 

Historyserver访问

http://127.0.0.1:19888

 

 

 

64位下编译Hadoop-2.2.0过程：

参考文章：

在Ubuntu12.04 64位上编译hadoop2.2.0-你若幸福，便是晴天 -博客频道 - CSDN.NET

http://blog.csdn.net/tianfei_1984/article/details/20030383

hadoop前戏配置三：hadoop2.2.0重新编译为64位，个人测试成功

http://www.360doc.com/content/14/0625/11/7839419_389587052.shtml

 

sudoapt-get install g++ autoconf automake libtool make cmake zlib1g-dev pkg-configlibssl-dev

 

下载protobuf-2.5.0.tar.gz到虚拟机

解压，依次运行

$./configure --prefix=/usr

$ sudomake

$ sudomake check

$ sudomake install

protoc--version

检查一下版本

 

ubuntu下用apt-get安装maven

$ sudoapt-get install maven

 

编译 hadoop 2.2.0

下载 hadoop 2.2.0 http://www.apache.org/dyn/closer.cgi/hadoop/common/

解压到用户目录 /home/rocketeer/Downloads

 

目前的2.2.0的Source Code压缩包解压出来的code有个bug需要patch后才能编译。否则编译hadoop-auth会提示上面错误。

解决办法如下：

修改下面的pom文件。该文件在hadoop源码包下寻找：

hadoop-common-project/hadoop-auth/pom.xml

打开上面的的pom文件，在54行加入如下的依赖：

    

      org.mortbay.jetty

     jetty-util

      test

    

    

       org.mortbay.jetty

      jetty

       test

    

 

进入 hadoop-2.2.0-src目录

因为已经安装了maven,protobuf, java环境也有了，compiler也有了所以直接运行

$ mvnpackage -Pdist,native -DskipTests -Dtar

 

然后重新运行编译指令即可。编译是一个缓慢的过程，耐心等待哦。

当看到下面的信息时，编译成功。

安装配置 hadoop 2.2.0

此时编译好的文件位于hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/目录中