Ubuntu上使用Hadoop 2.x 三 编译2.2.0 64bit版本
下载源代码：
wget http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
然后解压：
tar zxvf hadoop-2.2.0-src.tar.gz cd hadoop-2.2.0-src
运行下面的命令开始编译：
~/code/hadoop-2.2.0-src$ mvn package -Pdist,native -DskipTests -Dtar下载了很多maven的东东后，编译报错：
[ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle [ERROR] class file for org.mortbay.component.LifeCycle not found
编辑hadoop-common-project/hadoop-auth/pom.xml文件，添加依赖：
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
再次编译，这个错误解决了。 注意，2.3.0之后不需要修改pom.xml，这个问题已经解决。
之后遇到了没有安装protocol buffer库的错误，安装一下：
[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common --- [WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directory
sudo apt-get install libprotobuf-dev
hduser@hm1:/usr/src$ sudo wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz ... $ sudo ./configure $ sudo make $ sudo make check $ sudo make install $ sudo ldconfig $ protoc --version
hduser@hm1:~$ start-dfs.sh Starting namenodes on [localhost] localhost: namenode running as process 983. Stop it first. localhost: datanode running as process 1101. Stop it first. Starting secondary namenodes [0.0.0.0] 0.0.0.0: secondarynamenode running as process 1346. Stop it first.
再来编译。等...., 说明hadoop不准备64bit的库浪费了大家多少时间。
还有错：
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
安装openssl库
sudo apt-get install libssl-dev
再来一次，编译成功了。在目录/home/hduser/code/hadoop-2.2.0-src/hadoop-dist/target 下有文件：
hadoop-2.2.0.tar.gz解压后，进入目录，然后复制native 目录里的东西到制定位置，覆盖32bit文件
sudo cp -r ./hadoop-2.2.0/lib/native/* /usr/local/hadoop/lib/native/
现在回到~目录，运行下面的命令，成功了。
hduser@hm1:~$ start-dfs.sh Starting namenodes on [localhost] localhost: namenode running as process 983. Stop it first. localhost: datanode running as process 1101. Stop it first. Starting secondary namenodes [0.0.0.0] 0.0.0.0: secondarynamenode running as process 1346. Stop it first.
hduser@hm1:~$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hm1.out localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hm1.out
hduser@hm1:~$ jps 32417 Jps 1101 DataNode 1346 SecondaryNameNode 983 NameNode
现在运行例子程序， 注意/etc/hosts中我配置了
127.0.0.1 hm1
hduser@hm1:~/code/hadoop-2.2.0-src$ cd /usr/local/hadoop/ hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5 Number of Maps = 2 Samples per Map = 5 Wrote input for Map #0 Wrote input for Map #1 Starting Job 14/02/21 18:07:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/02/21 18:07:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC\ fountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/02/21 18:07:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumC\ ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/02/21 18:07:32 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumC\ ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
需要在yarn-site.xml中添加写配置：
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
重新启动虚拟机，再次启动服务，现在连接问题解决了。
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5 Number of Maps = 2 Samples per Map = 5 Wrote input for Map #0 Wrote input for Map #1 Starting Job 14/02/21 18:15:53 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032 14/02/21 18:15:54 INFO input.FileInputFormat: Total input paths to process : 2 14/02/21 18:15:54 INFO mapreduce.JobSubmitter: number of splits:2 14/02/21 18:15:54 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.specul\ ative 14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/02/21 18:15:54 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/02/21 18:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393006528872_0001 14/02/21 18:15:54 INFO impl.YarnClientImpl: Submitted application application_1393006528872_0001 to ResourceManager at /127.0.0.1:8032 14/02/21 18:15:55 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1393006528872_0001/ 14/02/21 18:15:55 INFO mapreduce.Job: Running job: job_1393006528872_0001
如果重来一次的话，顺序应该是先编译64bit版本，然后用这个版本进行配置安装。