下载：http://mirrors.cnnic.cn/apache/hadoop/common/
http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-1.2.1/
安装：http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html
mr 例子：http://hi.baidu.com/whyang2006/item/436a720c4e15a013addc70c4
主启动、停止：
$ start-all.sh
$ stop-all.sh
[root@Master ~]# su - hadoop -bash-3.2$ /usr/hadoop/bin/start-all.sh starting namenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-namenode-Master.Hadoop.out 192.168.1.101: starting datanode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-datanode-Slave1.Hadoop.out 192.168.1.100: starting secondarynamenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-Master.Hadoop.out starting jobtracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-Master.Hadoop.out 192.168.1.101: starting tasktracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-Slave1.Hadoop.out
安装成功测试：
主服务器：
-bash-3.2$ jps
10030 JobTracker
9790 NameNode
10143 Jps
9949 SecondaryNameNode
从服务器
-bash-3.2$ jps
11579 TaskTracker
11501 DataNode
11660 Jps
问题1：不能启动从节点
2013-12-24 00:49:01,198 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories are writable
2013-12-24 00:48:59,819 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: can not create directory: /usr/hadoop/tmp/dfs/data
解决：chown -R hadoop:hadoop hadoop
ssh无密传输：
su - hadoop
生成非对称加密的私钥和公钥
ssh-keygen -t rsa -P ''
一路回车，默认值。
运行命令后会再/home/hadoop 目录下生成.ssh的隐藏文件夹，运行ll -a 查看是否生成成功
[hadoop@Master ~]$ ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
c0:e3:2e:a5:38:a3:0d:25:0a:50:93:57:28:4d:b9:60 [email protected]
[hadoop@Master ~]$ ll -a
total 16
drwx------ 3 hadoop hadoop 4096 Dec 19 07:44 .
drwxr-xr-x 4 root root 4096 Aug 16 2012 ..
drwx------ 2 hadoop hadoop 4096 Dec 19 07:44 .ssh
修改授权文件的权限
chmod 600 ~/.ssh/authorized_keys
切换到root用户修改sshd_config文件，使ssh支持无密码登录
su - root
vi /etc/ssh/sshd_config
去掉注释
RSAAuthentication yes # 启用 RSA 认证
PubkeyAuthentication yes # 启用公钥私钥配对认证方式
AuthorizedKeysFile .ssh/authorized_keys #公钥文件路径(和上面生成的文件同
将master节点上的公钥复制到slave节点的/home/hadoop目录下
scp ~/.ssh/id_rsa.pub [email protected]:~/
hadoop用户连接slave机器
创建.ssh的隐藏文件夹
mkdir ~/.ssh
修改权限
chmod 700 ~/.ssh
将master的公钥信息加入到slave的授权文件中
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
完成从slave ssh登录master的过程，master同样需要slave的公钥id_rsa.pub
测试ssh成功：
ssh 192.168.1.101
如果没有提示输入密码则无密码登录设置成功否则设置失败
pig hive http://zhidao.baidu.com/link?
url=6N3qTl2WrtMZ6cl1sPekVrTjdFxtSBTGvQGEuMwuEK0WmAVWhCVmayS-JzfkZiu0GqFWm1WKXXdrsPmHfQVttK
# hadoop job -list
0 jobs currently running
JobId State StartTime UserName Priority SchedulingInfo
--------------
参考： http://blog.csdn.net/yy_done/article/details/7006489
cp /usr/hadoop/src/examples/org/apache/hadoop/examples/WordCount.java ~
mkdir classes
javac -classpath /usr/hadoop/hadoop-core-1.2.1.jar:/usr/hadoop/lib/commons-cli-1.2.jar -d classes WordCount.java
jar -cvf wordcount.jar -C classes/ .
hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output
-bash-3.2$ hadoop fs -put file01 input
-bash-3.2$ hadoop fs -put file02 input
-bash-3.2$ hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output
14/02/23 23:28:09 INFO input.FileInputFormat: Total input paths to process : 2
14/02/23 23:28:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/02/23 23:28:09 WARN snappy.LoadSnappy: Snappy native library not loaded
14/02/23 23:28:10 INFO mapred.JobClient: Running job: job_201402232134_0015
14/02/23 23:28:11 INFO mapred.JobClient: map 0% reduce 0%
14/02/23 23:28:29 INFO mapred.JobClient: map 100% reduce 0%
14/02/23 23:28:37 INFO mapred.JobClient: map 100% reduce 33%
14/02/23 23:28:38 INFO mapred.JobClient: map 100% reduce 100%
14/02/23 23:28:40 INFO mapred.JobClient: Job complete: job_201402232134_0015
14/02/23 23:28:40 INFO mapred.JobClient: Counters: 29
14/02/23 23:28:40 INFO mapred.JobClient: Job Counters
14/02/23 23:28:40 INFO mapred.JobClient: Launched reduce tasks=1
14/02/23 23:28:40 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=33728
14/02/23 23:28:40 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/02/23 23:28:40 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/02/23 23:28:40 INFO mapred.JobClient: Launched map tasks=2
14/02/23 23:28:40 INFO mapred.JobClient: Data-local map tasks=2
14/02/23 23:28:40 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9437
14/02/23 23:28:40 INFO mapred.JobClient: File Output Format Counters
14/02/23 23:28:40 INFO mapred.JobClient: Bytes Written=33
14/02/23 23:28:40 INFO mapred.JobClient: FileSystemCounters
14/02/23 23:28:40 INFO mapred.JobClient: FILE_BYTES_READ=55
14/02/23 23:28:40 INFO mapred.JobClient: HDFS_BYTES_READ=268
14/02/23 23:28:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=164949
14/02/23 23:28:40 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=33
14/02/23 23:28:40 INFO mapred.JobClient: File Input Format Counters
14/02/23 23:28:40 INFO mapred.JobClient: Bytes Read=38
14/02/23 23:28:40 INFO mapred.JobClient: Map-Reduce Framework
14/02/23 23:28:40 INFO mapred.JobClient: Map output materialized bytes=61
14/02/23 23:28:40 INFO mapred.JobClient: Map input records=2
14/02/23 23:28:40 INFO mapred.JobClient: Reduce shuffle bytes=61
14/02/23 23:28:40 INFO mapred.JobClient: Spilled Records=8
14/02/23 23:28:40 INFO mapred.JobClient: Map output bytes=62
14/02/23 23:28:40 INFO mapred.JobClient: CPU time spent (ms)=12970
14/02/23 23:28:40 INFO mapred.JobClient: Total committed heap usage (bytes)=248127488
14/02/23 23:28:40 INFO mapred.JobClient: Combine input records=6
14/02/23 23:28:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=230
14/02/23 23:28:40 INFO mapred.JobClient: Reduce input records=4
14/02/23 23:28:40 INFO mapred.JobClient: Reduce input groups=4
14/02/23 23:28:40 INFO mapred.JobClient: Combine output records=4
14/02/23 23:28:40 INFO mapred.JobClient: Physical memory (bytes) snapshot=327409664
14/02/23 23:28:40 INFO mapred.JobClient: Reduce output records=4
14/02/23 23:28:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1123270656
14/02/23 23:28:40 INFO mapred.JobClient: Map output records=6
-bash-3.2$ hadoop fs -cat output/part-r-00000
Bye 1
Goodbye 1
Hadoop 2
World 2
-------------
安装插件：把hadoop-eclipse-plugin-1.2.1.jar 放到eclipse\plugins下，重启eclipse。
eclispe调试worldcount修改如下：
conf.set("mapred.job.tracker", "192.168.1.100:9001");
String[] ars = new String[] { "input","output" };
String[] otherArgs = new GenericOptionsParser(conf, ars).getRemainingArgs();
eclipse运行报错：ClassNotFoundException: org.apache.hadoop.examples.WordCount$IntSumReducer
解决：eclipse4.2 换位3.7, hadoop-eclipse-plugin-1.2.1.jar 插件和eclipse4.2不兼容的问题。
-----
查看日志：
more /usr/hadoop/logs/hadoop-hadoop-namenode-Master.Hadoop.log
tail -200 /usr/hadoop/logs/hadoop-hadoop-namenode-Master.Hadoop.log
tail -200 /usr/hadoop/logs/hadoop-hadoop-datanode-Slave1.Hadoop.log
问题1:java.io.IOException: NameNode is not formatted.
解决：hadoop namenode -format
注意：Re-format filesystem in /usr/hadoop/tmp/dfs/name ? (Y or N) Y 注意输入大写Y,小写不行。
问题2：ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /usr/hadoop/tmp/dfs/data: namenode namespaceID = 2053604871; datanode namespaceID = 1352668130
解决：namespaceID 不一致问题。修改了datanode的/usr/hadoop/tmp/dfs/data/current的VERSION文件的namespaceID=2053604871
问题3：java.io.FileNotFoundException: /usr/hadoop/tmp/dfs/name/current/VERSION (Permission denied)
解决：不小心用了root管理员账号启动了start-all.sh，后来取消了，但是还是在/home/hadoop/tmp/生成了一些所有者为root的文件。将文件全部删除 或者改为hadoop用户 # chown -R hadoop:hadoop /usr/hadoop/tmp