ubuntu 14.04.1 hadoop 2.6
sudo addgroup hadoop
sudo adduser --ingroup hadoop hadoop
sudo vim /etc/sodoers
hadoop ALL=(ALL)ALL
sudo apt-get install openjdk-7-jre oenjdk-7-jdk
wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
sudo tar xvf hadoop-2.6.0.tar.gz -C /usr/local/
sudo chown -R hadoop:hadoop hadoop-2.6.0
sudo vim etc/hadoop/hadoop-evn.sh
export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
使用hadoop帐号登入
ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
vim /etc/hosts
127.0.0.1 localhost
192.168.0.140 h1
192.168.0.141 h2
192.168.0.142 h3
192.168.0.143 h4
设置： slaves、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml
1, 文件 slave
cd /usr/local/hadoop-2.6.0/etc/hadoop vim slaves
将原来 localhost 删除，把所有Slave的主机名写上，每行一个。
2, 文件 core-site.xml ，将原本的如下内容：
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://h1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop-2.6.0/tmp</value> <description>Abase for other temporary directories.</description> </property> </configuration>
3, 文件hdfs-site.xml，因为只有一个Slave，所以dfs.replication的值设为1。
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>h1:50090</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop-2.6.0/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop-2.6.0/tmp/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>
4, 文件mapred-site.xml，这个文件不存在，首先需要从模板中复制一份：
cp mapred-site.xml.template mapred-site.xml
然后配置修改如下：
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
5, 文件yarn-site.xml：
<property> <name>yarn.resourcemanager.hostname</name> <value>h1</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
KVM虚拟机克隆
sudo virt-clone -o h1 -n h2 -f /kvm/os/h2.img
sudo virt-clone -o h1 -n h3 -f /kvm/os/h2.img
sudo virt-clone -o h1 -n h4 -f /kvm/os/h2.img
然后在h上就可以启动hadoop了。
cd /usr/local/hadoop/ bin/hdfs namenode -format # 首次运行需要执行初始化，后面不再需要 sbin/start-dfs.sh sbin/start-yarn.sh
jps查看h1启动了NameNode、SecondrryNameNode、ResourceManager进程
h2 h3 h4 启动了DataNode和NodeManager进程
关闭Hadoop集群也是在h1节点上执行：
sbin/stop-dfs.sh sbin/stop-yarn.sh
可以在h1节点上通过命令bin/hdfs dfsadmin -report查看DataNode是否正常启动
hadoop@h1:/usr/local/hadoop-2.6.0$ bin/hdfs dfsadmin -report
Configured Capacity: 154960711680 (144.32 GB)
Present Capacity: 140631048192 (130.97 GB)
DFS Remaining: 140630974464 (130.97 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 192.168.0.141:50010 (h2)
Hostname: h2
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776554496 (4.45 GB)
DFS Remaining: 46876991488 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015
Name: 192.168.0.142:50010 (h3)
Hostname: h3
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776558592 (4.45 GB)
DFS Remaining: 46876987392 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015
Name: 192.168.0.143:50010 (h4)
Hostname: h4
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776550400 (4.45 GB)
DFS Remaining: 46876995584 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015