hadoop2.6.3 集群部署
准备工作
准备三台ubuntu1404 环境
master 192.168.12.127
slave1 192.168.12.132
slave2 192.168.12.133
本例是通过openstack 创建的三台VM
配置三个节点间ssh 免密码登录
在slave1上
生成密钥，ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
在slave2上
生成密钥，ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
在master 上
1. 生成密钥，ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
2. 将公钥追加到授权的key中，cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
3,scp [email protected]:/home/supermap/.ssh/id_dsa.pub ~/.ssh/id_dsa_132.pub
4,scp [email protected]:/home/supermap/.ssh/id_dsa.pub ~/.ssh/id_dsa_133.pub
5,cat ~/.ssh/id_dsa_132.pub >> ~/.ssh/authorized_keys
6,cat ~/.ssh/id_dsa_133.pub >> ~/.ssh/authorized_keys
7,scp ~/.ssh/authorized_keys [email protected]:/home/supermap/.ssh/authorized_keys
8,scp ~/.ssh/authorized_keys [email protected]:/home/supermap/.ssh/authorized_keys
测试ssh
supermap@master:~$ ssh slave1
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Fri Jan 8 15:58:58 CST 2016
System load: 0.01 Processes: 102
Usage of /: 7.2% of 27.20GB Users logged in: 1
Memory usage: 2% IP address for eth0: 192.168.12.132
Swap usage: 0%
Graph this data and manage this system at:
https://landscape.canonical.com/
Last login: Fri Jan 8 15:50:22 2016 from 192.168.13.8
supermap@slave1:~$ 安装jdk1.7
apt-get install openjdk-7-jdk
查看jdk默认 安装目录
root@slave1:~# ll /usr/lib/jvm/java-7-openjdk-amd64
total 28
drwxr-xr-x 7 root root 4096 Jan 8 11:02 ./
drwxr-xr-x 3 root root 4096 Jan 8 10:58 ../
lrwxrwxrwx 1 root root 22 Nov 19 18:39 ASSEMBLY_EXCEPTION -> jre/ASSEMBLY_EXCEPTION
lrwxrwxrwx 1 root root 22 Nov 19 18:39 THIRD_PARTY_README -> jre/THIRD_PARTY_README
drwxr-xr-x 2 root root 4096 Jan 8 11:02 bin/
lrwxrwxrwx 1 root root 41 Nov 19 18:39 docs -> ../../../share/doc/openjdk-7-jre-headless/
drwxr-xr-x 3 root root 4096 Jan 8 11:02 include/
drwxr-xr-x 5 root root 4096 Jan 8 10:58 jre/
drwxr-xr-x 3 root root 4096 Jan 8 11:02 lib/
drwxr-xr-x 4 root root 4096 Jan 8 10:58 man/
lrwxrwxrwx 1 root root 20 Nov 19 18:39 src.zip -> ../openjdk-7/src.zip
root@slave1:~#安装hadoop
下载hadoop2.6.3,下载链接 http://hadoop.apache.org/releases.html
解压 tar -xvf hadoop-2.6.3.tar.gz ，并在主目录下创建tmp、dfs、dfs/name、dfs/node
supermap@slave1:~$ pwd
/home/supermap
supermap@slave1:~$ ll hadoop-2.6.3
total 72
drwxr-xr-x 12 supermap supermap 4096 Jan 8 11:33 ./
drwxr-xr-x 5 supermap supermap 4096 Jan 8 15:53 ../
-rw-r--r-- 1 supermap supermap 15429 Dec 18 09:52 LICENSE.txt
-rw-r--r-- 1 supermap supermap 101 Dec 18 09:52 NOTICE.txt
-rw-r--r-- 1 supermap supermap 1366 Dec 18 09:52 README.txt
drwxr-xr-x 2 supermap supermap 4096 Dec 18 09:52 bin/
drwxrwxr-x 4 supermap supermap 4096 Jan 8 11:34 dfs/
drwxrwxr-x 2 supermap supermap 4096 Jan 8 11:33 dsf/
drwxr-xr-x 3 supermap supermap 4096 Dec 18 09:52 etc/
drwxr-xr-x 2 supermap supermap 4096 Dec 18 09:52 include/
drwxr-xr-x 3 supermap supermap 4096 Dec 18 09:52 lib/
drwxr-xr-x 2 supermap supermap 4096 Dec 18 09:52 libexec/
drwxr-xr-x 2 supermap supermap 4096 Dec 18 09:52 sbin/
drwxr-xr-x 4 supermap supermap 4096 Dec 18 09:52 share/
drwxrwxr-x 2 supermap supermap 4096 Jan 8 11:33 tmp/配置hadoop
配置hadoop 守护进程的运行环境
编辑etc/hadoop/hadoop-env.sh
修改其中JAVA_HOME=实际安装目录export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
配置hadoop 守护进程的运行参数
配置 core-site.xml文件-->>增加hadoop核心配置（hdfs文件端口是9000、file:/home/supermp/hadoop-2.6.3/tmp、）
fs.defaultFS
hdfs://master:9000
io.file.buffer.size
131072
hadoop.tmp.dir
file:/home/supermap/hadoop-2.6.3/tmp
Abasefor other temporary directories.
hadoop.proxyuser.spark.hosts
*
hadoop.proxyuser.spark.groups
*
配置 hdfs-site.xml 文件-->>增加hdfs配置信息（namenode、datanode端口和目录位置）
dfs.namenode.secondary.http-address
master:9001
dfs.namenode.name.dir
file:/home/supermap/hadoop-2.6.3/dfs/name
dfs.datanode.data.dir
file:/home/supermap/hadoop-2.6.3/dfs/data
dfs.replication
3
dfs.webhdfs.enabled
true
配置 mapred-site.xml 文件-->>增加mapreduce配置（使用yarn框架、jobhistory使用地址以及web地址）
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
master:10020
mapreduce.jobhistory.webapp.address
master:19888
配置 yarn-site.xml 文件-->>增加yarn功能
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.resourcemanager.address
master:8032
yarn.resourcemanager.scheduler.address
master:8030
yarn.resourcemanager.resource-tracker.address
master:8035
yarn.resourcemanager.admin.address
master:8033
yarn.resourcemanager.webapp.address
master:8088
同步配置到两个salve
supermap@master:~/hadoop-2.6.3/etc$ scp -r hadoop supermap@slave1:/home/supermap/hadoop-2.6.3/etc
添加 slave1,slave2 到集群
编辑slaves 文件
supermap@master:~/hadoop-2.6.3/etc/hadoop$ cat slaves
master
slave1
slave2启动hadoop 集群
格式化一个新的分布式文件系统：
$ bin/hadoop namenode -format
启动HDFS
$ sbin/start-dfs.sh
sbin/start-dfs.sh脚本会参照 slaves文件的内容，在所有列出的slave上启动DataNode守护进程
supermap@master:~/hadoop-2.6.3$ ./sbin/start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/supermap/hadoop-2.6.3/logs/hadoop-supermap-namenode-master.out
slave1: starting datanode, logging to /home/supermap/hadoop-2.6.3/logs/hadoop-supermap-datanode-slave1.out
slave2: starting datanode, logging to /home/supermap/hadoop-2.6.3/logs/hadoop-supermap-datanode-slave2.out
master: starting datanode, logging to /home/supermap/hadoop-2.6.3/logs/hadoop-supermap-datanode-master.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /home/supermap/hadoop-2.6.3/logs/hadoop-supermap-secondarynamenode-master.out启动yarn
supermap@master:~/hadoop-2.6.3$ ./sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/supermap/hadoop-2.6.3/logs/yarn-supermap-resourcemanager-master.out
master: starting nodemanager, logging to /home/supermap/hadoop-2.6.3/logs/yarn-supermap-nodemanager-master.out
slave2: starting nodemanager, logging to /home/supermap/hadoop-2.6.3/logs/yarn-supermap-nodemanager-slave2.out
slave1: starting nodemanager, logging to /home/supermap/hadoop-2.6.3/logs/yarn-supermap-nodemanager-slave1.out检查集群状态
supermap@master:~/hadoop-2.6.3$ ./bin/hdfs dfsadmin -report
Configured Capacity: 58405412864 (54.39 GB)
Present Capacity: 51191595008 (47.68 GB)
DFS Remaining: 51191545856 (47.68 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 192.168.12.132:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 29202706432 (27.20 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 3606908928 (3.36 GB)
DFS Remaining: 25595772928 (23.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.65%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jan 08 16:51:01 CST 2016
Name: 192.168.12.133:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 29202706432 (27.20 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 3606908928 (3.36 GB)
DFS Remaining: 25595772928 (23.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.65%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jan 08 16:51:01 CST 2016查看集群
通过http://master:8088/cluster/nodes 查看集群
测试集群
使用hadoop自带的实例进行测试
首先在hdfs中创建一个文件夹
bin/hdfs dfs -mkdir /input
导入一个文件到hdfs中
bin/hdfs dfs -copyFromLocal LICENSE.txt /input
进入share/hadoop/mapreduce目录下执行
../../../bin/hadoop jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input /output
supermap@master:~/hadoop-2.6.3/share/hadoop/mapreduce$ ../../../bin/hadoop jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input /output
16/01/08 16:56:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.12.127:8032
16/01/08 16:56:53 INFO input.FileInputFormat: Total input paths to process : 0
16/01/08 16:56:53 INFO mapreduce.JobSubmitter: number of splits:0
16/01/08 16:56:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1452243042219_0002
16/01/08 16:56:55 INFO impl.YarnClientImpl: Submitted application application_1452243042219_0002
16/01/08 16:56:55 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1452243042219_0002/
16/01/08 16:56:55 INFO mapreduce.Job: Running job: job_1452243042219_0002
16/01/08 16:57:03 INFO mapreduce.Job: Job job_1452243042219_0002 running in uber mode : false
16/01/08 16:57:03 INFO mapreduce.Job: map 0% reduce 0%
16/01/08 16:57:10 INFO mapreduce.Job: map 0% reduce 100%
16/01/08 16:57:11 INFO mapreduce.Job: Job job_1452243042219_0002 completed successfully
16/01/08 16:57:11 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=106221
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=0
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched reduce tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=3848
Total time spent by all reduce tasks (ms)=3848
Total vcore-milliseconds taken by all reduce tasks=3848
Total megabyte-milliseconds taken by all reduce tasks=3940352
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=0
Reduce shuffle bytes=0
Reduce input records=0
Reduce output records=0
Spilled Records=0
Shuffled Maps =0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=24
CPU time spent (ms)=400
Physical memory (bytes) snapshot=167751680
Virtual memory (bytes) snapshot=843304960
Total committed heap usage (bytes)=110624768
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=0查看结果
bin/hadoop fs -cat /output/*
执行结束后可以通过 http://master:8088 查看任务
