Hadoop分布式安装

目的

3台虚拟机实现1个namenode，2个datanode的hadoop集群。

环境

硬件：MAC * 1；Parallels Desktop虚拟机*3：CentOS-7

使用ssh客户端操作3台机器

JDK版本

[keon@centos-linux bin]$ /usr/jdk1.8.0_101/bin/java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

HADOOP版本

[keon@centos-linux bin]$ ./hdfs version
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /usr/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar

配置过程

1.修改host

[keon@centos-linux bin]$ vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.211.55.6 h1
10.211.55.7 h3
10.211.55.8 h2

2.配置免密码

在h1生成密匙

[keon@centos-linux bin]$ ssh-keygen -t rsa
[keon@centos-linux ~]$ ls .ssh/
id_rsa id_rsa.pub known_hosts

在h2,h3拷贝公匙

scp keon@h1:/home/keon/.ssh/id_rsa.pub /home/keon/.ssh/authorized_keys

在h1验证是否免密码成功

[keon@centos-linux ~]$ ssh h2
Last login: Tue Aug 2 10:34:14 2016 from h1
[keon@centos-linux ~]$ exit
logout
Connection to h2 closed.
[keon@centos-linux ~]$ ssh h3
Last login: Tue Aug 2 10:34:25 2016 from h1

3.修改配置文件

[keon@centos-linux hadoop-2.7.2]$ pwd
/usr/hadoop-2.7.2

(1)etc/hadoop/hadoop-env.sh

[keon@centos-linux hadoop-2.7.2]$ vim etc/hadoop/hadoop-env.sh

修改java环境地址

export JAVA_HOME=/usr/jdk1.8.0_101

(2)etc/hadoop/core-site.xml

fs.default.name
hdfs://h1:9000/


hadoop.tmp.dir
/usr/hadoop-2.7.2/tmp

(3)etc/hadoop/hdfs-site.xml

dfs.namenode.name.dir
file:/usr/hadoop-2.7.2/namenode


dfs.datanode.data.dir
file:/usr/hadoop-2.7.2/datanode


dfs.replication
2

(4)etc/hadoop/mapred-site.xml

mapreduce.framework.name
yarn

修改完后，推送到每一个节点

启动测试

1.格式化namenode

[keon@centos-linux hadoop-2.7.2]$ ./bin/hdfs namenode -format

16/08/02 11:25:45 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = h1/10.211.55.6
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = /usr/hadoop-2.7.2/etc
.
.
.
16/08/02 11:25:47 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/08/02 11:25:47 INFO util.GSet: VM type = 64-bit
16/08/02 11:25:47 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/08/02 11:25:47 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /usr/hadoop-2.7.2/namenode ? (Y or N) y
16/08/02 11:25:49 INFO namenode.FSImage: Allocated new BlockPoolId: BP-566776757-10.211.55.6-1470108349176
16/08/02 11:25:49 INFO common.Storage: Storage directory /usr/hadoop-2.7.2/namenode has been successfully formatted.
16/08/02 11:25:49 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/08/02 11:25:49 INFO util.ExitUtil: Exiting with status 0
16/08/02 11:25:49 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h1/10.211.55.6
************************************************************/

2.启动服务
启动hdfs

./sbin/start-dfs.sh

Starting namenodes on [h1]
h1: starting namenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-namenode-centos-linux.shared.out
h3: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
h2: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-secondarynamenode-centos-linux.shared.out

启动yarn

./sbin/start-yarn.sh

starting yarn daemons
starting resourcemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-resourcemanager-centos-linux.shared.out
h3: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out
h2: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out

3.验证服务
h1

[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
2930 Jps
1442 SecondaryNameNode
1045 NameNode
2443 ResourceManager

h2

[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
7703 DataNode
29240 Jps
8286 NodeManager

h3

[root@centos-linux keon]# /usr/jdk1.8.0_101/bin/jps
11584 NodeManager
12273 Jps
10346 DataNode

相关进程已启动