一、Hadoop集群配置如下
主机名 | IP | JDK版本 | 描述 |
m1 | 192.168.142.144 | 1.8.0_161 | Hadoop集群2.7 NameNode |
s1 | 192.168.142.145 | 1.8.0_161 | Hadoop集群2.7 DataNode1 |
s2 | 192.168.142.146 | 1.8.0_161 | Hadoop集群2.7 DataNode2 |
别忘了在/etc/目录下,修改host文件,添加主机名与IP地址之间的映射
二、ZooKeeper - 3.4.10 下载地址
百度网盘下载地址:zookeeper-3.4.10.tar.gz
官网下载地址:https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/
三、ZooKeeper 安装
下载后的tar.gz上传至m1机器上的/usr/local目录下,并解压
tar -zxvf zookeeper-3.4.10.tar.gz
解压后,进入zookeeper目录如下
四、ZooKeeper 配置
vim /etc/profile (添加如下内容)
export JAVA_HOME=/usr/local/jdk1.8.0_161
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HBASE_HOME=/usr/local/hbase-1.3.1
export HADOOP_HOME=/usr/local/hadoop-2.7.6
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.10
export PATH=$PATH:$HADOOP_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin
添加后,使用source命令,使刚才修改的配置立即生效
source /etc/profile
(2)配置conf
cp zoo_sample.cfg zoo.cfg
打开zoo.cfg,需要配置的内容如下:
vim zoo.cfg
dataDir=/usr/local/zookeeper-3.4.10/data
dataLogDir=/usr/local/zookeeper-3.4.10/data/logs
server.1=m1:2888:3888
server.2=s1:2888:3888
server.3=s2:2888:3888
1、指定Zookeeper的数据存放路径(/usr/local/zookeeper-3.4.10/data)
2、指定Zookeeper的数据日志存放路径(/usr/local/zookeeper-3.4.10/data/logs)
3、客户端连接zookeeper服务端的默认端口为2181
4、Zookeeper集群最好是奇数个节点,根据选举机制,半数以上的票数才能当选为leader(假如有两个节点,你选我,我选你,根据半数以上票数规则,是无法确定集群中到底是你来当leader还是我来当),因此,我们基于Hadoop的集群,分别以NaneNode和两个DataNode作为Zookeeper集群的三个节点
5、server.1 、server.2、server.3表示Zookeeper集群中各个节点所在的主机及服务端口号
6、端口2888:Zookeeper服务之间的通信端口,比如leader和follower之间的通信
7、端口3888:Zookeeper选举leader用的端口
(3)创建data和日志目录
切换到zookeeper目录下,创建data目录和日志目录(-p 递归创建目录,检查目录是否存在,不存在则创建)
mkdir -p data
mkdir -p data/logs
(4)创建myid文件
由于Zookeeper集群中的每一个Zk服务都会有一个唯一标识自己的ID(zk进程ID),这个ID应用在myid和zoo.cfg两个文件中,其中myid文件中存放的是当前Zk服务的标识ID,而zoo.cfg中以server.ID来指定当前的Zk服务进程对应的是哪一台主机,当Zk服务启动的时候,会首先读取myid中的id标识,拿着这个id再去zoo.cfg文件中进行匹配查找,查找的目的是为了弄清楚当前Zk服务在整个ZooKeeper集群中的位置
因此,如果没有指定Zk服务的myid,Zk是启动不了的
分别在m1、s1、s2三台主机上的ZK安装目录下的data目录下创建myid文件,并依次写入值:1,2,3如下:
(5)分别在三台机器上启动zk服务
zkServer.sh start
启动集群后,zkServer之间会互相进行投票选举,选出来一个leader,查看zkServer状态如下:
zkServer.sh status
(6)客户端连接Zk集群
zkCli.sh -server m1:2181,s1:2181,s2:2181
Connecting to m1:2181,s1:2181,s2:2181
2018-07-12 21:04:11,689 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-07-12 21:04:11,693 [myid:] - INFO [main:Environment@100] - Client environment:host.name=m1
2018-07-12 21:04:11,693 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_161
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/local/jdk1.8.0_161/jre
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper-3.4.10/bin/../build/classes:/usr/local/zookeeper-3.4.10/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/usr/local/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.10/bin/../conf:.:/usr/local/jdk1.8.0_161/lib/dt.jar:/usr/local/jdk1.8.0_161/lib/tools.jar
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2018-07-12 21:04:11,696 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2018-07-12 21:04:11,697 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64
2018-07-12 21:04:11,697 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2018-07-12 21:04:11,697 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2018-07-12 21:04:11,697 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root
2018-07-12 21:04:11,698 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=m1:2181,s1:2181,s2:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@446cdf90
Welcome to ZooKeeper!
2018-07-12 21:04:11,724 [myid:] - INFO [main-SendThread(s2:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server s2/192.168.142.146:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-07-12 21:04:11,839 [myid:] - INFO [main-SendThread(s2:2181):ClientCnxn$SendThread@876] - Socket connection established to s2/192.168.142.146:2181, initiating session
[zk: m1:2181,s1:2181,s2:2181(CONNECTING) 0] 2018-07-12 21:04:11,907 [myid:] - INFO [main-SendThread(s2:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server s2/192.168.142.146:2181, sessionid = 0x36491c9d1e60000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null