ZooKeeper是一个开源的分布式的,为分布式应用提供协调服务的Apache项目。
ZooKeeper从设计模式角度来理解:是一个基于观察者模式设计的分布式服务管理框架,它负责存储和管理大家都关心的数据,然后接受观察者的注册,一旦这些数据的状态发生变化,ZooKeeper就将负责通知已经在ZooKeeper上注册的那些观察者做出相应的反应。 例如:一组服务器集群、一组客户端,全部通过ZooKeeper注册,如果一台服务节点宕机,则ZooKeeper集群可以监测宕机几点服务器状态并通知客户端。
它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、域名服务、分布式同步、组服务等。
下载zookeeper的安装包,下载地址:http://zookeeper.apache.org/, 上传到linux集群环境下,规划自己的管理目录,解压安装包。
[root@master apache-zookeeper-3.6.3-bin]# cp apache-zookeeper-3.6.3-bin.tar.gz /opt/
[root@master apache-zookeeper-3.6.3-bin]# cd /opt/
[root@master apache-zookeeper-3.6.3-bin]# tar -xvf apache-zookeeper-3.6.3-bin.tar.gz
进入conf目录,复制zoo-sample.cfg重命名为zoo.cfg,通过修改zoo.cfg来对zookeeper进行配置。这个名字固定写死,因为zookeeper启动会检查这个文件,根据这个配置文件里的信息来启动服务。
[root@master opt]# cd apache-zookeeper-3.6.3-bin/conf/
[root@master conf]# cp zoo_sample.cfg zoo.cfg
## 修改如下
dataDir=/opt/apache-zookeeper-3.6.3-bin/data
##在那最后添加如下
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
配置myid
到之前配置的zookeeper数据文件所在的目录下( /opt/apache-zookeeper-3.6.3-bin/data )生成一个文件叫myid,其中写上一个数字表明当前机器是哪一个编号的机器。
[root@master conf]# cd ..
[root@master apache-zookeeper-3.6.3-bin]# ls
bin conf docs lib LICENSE.txt NOTICE.txt README.md README_packaging.md
[root@master apache-zookeeper-3.6.3-bin]# mkdir data
[root@master apache-zookeeper-3.6.3-bin]# cd data
[root@master data]# echo 1 > myid
[root@master data]# cat myid
1
注意:文件名称必须是myid,文件内容只需要一个数字即服务器列表中当前服务器的编号。
拷贝: 将以上Zookeeper文件夹远程拷贝到另外两台服务器中(注意myid 需要修改)
[root@master opt]# scp -r apache-zookeeper-3.6.3-bin slave1:/opt
[root@master opt]# scp -r apache-zookeeper-3.6.3-bin slave2:/opt
###注意myid 需要修改
启动zookeeper的各种命令操作如下,可以使用绝对路径操作这些命令,也可使用相对路径操作这些命令,相对路径需要进到zookeeper服务的bin目录进行操作。
#启动ZK服务:
bin/zkServer.sh start
#停止ZK服务:
bin/zkServer.sh stop
#重启ZK服务:
bin/zkServer.sh restart
#查看ZK服务状态:
bin/zkServer.sh status
Zookeeper集群需要每台挨个启动。
可以用 `jps命令 `查看线程。
启动集群的时候,集群数量启动没有超过一半,状态会有错误提示,当集群启动数量超过一半就会自动转为正常状态,并且此台使集群进入正常工作状态的服务器会成为leader角色,集群中其他服务器的角色为follower。
[root@slave2 apache-zookeeper-3.6.3-bin]# ./bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave1 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave1 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@master conf]# ../bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@master conf]# ../bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@slave2 logs]# ./../bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
… 查看Hadoop搭建(一)
[root@master conf]# jps
9157 Jps
8392 DataNode
7961 QuorumPeerMain
8508 NodeManager
9100 JournalNode
8286 NameNode
[root@slave1 bin]# jps
6176 JournalNode
6325 Jps
5719 NodeManager
5372 QuorumPeerMain
5644 DataNode
[root@slave2 logs]# jps
6512 JournalNode
5732 DataNode
5988 SecondaryNameNode
6677 Jps
5917 NodeManager
5295 QuorumPeerMain
到这里下载 https://mirrors.bfsu.edu.cn/apache/hbase/ , 选择V1.4.13版本下载, 并上传到服务器;
[root@master ~]# tar -xvf hbase-1.4.13-bin.tar.gz -C /opt/
[root@master ~]# cd /opt/hbase-1.4.13/
[root@master hbase-1.4.13]# ls
bin CHANGES.txt conf docs hbase-webapps LEGAL lib LICENSE.txt NOTICE.txt README.txt
[root@master conf]# cp hbase-env.sh hbase-env.sh.bak
[root@master conf]# vi hbase-env.sh
#修改
export JAVA_HOME=/usr/java/jdk1.8.0_291-amd64
// 告诉hbase使用外部的zookeeper
export HBASE_MANAGES_ZK=false
[root@master conf]# cp hbase-site.xml hbase-site.xml.bak
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master:2181,slave1:2181,slave2:2181</value>
</property>
### 2181为默认的 clientPort
[root@master conf]# vim regionservers
master
slave1
slave2
[root@master opt]# scp -r hbase-1.4.13/ slave1:/opt
[root@master opt]# scp -r hbase-1.4.13/ slave2:/opt
在3台机器上安装ntp
[root@master opt]# yum install ntp
配置master为备选ntp时间同步服务器(当节点无法连接外网进行时间同步), slave1和slave2无法向外网同步时间时候, 向master看齐;
当master也无法同步外网时间时,都使用master的本地时钟;( 如果只有master无法同步外网, 可能时间会不同步 =.=)
[root@master opt]# vi /etc/ntp.conf
...
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 8
...
[root@master opt]# systemctl start ntpd.service //启动ntp服务
[root@master opt]# systemctl enable ntpd.service //开机自启
[root@master opt]# ntpq -p // 查看同步时间
remote refid st t when poll reach delay offset jitter
==============================================================================
+time.cloudflare 10.12.2.186 3 u 6 64 7 173.979 -0.434 0.420
*tick.ntp.infoma .GPS. 1 u 4 64 7 180.762 12.364 9.194
+makaki.miuku.ne 218.186.3.36 2 u 8 64 7 118.586 0.959 3.578
ntp1.ams1.nl.le 130.133.1.10 2 u 5 64 7 178.315 -0.166 1.903
LOCAL(0) .LOCL. 8 l 78 64 6 0.000 0.000 0.000
//本地时钟已经成功和外面的 NTP 服务器同步。ntpq 报告中第一列 * 表示目前选择的主同步服务器,标 + 的表示有可能被用来进一步提高同步精度的次要服务器。
在slave1和slave2配置ntp
[root@slave1 opt]# vi /etc/ntp.conf
### 添加
server master
[root@slave1 opt]# systemctl restart ntpd
[root@slave1 opt]# ntpstat
synchronised to NTP server (185.209.85.222) at stratum 3
time correct to within 156 ms
polling server every 64 s
###同理配置slave2
这里已经启动了hdfs, zookeeper,yarn等服务;
[root@master bin]# pwd
/opt/hbase-1.4.13/bin
[root@master bin]# ls
draining_servers.rb hbase-common.sh hirb.rb region_status.rb stop-hbase.cmd
get-active-master.rb hbase-config.cmd local-master-backup.sh replication stop-hbase.sh
graceful_stop.sh hbase-config.sh local-regionservers.sh rolling-restart.sh test
hbase hbase-daemon.sh master-backup.sh shutdown_regionserver.rb thread-pool.rb
hbase-cleanup.sh hbase-daemons.sh region_mover.rb start-hbase.cmd zookeepers.sh
hbase.cmd hbase-jruby regionservers.sh start-hbase.sh
[root@master bin]# ./start-hbase.sh
running master, logging to /opt/hbase-1.4.13/bin/../logs/hbase-root-master-master.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
slave1: running regionserver, logging to /opt/hbase-1.4.13/bin/../logs/hbase-root-regionserver-slave1.out
slave2: running regionserver, logging to /opt/hbase-1.4.13/bin/../logs/hbase-root-regionserver-slave2.out
master: running regionserver, logging to /opt/hbase-1.4.13/bin/../logs/hbase-root-regionserver-master.out
slave2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
slave2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
[root@master bin]# jps
17328 Jps
8392 DataNode
7961 QuorumPeerMain
9100 JournalNode
17005 HRegionServer
8286 NameNode
16879 HMaster
[root@slave1 logs]# jps
6176 JournalNode
13315 Jps
5372 QuorumPeerMain
5644 DataNode
13167 HRegionServer
[root@slave1 logs]# jps
6176 JournalNode
13315 Jps
5372 QuorumPeerMain
5644 DataNode
13167 HRegionServer
[root@slave2 logs]# jps
6512 JournalNode
13859 Jps
5732 DataNode
5988 SecondaryNameNode
13662 HRegionServer
5295 QuorumPeerMain
1.stdent表创建
指定命名空间school, 表名为student , 以及列族 essential , additional 其中列族 essential 有列 name, sex, 列族 additional有列
interest
hbase(main):003:0> list_namespace
NAMESPACE default hbase
2 row(s) in 0.0800 seconds
--------------------------------------------
###创建namespace
hbase(main):005:0> create_namespace 'school'
0 row(s) in 1.5470 seconds
hbase(main):006:0> list_namespace
NAMESPACE default hbase school 3 row(s) in 0.0290 seconds
--------------------------------------------
###在命名空间school下,创建student表
hbase(main):007:0> create 'school:student',{NAME => 'essential'},{NAME => 'additional'}
0 row(s) in 3.5560 seconds
=> Hbase::Table - school:student
--------------------------------------------
默认命名空间, 表名 teacher以及列族essential
hbase(main):011:0> create 'teacher',{NAME => 'essential'}
0 row(s) in 4.2930 seconds
=> Hbase::Table - teacher
hbase(main):012:0> list
TABLE
school:student
teacher
2 row(s) in 0.0240 seconds
=> ["school:student", "teacher"]
2.teacher表中删除列族essential并新增列族additional
hbase(main):013:0> alter 'teacher', {NAME => 'additional'}, {NAME => 'essential', METHOD => 'delete'}
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 2.3850 seconds
hbase(main):016:0> desc 'teacher'
Table teacher is ENABLED
teacher
COLUMN FAMILIES DESCRIPTION
{NAME => 'additional', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE',
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true',
BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.0700 seconds
hbase(main):017:0> disable 'teacher'
0 row(s) in 2.3320 seconds
hbase(main):018:0> drop'teacher'
0 row(s) in 1.2900 seconds
hbase(main):019:0> list
TABLE
school:student
1 row(s) in 0.0290 seconds
=> ["school:student"]
------------------------------------------------------------------------------"""
put 'school:student', '001','essential:name','Lixibin'
put 'school:student', '001','essential:sex','男'
put 'school:student', '001','additional:interest','fzuer'
"""
hbase(main):022:0> put 'school:student', '001','essential:name','Lixibin'
0 row(s) in 0.1380 seconds
hbase(main):023:0> put 'school:student', '001','essential:sex','男'
0 row(s) in 0.0260 seconds
hbase(main):024:0> put 'school:student', '001','additional:interest','fzuer'
0 row(s) in 0.0710 seconds
------------------------------------------------------------------------------
"""
put 'school:student', '002','essential:name','Lixibin222'
put 'school:student', '002','essential:sex','女'
put 'school:student', '002','additional:interest','fzuer222'
"""
hbase(main):025:0> put 'school:student', '002','essential:name','Lixibin222'
0 row(s) in 0.0460 seconds
hbase(main):026:0> put 'school:student', '002','essential:sex','女'
0 row(s) in 0.0200 seconds
hbase(main):027:0> put 'school:student', '002','additional:interest','fzuer222'
0 row(s) in 0.0350 seconds
------------------------------------------------------------------------------
#同理再添加几组
"""
put 'school:student', '003','essential:name','Lixibin333'
put 'school:student', '002','essential:sex','女'
put 'school:student', '003','essential:sex','未知'
"""
"""
put 'school:student', '004','essential:name','Lixibin333'
put 'school:student', '004','essential:sex','略略'
"""
"""
put 'school:student', '005','essential:name','Lixibin444'
put 'school:student', '005','essential:sex','444'
put 'school:student', '005','essential:sex','555'
put 'school:student', '005','essential:sex','666'
"""
--------------------------------------------------------------------------
"""
# 设置可以设置3个版本的数据
alter 'school:student',{NAME=>'additional',VERSIONS=>3}
"""
hbase(main):060:0> alter 'school:student',{NAME=>'additional',VERSIONS=>3}
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 3.2530 seconds
--------------------------------------------------------------------------
"""
put 'school:student', '006','essential:name','lixibin666'
put 'school:student', '006','essential:sex','sex666'
put 'school:student', '006','additional:interest','interest666'
put 'school:student', '006','additional:interest','interest666_6'
put 'school:student', '006','additional:interest','interest666_6_6'
"""
get 'school:student','006',{COLUMN=>'additional:interest', VERSIONS=>3}
get 'school:student','005',{COLUMN=>'essential:sex', VERSIONS=>3}
"""
hbase(main):068:0> get 'school:student','006',{COLUMN=>'additional:interest', VERSIONS=>3}
COLUMN CELL
additional:interest timestamp=1625479193891, value=interest666_6_6
additional:interest timestamp=1625479192882, value=interest666_6
additional:interest timestamp=1625479192793, value=interest666
1 row(s) in 0.0720 seconds
hbase(main):069:0> scan 'school:student'
ROW COLUMN+CELL
001 column=additional:interest, timestamp=1625478586795, value=fzuer
001 column=essential:name, timestamp=1625478584206, value=Lixibin
001 column=essential:sex, timestamp=1625478584284, value=\xE7\x94\xB7
002 column=additional:interest, timestamp=1625478636679, value=fzuer222
002 column=essential:name, timestamp=1625478635392, value=Lixibin222
002 column=essential:sex, timestamp=1625478697069, value=\xE5\xA5\xB3
003 column=essential:name, timestamp=1625478696173, value=Lixibin333
003 column=essential:sex, timestamp=1625478763282, value=\xE6\x9C\xAA\xE7\x9F\xA5
004 column=essential:name, timestamp=1625478774239, value=Lixibin333
004 column=essential:sex, timestamp=1625478775059, value=\xE7\x95\xA5\xE7\x95\xA5
005 column=essential:name, timestamp=1625478910610, value=Lixibin444
005 column=essential:sex, timestamp=1625478911901, value=666
006 column=additional:interest, timestamp=1625479193891, value=interest666_6_6
006 column=essential:name, timestamp=1625479192689, value=lixibin666
006 column=essential:sex, timestamp=1625479192758, value=sex666
6 row(s) in 0.0970 seconds
使用Maven导入相关依赖:
<dependencies>
<dependency>
<groupId>org.apache.hbasegroupId>
<artifactId>hbase-clientartifactId>
<version>1.4.13version>
dependency>
<dependency>
<groupId>junitgroupId>
<artifactId>junitartifactId>
<version>4.12version>
<scope>testscope>
dependency>
dependencies>
将hbase配置文件 hbase-site.xml
复制到resource目录
下;
在C:\Windows\System32\drivers\etc 的hosts文件中添加
192.168.233.233 master
192.168.233.234 slave1
192.168.233.235 slave2
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;
import javax.sound.midi.Soundbank;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class MyHBaseClient {
//获取配置信息
public static Configuration conf;
static{
conf = HBaseConfiguration.create();
}
//1.判断一张表是否存在
public static boolean isExist(String tableName){
//对表操作需要使用HbaseAdmin
try {
Connection connection = ConnectionFactory.createConnection(conf);
//管理表
HBaseAdmin admin = (HBaseAdmin) connection.getAdmin();
return admin.tableExists(TableName.valueOf(tableName));
} catch (IOException e) {
e.printStackTrace();
}
return false;
}
//2.在hbase创建表
public static void createTable(String tableName,String... columnfamily){
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
//管理表
HBaseAdmin admin = (HBaseAdmin) connection.getAdmin();
//1.表如果存在,请输入其他表名
if(isExist(tableName)){
System.out.println("表存在,请输入其他表名");
}else{
//2.注意:创建表的话,需要创建一个描述器
HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(tableName));
//3.创建列族
for(String cf:columnfamily){
htd.addFamily(new HColumnDescriptor(cf));
}
//4.创建表
admin.createTable(htd);
System.out.println("表已经创建成功!");
}
} catch (IOException e) {
e.printStackTrace();
}
}
//3.删除hbase的表
public static void deleteTable(String tableName) {
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
//管理表
HBaseAdmin admin = (HBaseAdmin) connection.getAdmin();
//1.表如果存在,请输入其他表名
if (!isExist(tableName)) {
System.out.println("表不存在");
} else {
//2.如果表存在,删除
admin.disableTable(TableName.valueOf(tableName));
admin.deleteTable(TableName.valueOf(tableName));
System.out.println("表删除了");
}
} catch (IOException e) {
e.printStackTrace();
}
}
//4.添加数据 put 'user','rowkey','info:name','tony'
public static void addRow(String tableName,String rowkey,String cf,String column,String value){
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
Table t = connection.getTable(TableName.valueOf(tableName));
//1.表如果存在,请输入其他表名
if (!isExist(tableName)) {
System.out.println("表不存在");
} else {
//2.用put方式加入数据
Put p = new Put(Bytes.toBytes(rowkey));
//3.加入数据
p.addColumn(Bytes.toBytes(cf),Bytes.toBytes(column),Bytes.toBytes(value));
t.put(p);
}
} catch (IOException e) {
e.printStackTrace();
}
}
//5.删除表中一行数据
public static void deleteRow(String tableName,String rowkey,String cf ){
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
Table t = connection.getTable(TableName.valueOf(tableName));
//1.表如果存在,请输入其他表名
if (!isExist(tableName)) {
System.out.println("表不存在");
} else {
//1.根据rowkey删除数据
Delete delete = new Delete(Bytes.toBytes(rowkey));
//2.删除
t.delete(delete);
System.out.println("删除成功");
}
} catch (IOException e) {
e.printStackTrace();
}
}
//6.删除多行数据
public static void deleteAll(String tableName,String... rowkeys){
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
Table t = connection.getTable(TableName.valueOf(tableName));
//1.表如果存在,请输入其他表名
if (!isExist(tableName)) {
System.out.println("表不存在");
} else {
//1.把delete封装到集合
List<Delete> list = new ArrayList<Delete>();
//2.遍历
for (String row:rowkeys){
Delete d = new Delete(Bytes.toBytes(row));
list.add(d);
}
t.delete(list);
System.out.println("删除成功");
}
} catch (IOException e) {
e.printStackTrace();
}
}
//7. 扫描表数据 scan全表扫描
public static void scanAll(String tableName){
try {
//对表操作需要使用HbaseAdmin
Connection connection = ConnectionFactory.createConnection(conf);
Table t = connection.getTable(TableName.valueOf(tableName));
//1.实例scan
Scan s = new Scan();
//2.拿到Scanner对象
ResultScanner rs = t.getScanner(s);
//3.遍历
for (Result r:rs){
Cell[] cells = r.rawCells();
//遍历具体数据
for (Cell c : cells){
System.out.print("行键为:"+ Bytes.toString(CellUtil.cloneRow(c))+" ");
System.out.print("列族为:"+Bytes.toString(CellUtil.cloneFamily(c))+" ");
System.out.print("列名为:"+Bytes.toString(CellUtil.cloneQualifier(c))+" ");
System.out.println("值为:"+Bytes.toString(CellUtil.cloneValue(c)));
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
// System.out.println(isExist("school:student"));
// scanAll("school:student");
createTable("school:teacher", "essential", "additional");
}
}
1. 创建表teacher并指定命名空间为school,列族为essential和additional
看到创建成功!
插入5条数据
插入多版本数据和获取