手动部署OceanBase集群（三节点）
一、实验目标
- 手动部署一个 OB 单副本集群(实际创建三副本)，包括一个 OBProxy 节点。
- 创建一个业务租户、一个业务数据库，以及一些表等。
二、环境说明（准备）
本次部署使用环境如下：
- 宿主机win10 专业版
- 虚拟机软件VMware Workstation 16 Pro
- 虚拟主机3台 8c12g，1台4c8g
- centos7.6 iso 系统镜像
三、部署规划
| ip | 部署内容 | 备注 |
| 192.168.35.101 | obproxy,obclient | ob代理及客户端 |
| 192.168.35.102 | observer | ob集群zone1节点 |
| 192.168.35.103 | observer | ob集群zone2节点 |
| 192.168.35.104 | observer | ob集群zone3节点 |
四、部署配置过程
1、创建虚拟机，具体操作过程略，说明如下：
- 创建一台虚拟机；使用nat网络；100g虚拟磁盘。
- 操作系统最小安装，选择如下包 system administrator tools、compatibility libraries、development tools
- 安装过程中界面设置好网络IP
2、系统配置
- 安装必要的工具包
yum install -y wget net-tools vim tree
- 操作系统参数设置，参照如下进行设置
OceanBase 社区版
https://open.oceanbase.com/docs/tutorials/quickstart/V1.0.0/2-4-how-to-initialize-the-server-environment
3、下载OB软件上传虚拟机并使用admin用户进行安装。
rpm包如下： oceanbase-ce-3.1.1-4.el7.x86_64.rpm oceanbase-ce-libs-3.1.1-4.el7.x86_64.rpm
- 查看安装目录情况
[admin@localhost ~]$ tree oceanbase/
oceanbase/
├── bin
│ ├── import_time_zone_info.py
│ └── observer
├── etc
│ └── timezone_V1.log
└── lib
├── libaio.so -> libaio.so.1.0.1
├── libaio.so.1 -> libaio.so.1.0.1
├── libaio.so.1.0.1
├── libmariadb.so -> libmariadb.so.3
└── libmariadb.so.3
3 directories, 8 files
- 初始化数据目录
su - admin
mkdir -p ~/oceanbase/store/obdemo /data/obdemo/{sstable,etc3} /redo/obdemo/{clog,ilog,slog,etc2}
for f in {clog,ilog,slog,etc2}; do ln -s /redo/obdemo/$f ~/oceanbase/store/obdemo/$f ; done
for f in {sstable,etc3}; do ln -s /data/obdemo/$f ~/oceanbase/store/obdemo/$f; done- 查看目录
[admin@localhost ~]$ tree ~/oceanbase/store/ /data/ /redo/
/home/admin/oceanbase/store/
└── obdemo
├── clog -> /redo/obdemo/clog
├── etc2 -> /redo/obdemo/etc2
├── etc3 -> /data/obdemo/etc3
├── ilog -> /redo/obdemo/ilog
├── slog -> /redo/obdemo/slog
└── sstable -> /data/obdemo/sstable
/data/
└── obdemo
├── etc3
└── sstable
/redo/
└── obdemo
├── clog
├── etc2
├── ilog
└── slog设置环境变量
su - admin
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/oceanbase/lib' >> ~/.bash_profile
. ~/.bash_profile
4、克隆三台虚拟机
- 使用vmware的克隆功能克隆三台虚拟机
- 分别开启克隆的三台虚拟机，修改主机名按规划设置IP地址，（修改前不能同时开启三台机器）
5、配置及启动OceanBase
- 启动 OBSERVER 进程
192.168.35.102 主机
su - admin
cd ~/oceanbase && bin/observer \
-i ens33 -p 2881 -P 2882 -z zone1 \
-d ~/oceanbase/store/obdemo \
-r '192.168.35.102:2882:2881;192.168.35.103:2882:2881;192.168.35.104:2882:2881' \
-c 20211218 -n obdemo \
-o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=268435456, \
system_memory=3G,memory_chunk_cache_size=128M,cpu_count=16,net_thread_count=4, \
datafile_size=50G,stack_size=1536K, \
config_additional_dir=/data/obdemo/etc3;/redo/obdemo/etc2" 192.168.35.103 主机
su - admin
cd ~/oceanbase && bin/observer \
-i ens33 -p 2881 -P 2882 -z zone2 \
-d ~/oceanbase/store/obdemo \
-r '192.168.35.102:2882:2881;192.168.35.103:2882:2881;192.168.35.104:2882:2881' \
-c 20211218 -n obdemo \
-o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=268435456, \
system_memory=3G,memory_chunk_cache_size=128M,cpu_count=16,net_thread_count=4, \
datafile_size=50G,stack_size=1536K, \
config_additional_dir=/data/obdemo/etc3;/redo/obdemo/etc2"192.168.35.104 主机
su - admin
cd ~/oceanbase && bin/observer \
-i ens33 -p 2881 -P 2882 -z zone3 \
-d ~/oceanbase/store/obdemo \
-r '192.168.35.102:2882:2881;192.168.35.103:2882:2881;192.168.35.104:2882:2881' \
-c 20211218 -n obdemo \
-o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=268435456, \
system_memory=3G,memory_chunk_cache_size=128M,cpu_count=16,net_thread_count=4, \
datafile_size=50G, stack_size=1536K, \
config_additional_dir=/data/obdemo/etc3;/redo/obdemo/etc2"
- 检查端口，三台机器都查看
netstat -ntlp
- 集群自举（初始化）（任意一台机器执行，执行前先安装下两个客户端，rpm安装即可，）
[admin@mydog2 soft]$ obclient -h 192.168.35.104 -u root -P 2881 -p -c -A
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 3221225472
Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> set session ob_query_timeout=1000000000;
Query OK, 0 rows affected (0.002 sec)
MySQL [(none)]> alter system bootstrap ZONE 'zone1' SERVER '192.168.35.102:2882', ZONE 'zone2' SERVER '192.168.35.103:2882',ZONE 'zone3' SERVER '192.168.35.104:2882';
Query OK, 0 rows affected (41.964 sec)
MySQL [(none)]>
- 修改root用户sys租户的密码
初始化完后，重新登录数据库，执行 alter user root identified by 'xxxxxx' 修改密码。
6、安装、配置obproxy及客户端
192.168.35.101主机，调低该虚拟机的配置为4c8g
- 社区版官方下载rpm包, 使用admin用户安装obproxy,
[admin@mydog1 soft]$ rpm -ivh obproxy-3.2.0-1.el7.x86_64.rpm
[admin@mydog1 ~]$ tree obproxy-3.2.0/
obproxy-3.2.0/
└── bin
├── obproxy
└── obproxyd.sh- 安装ob客户端（官方下载客户端相关的两个包，rpm安装即可）
- 启动obproxy，并查看端口
cd ~/obproxy-3.1.0/ && bin/obproxy \
-r "192.168.35.102:2881;192.168.35.102:2881;192.168.35.102:2881" \
-p 2883 -o "enable_strict_kernel_release=false,enable_cluster_checkout=false, \
enable_metadb_used=false" -c obdemo[admin@mydog1 obproxy-3.2.0]$ netstat -ntlp |grep obproxy
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 9775/bin/obproxy
tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 9775/bin/obproxy - ob集群创建proxyro用户，并授权（obproxy连接集群使用）
[admin@mydog1 ~]$ obclient -h 192.168.35.104 -u root@sys -P 2881 -p -c -A
create user proxyro;
grant select on oceanbase.* to proxyro identified by 'SWoLCQRH' ;- 设置obproxy用密码及访问ob集群的密码
obclient -h 192.168.35.101 -u root@proxysys -P 2883 -p
show proxyconfig like '%sys_password%';
alter proxyconfig set obproxy_sys_password = 'wPhGddup' ;
alter proxyconfig set observer_sys_password = 'SWoLCQRH' ;
//通过OBproxy 链接数据库（输入ob中root的密码）
- 测试通过obproxy访问ob集群
[admin@mydog1 obproxy-3.2.0]$ obclient -h 192.168.35.101 -uroot@sys#obdemo -P2883 -p -c -A oceanbase
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.6.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [oceanbase]> show databases;
+--------------------+
| Database |
+--------------------+
| oceanbase |
| information_schema |
| mysql |
| SYS |
| LBACSYS |
| ORAAUDITOR |
| test |
+--------------------+
7 rows in set (0.008 sec)
7、集群测试
-
创建资源单元规格
CREATE resource unit u1 max_cpu=2, min_cpu=2, max_memory='2G', min_memory='1G', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='1024G';
CREATE resource unit u2 max_cpu=3, min_cpu=3, max_memory='3G', min_memory='1G', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='1024G';
MySQL [oceanbase]> select * from __all_unit_config;
+----------------------------+----------------------------+----------------+-----------------+---------+---------+------------+------------+----------+----------+---------------+---------------------+
| gmt_create | gmt_modified | unit_config_id | name | max_cpu | min_cpu | max_memory | min_memory | max_iops | min_iops | max_disk_size | max_session_num |
+----------------------------+----------------------------+----------------+-----------------+---------+---------+------------+------------+----------+----------+---------------+---------------------+
| 2021-12-19 01:33:51.963582 | 2021-12-19 01:33:51.963582 | 1 | sys_unit_config | 5 | 2.5 | 1610612736 | 1342177280 | 10000 | 5000 | 53687091200 | 9223372036854775807 |
| 2021-12-19 10:43:47.767246 | 2021-12-19 10:43:47.767246 | 1001 | u1 | 2 | 2 | 2147483648 | 1073741824 | 10000 | 1000 | 1099511627776 | 1000000 |
| 2021-12-19 10:44:09.250366 | 2021-12-19 10:44:09.250366 | 1002 | u2 | 3 | 3 | 3221225472 | 1073741824 | 10000 | 1000 | 1099511627776 | 1000000 |
+----------------------------+----------------------------+----------------+-----------------+---------+---------+------------+------------+----------+----------+---------------+---------------------+
3 rows in set (0.002 sec)
MySQL [oceanbase]>
- 创建资源池
create resource pool pool_1 unit='u1' , unit_num=1, zone_list=('zone1' ,'zone2') ;
create resource pool pool_2 unit='u2' , unit_num=1, zone_list=('zone3');
MySQL [oceanbase]> select t1.name resource_pool_name, t2.`name` unit_config_name, t2.max_cpu, t2.min_cpu, t2.max_memory/1024/1024/1024 max_mem_gb, t2.min_memory/1024/1024/1024 min_mem_gb, t3.unit_id, t3.zone, concat(t3.svr_ip,':',t3.`svr_port`) observer,t4.tenant_id, t4.tenant_name
-> from __all_resource_pool t1 join __all_unit_config t2 on (t1.unit_config_id=t2.unit_config_id)
-> join __all_unit t3 on (t1.`resource_pool_id` = t3.`resource_pool_id`)
-> left join __all_tenant t4 on (t1.tenant_id=t4.tenant_id)
-> order by t1.`resource_pool_id`, t2.`unit_config_id`, t3.unit_id
-> ;
+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+---------------------+-----------+-------------+
| resource_pool_name | unit_config_name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | unit_id | zone | observer | tenant_id | tenant_name |
+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+---------------------+-----------+-------------+
| sys_pool | sys_unit_config | 5 | 2.5 | 1.500000000000 | 1.250000000000 | 1 | zone1 | 192.168.35.102:2882 | 1 | sys |
| sys_pool | sys_unit_config | 5 | 2.5 | 1.500000000000 | 1.250000000000 | 2 | zone2 | 192.168.35.103:2882 | 1 | sys |
| sys_pool | sys_unit_config | 5 | 2.5 | 1.500000000000 | 1.250000000000 | 3 | zone3 | 192.168.35.104:2882 | 1 | sys |
| pool_1 | u1 | 2 | 2 | 2.000000000000 | 1.000000000000 | 1001 | zone1 | 192.168.35.102:2882 | NULL | NULL |
| pool_1 | u1 | 2 | 2 | 2.000000000000 | 1.000000000000 | 1002 | zone2 | 192.168.35.103:2882 | NULL | NULL |
| pool_2 | u2 | 3 | 3 | 3.000000000000 | 1.000000000000 | 1003 | zone3 | 192.168.35.104:2882 | NULL | NULL |
+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+---------------------+-----------+-------------+
6 rows in set (0.002 sec)
MySQL [oceanbase]>
MySQL [oceanbase]> select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, cpu_assigned, (cpu_total-cpu_assigned) cpu_free, mem_total/1024/1024/1024 mem_total_gb, mem_assigned/1024/1024/1024 mem_assign_gb, (mem_total-mem_assigned)/1024/1024/1024 mem_free_gb
-> from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)
-> order by a.zone, a.svr_ip
-> ;
+-------+---------------------+-----------+--------------+----------+----------------+----------------+----------------+
| zone | observer | cpu_total | cpu_assigned | cpu_free | mem_total_gb | mem_assign_gb | mem_free_gb |
+-------+---------------------+-----------+--------------+----------+----------------+----------------+----------------+
| zone1 | 192.168.35.102:2882 | 14 | 4.5 | 9.5 | 5.000000000000 | 2.250000000000 | 2.750000000000 |
| zone2 | 192.168.35.103:2882 | 14 | 4.5 | 9.5 | 5.000000000000 | 2.250000000000 | 2.750000000000 |
| zone3 | 192.168.35.104:2882 | 14 | 5.5 | 8.5 | 5.000000000000 | 2.250000000000 | 2.750000000000 |
+-------+---------------------+-----------+--------------+----------+----------------+----------------+----------------+
3 rows in set (0.005 sec)
MySQL [oceanbase]>
- 创建租户
MySQL [oceanbase]> create tenant obmysql resource_pool_list=('pool_1','pool_2'), primary_zone='RANDOM',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%' ;
Query OK, 0 rows affected (2.919 sec)
MySQL [oceanbase]>
MySQL [oceanbase]> select * from gv$tenant;
+-----------+-------------+-------------------+-------------------+----------------+-----------------------+-----------+---------------------------------------------+
| tenant_id | tenant_name | zone_list | primary_zone | collation_type | info | read_only | locality |
+-----------+-------------+-------------------+-------------------+----------------+-----------------------+-----------+---------------------------------------------+
| 1 | sys | zone1;zone2;zone3 | zone1;zone2,zone3 | 0 | system tenant | 0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
| 1001 | obmysql | zone1;zone2;zone3 | RANDOM | 0 | mysql tenant/instance | 0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
+-----------+-------------+-------------------+-------------------+----------------+-----------------------+-----------+---------------------------------------------+
2 rows in set (0.009 sec)
MySQL [oceanbase]>
- 新租户下创建数据库及用户
obclient -h 192.168.35.101 -uroot@obmysql#obdemo -P2883 -p -c -A oceanbase
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| oceanbase |
| information_schema |
| mysql |
| test |
+--------------------+
4 rows in set (0.003 sec)
MySQL [(none)]>
MySQL [oceanbase]> create database tpccdb;
Query OK, 1 row affected (0.044 sec)
MySQL [oceanbase]> show databases;
+--------------------+
| Database |
+--------------------+
| oceanbase |
| information_schema |
| mysql |
| test |
| tpccdb |
+--------------------+
5 rows in set (0.011 sec)
create user user1 identified by 'user1';
grant all privileges on tpccdb.* to user1 identified by 'user1';- 创建表及插入测试数据
[admin@mydog1 ~]$ obclient -h 192.168.35.101 -uuser1@obmysql#obdemo -P2883 -p -c -A tpccdb
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 1048588
Server version: 5.6.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [tpccdb]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| tpccdb |
+--------------------+
2 rows in set (0.018 sec)
MySQL [tpccdb]>
MySQL [tpccdb]> create table ware(w_id int
-> , w_ytd decimal(12,2)
-> , w_tax decimal(4,4)
-> , w_name varchar(10)
-> , w_street_1 varchar(20)
-> , w_street_2 varchar(20)
-> , w_city varchar(20)
-> , w_state char(2)
-> , w_zip char(9)
-> , unique(w_name, w_city)
-> , primary key(w_id)
-> );
Query OK, 0 rows affected (0.228 sec)
MySQL [tpccdb]> create table cust (c_w_id int NOT NULL
-> , c_d_id int NOT null
-> , c_id int NOT null
-> , c_discount decimal(4, 4)
-> , c_credit char(2)
-> , c_last varchar(16)
-> , c_first varchar(16)
-> , c_middle char(2)
-> , c_balance decimal(12, 2)
-> , c_ytd_payment decimal(12, 2)
-> , c_payment_cnt int
-> , c_credit_lim decimal(12, 2)
-> , c_street_1 varchar(20)
-> , c_street_2 varchar(20)
-> , c_city varchar(20)
-> , c_state char(2)
-> , c_zip char(9)
-> , c_phone char(16)
-> , c_since date
-> , c_delivery_cnt int
-> , c_data varchar(500)
-> , index icust(c_last, c_d_id, c_w_id, c_first, c_id)
-> , FOREIGN KEY (c_w_id) REFERENCES ware(w_id)
-> , primary key (c_w_id, c_d_id, c_id)
-> );
Query OK, 0 rows affected (0.181 sec)
MySQL [tpccdb]> show tables;
+------------------+
| Tables_in_tpccdb |
+------------------+
| cust |
| ware |
+------------------+
2 rows in set (0.003 sec)
MySQL [tpccdb]>
MySQL [tpccdb]> select * from t1;
+------+--------+
| id | name |
+------+--------+
| 1 | mydog1 |
+------+--------+
1 row in set (0.004 sec)
MySQL [tpccdb]> insert into t1 values(1,'mycat1');
Query OK, 1 row affected (0.005 sec)
MySQL [tpccdb]> select * from t1;
+------+--------+
| id | name |
+------+--------+
| 1 | mydog1 |
| 1 | mycat1 |
+------+--------+
2 rows in set (0.002 sec)
MySQL [tpccdb]>
8、配置过程中遇到的坑
1、启动OB出现如下错误
[2021-12-18 17:19:27.093988] ERROR [SERVER] init_config (ob_server.cpp:913) [33906][0][Y0-0000000000000000] [lt=3] local address isn't valid(self_addr_
="0.0.0.0:2882", ret=-4002) BACKTRACE:0x9786d2e 0x96d0bdb 0x22602e3 0x2363faa 0x913cbc3 0x9138a6f 0x2222045 0x2b68ebbc73d5 0x2220d29
//复制启动参数后，网卡没改成自己主机对应的网卡