oceanbase部署--使用OBD自动化部署三节点集群

准备步骤：

先将192.168.152.30 现有单节点集群stop，然后克隆两台虚拟机并在workstations重置网卡MAC，配置新主机IP

[admin@oceanbase ~]$ obd cluster list
+----------------------------------------------------------------------+
|                             Cluster List                             |
+-------------+--------------------------------------+-----------------+
| Name        | Configuration Path                   | Status (Cached) |
+-------------+--------------------------------------+-----------------+
| obce-single | /home/admin/.obd/cluster/obce-single | running         |
+-------------+--------------------------------------+-----------------+
[admin@oceanbase ~]$ obd cluster stop obce-single
Get local repositories ok
Search plugins ok
Open ssh connection ok
Stop observer ok
obce-single stopped
[admin@oceanbase ~]$ free -g
              total        used        free      shared  buff/cache   available
Mem:             15           0          13           0           1          11
Swap:             2           0           2

开始部署：

资源列表

IP     192.168.152.30、192.168.152.31、192.168.152.32
网卡名 ens33
OS     rhel80
CPU    4C
内存   总内存 16G，可用内存 11G以上
磁盘1  /dev/sdb 30G（一块也够）
磁盘2  /dev/sdc 30G

机器和角色划分

OBD       192.168.152.30 中控机，自动化部署软件
OBServer  192.168.152.30 OceanBase 数据库 zone1
OBServer  192.168.152.31 OceanBase 数据库 zone2
OBServer  192.168.152.32 OceanBase 数据库 zone3
OBProxy   192.168.152.30 OceanBase 访问反向代理  --若环境资源允许，可以每台observer均部署obproxy
OBClient  192.168.152.30 OceanBase 命令行客户端

配置互信和时钟同步

三台机子均执行如下命令：

ssh-keygen -t rsa
 ssh-copy-id 192.168.152.30
 ssh-copy-id 192.168.152.31
 ssh-copy-id 192.168.152.32
 ssh 192.168.152.30
 ssh 192.168.152.31
 ssh 192.168.152.32

时钟同步配置
服务端：vi /etc/chrony.conf
server 127.0.0.1
allow 192.168.0.0/16
local stratum 10

客户端:vi /etc/chrony.conf
server 192.168.152.30

systemctl enable chronyd.service
systemctl restart chronyd.service
chronyc activity 
[root@obce01 ~]# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* time.neu.edu.cn               1   7   377    85  -1452us[-1502us] +/-   19ms
^- ntp.wdc2.us.leaseweb.net      2   8   377   144  +1276us[+1228us] +/-  247ms
^+ time.neu.edu.cn               1   7   377    27  +2004us[+2004us] +/-   18ms
^+ 139.199.214.202               2   8   377    93  +1098us[+1049us] +/-   56ms
^? localhost                     0   9     0     -     +0ns[   +0ns] +/-    0ns

sudo clockdiff 192.168.152.30

部署三节点 observer 和 obproxy 进程：

下载yaml模板

https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-with-obproxy-example.yaml

su - admin
vi /home/admin/obcluster-3zones.yaml

## Only need to configure when remote login is required
user:
    username: admin
    key_file: /home/admin/.ssh/id_rsa.pub 
    port: your ssh port, default 22
oceanbase-ce:
  servers:
    - name: obce01
      ip: 192.168.152.30
    - name: obce02
      ip: 192.168.152.31
    - name: obce03
      ip: 192.168.152.32
  global:
    devname: ens33
    cluster_id: 2
    memory_limit: 8G # The maximum running memory for an observer
    system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    datafile_size: 10G # Size of the data file. 
    log_disk_size: 3G # The size of disk space used by the clog files.
    cpu_count: 16
    production_mode: false
    syslog_level: WARN # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 4 
    appname: obcluster-3zones  # observer cluster name, consistent with obproxy's cluster_name
    root_password: 0EI5N08d# root_password: # root user password, can be empty
    proxyro_password: uY7Yf8zx# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
  obce01:
    mysql_port: 3881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 3882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone1
  obce02:
    mysql_port: 3881 
    rpc_port: 3882 
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone2
  obce03:
    mysql_port: 3881 
    rpc_port: 3882 
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone3
obproxy:
  # Set dependent components for the component.
  # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
  depends: #注意上面那行注释的意思，参数将继承depends所配置的组件
    - oceanbase-ce
  servers:
          - 192.168.152.30
          # - 192.168.152.31  #本人环境资源有限，obproxy只装在obd上
          # - 192.168.152.32
  global:
    listen_port: 3883 # External port. The default value is 2883.
    prometheus_listen_port: 3884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/obproxy
    rs_list: 192.168.152.30:3881;192.168.152.31:3881;192.168.152.32:3881
    enable_cluster_checkout: false
    cluster_name: obcluster-3zones
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    obproxy_sys_password: 0MdTv1tm # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    observer_sys_password: uY7yf8zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

1）若yaml文件部署报参数错误，请仔细检查参数。每个配置项后面冒号（:）跟后面的值之间必须有个空格 （’ '）。
2）若报密码obproxy_sys_password和observer_sys_password相关报错，配置文件中可以删除该部分配置，待集群创建完成后自行配置即可

OBD 部署集群：

i.复制软件到相应节点，并在各节点安装软件，本环境方案中，各节点为克隆虚拟机，无需安装。
ii.在相应节点创建相关目录。
iii.obd cluster deploy [集群名] -c 集群配置文件

obd cluster deploy obcluster-3zones -c /home/admin/obcluster-3zones.yaml

[ERROR] OBD-1002: Fail to init obce01(192.168.152.30) home path: /home/admin/oceanbase-ce is not empty.
初次安装不会报我这个错

IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for IP in $IPS;do echo $IP; ssh $IP "/bin/rm -rf ~/oceanbase-ce/ /data/ /redo/ ~/obproxy/ "; done

解决办法：
配置文件中注释掉图中obproxy_sys_password: 0MdTv1tm 和observer_sys_password: uY7yf8zx，待集群创建完成后自行配置即可（后面*连接obproxy集群 部分有介绍）

初始化并启动集群

obd cluster start obcluster-3zones

如果集群节点内核参数和会话限制参数不符合要求，安装时会给出提示。 这个命令会进行几分钟的 bootstrap。当可用内存不足 8G 或者日志目录剩余可用空间比例不足 5% 的时候，这个bootstrap 是很可能会失败的。

[admin@obce01 ~]$ obd cluster list
+--------------------------------------------------------------------------------+
|                                  Cluster List                                  |
+------------------+-------------------------------------------+-----------------+
| Name             | Configuration Path                        | Status (Cached) |
+------------------+-------------------------------------------+-----------------+
| obce-single      | /home/admin/.obd/cluster/obce-single      | destroyed       |
| obcluster-3zones | /home/admin/.obd/cluster/obcluster-3zones | running         |
+------------------+-------------------------------------------+-----------------+

[admin@obce01 oceanbase-ce]$ obd cluster display obcluster-3zones
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
Connect to observer ok
Wait for observer init ok
+--------------------------------------------------+
|                     observer                     |
+----------------+---------+------+-------+--------+
| ip             | version | port | zone  | status |
+----------------+---------+------+-------+--------+
| 192.168.152.30 | 3.1.0   | 3881 | zone1 | active |
| 192.168.152.31 | 3.1.0   | 3881 | zone2 | active |
| 192.168.152.32 | 3.1.0   | 3881 | zone3 | active |
+----------------+---------+------+-------+--------+
obclient -h192.168.152.30 -P3881 -uroot -p0EI5N08d -Doceanbase -A

Connect to obproxy ok
+--------------------------------------------------+
|                     obproxy                      |
+----------------+------+-----------------+--------+
| ip             | port | prometheus_port | status |
+----------------+------+-----------------+--------+
| 192.168.152.30 | 3883 | 3884            | active |
+----------------+------+-----------------+--------+

若

检查 OceanBase 集群各个节点信息

i.进程
OceanBase 数据库是单进程软件，进程名为 observer ，可运行以下命令查看这个进程。

IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done

查看observer启动目录
ls -l /proc/进程号/cwd ，与配置文件一致
[admin@obce01 ~]$ cat obcluster-3zones.yaml |grep home
    home_path: /home/admin/oceanbase-ce

2.监听

IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done

3.查看目录结构

tree ~/oceanbase-ce/ /data/ /redo/ ~/obproxy/

使用ob集群

i.直连ob集群、连接obproxy集群以及通过obproxy代理连ob集群的方法

1.直连 OceanBase 集群的内部实例（sys)，并查看相关资源
obclient -h192.168.152.30 -P3881 -uroot -p0EI5N08d -Doceanbase -A

select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024) 
mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version
from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)order by a.zone, a.svr_ip;

---

2.连接obproxy集群

obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p0MdTv1tm

发现报密码相关错误：

原因如下：ymal文件在集群初始化时报错，obproxy的obproxy_sys_password和observer_sys_password被我从ymal文件注释掉了。proxysys 的 root密码为空。
obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p
Enter password:此处直接回车即可

[admin@obce01 ~]$ obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p
Enter password:
此处省略部分显示
MySQL [(none)]> show proxyconfig like '%sys_password%';
+-----------------------+-------+--------------------------------+-------------+---------------+
| name                  | value | info                           | need_reboot | visible_level |
+-----------------------+-------+--------------------------------+-------------+---------------+
| observer_sys_password |       | password for observer sys user | false       | SYS           |     ======================value为空
| obproxy_sys_password  |       | password for obproxy sys user  | false       | SYS           |
+-----------------------+-------+--------------------------------+-------------+---------------+     ======================value为空
2 rows in set (0.006 sec)

MySQL [(none)]> alter  proxyconfig  set obproxy_sys_password="0MdTv1tm";      
Query OK, 0 rows affected (0.007 sec)

MySQL [(none)]> alter  proxyconfig  set observer_sys_password="uY7Yf8zx";                          =====================consistent with oceanbase-ce's proxyro_password
Query OK, 0 rows affected (0.008 sec)

MySQL [(none)]> show proxyconfig like '%sys_password%';
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
| name                  | value                                    | info                           | need_reboot | visible_level |
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
| observer_sys_password | 34864da3a09bda2d587ac2c9d8e4521981586fc3 | password for observer sys user | false       | SYS           |
| obproxy_sys_password  | 9fe82c02e25ae612ed310ab0ac81c97bd813dd4e | password for obproxy sys user  | false       | SYS           |
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
2 rows in set (0.015 sec)

MySQL [(none)]> Bye

---

带上密码重新尝试连接proxysys的root用户

obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p0MdTv1tm 成功

3.使用obclient通过obproxy连接集群

obclient -h 192.168.152.30 -uroot@sys#obcluster-3zones -P3883 -p0EI5N08d -c -A oceanbase  
此处在通过obproxy登录observer，用户为root,密码为observer的root_password: 0EI5N08d(不是proxyro的password)

ii.简单使用运维命令

查看参数
show parameters like '%_password%';
show parameters like 'memory_limit%';
show parameters where name in ('memory_limit','memory_limit_ percentage');
show parameters like 'config_additional_dir';
修改参数
alter system set syslog_level = 'USER_ERR' server='192.168.152.30:3882';

iii.OceanBase 集群参数文件

show parameters like 'config_additional_dir'

通常 OceanBase 集群每个节点的启动目录下都会有一个目录 etc，保存了该节点进程的参数文件observer.config.bin。
observer.config.bin 是一个 binary 类型的文件,不能直接用 cat命令读取，可使用 strings 命令读取。
该文件也不建议直接修改，可通过上面提到的命令进行修改 alter 。
observer进程默认会读取文件夹 etc 中的参数文件，其他两个目录是参数文件的备份，这个备份路径也是通过参数config_additional_dir 指定的，默认值是同一个启动目录的 etc2 和 etc3。
生产环境建议将 etc设置到其他磁盘，这样会更加安全。

[admin@obce01 etc]$ pwd
/home/admin/oceanbase-ce/etc
[admin@obce01 etc]$ strings observer.config.bin
_enable_oracle_priv_check=True
_max_trx_size=100M
all_server_list=192.168.152.30:3882,192.168.152.31:3882,192.168.152.32:3882
__min_full_resource_pool_memory=268435456
min_observer_version=3.1.0
enable_syslog_recycle=True
enable_syslog_wf=False
max_syslog_file_count=4
syslog_level=WARN
cluster_id=2
cluster=obcluster-3zones
rootservice_list=192.168.152.30:3882:3881;192.168.152.31:3882:3881;192.168.152.32:3882:3881
_partition_balance_strategy=standard
enable_one_phase_commit=False
cpu_count=16
system_memory=3G
memory_limit=8G
zone=zone1
devname=ens33
mysql_port=3881
rpc_port=3882
datafile_size=10G
data_dir=/data

使用 OBD 修改 OceanBase 集群参数：

obd cluster edit-config obcluster-3zones
obd cluster reload obcluster-3zones

手动启动observer进程（应急）

进入home_path:/home/admin/oceanbase-ce
cd /home/admin/oceanbase-ce
bin/observer -o "max_syslog_file_count=4,datafile_size=10G"