Centos 7 部署 OpenStack_Rocky版高可用集群3-1
Centos 7 部署 OpenStack_Rocky版高可用集群3-1
文章目录
- Centos 7 部署 OpenStack_Rocky版高可用集群3-1
- 1、前期准备
- 1.1 软硬件准备
- 1.1.1组件
- 1.1.2 网络拓扑（逻辑）
- 1.1.3 物理拓扑（部分参考）
- 1.2 配置hosts文件(集群所有电脑配置)
- 1.3 节点间设置免密互相登陆
- 1.4 配置NTP服务
- 1.4.1 在控制节点进行配置（cont01、cont02、cont03）
- 1.4.2 在其他节点上安装NTP软件（mon01、comp01、comp02、comp03）
- 1.5 系统初始设置
- 1.5.1 所有节点关闭防火墙和selinux
- 2、部署MariaDB Galeria集群
- 2.1 控制节点安装mariadb(cont01、cont02、cont03)
- 2.1.1 控制节点添加mariadb-10.4.11的yum源
- 2.1.2 控制节点yum安装mariadb软件包并开启服务
- 2.1.3 控制节点数据库安全设置、登陆数据库设置密码并授权
- 2.1.4 配置MariaDB Galeria 集群 (控制节点全修改配置文件/etc/my.cnf.d/server.cnf)
- 2.1.5 控制节点停止mariadb.service
- 2.1.6 启动初始化集群
- 2.2 登录数据库，查看集群状态
- 2.3 每个控制节点/usr/bin 目录下面编辑心跳检测clustercheck脚本
- 3、部署xinetd服务
- 3.1 控制节点安装xinetd服务，用于监控Galera服务(cont01、cont02、cont03)
- 3.2 控制节点创建xinetd配置文件
- 3.3 控制节点修改/etc/services
- 3.4 控制节点启动xinetd服务(cont01、cont02、cont03)
- 3.5 故障场景的恢复
- 4、部署rabbitmq消息队列
- 4.1、所有控制节点安装rabbitmq和erlang软件包（cont01、cont02、cont03、~~mon01~~）
- 4.2 查看集群状态
- 4.3 编辑rabbitmq变量文件
- 4.4 查看rabbitmq插件
- 4.5 所有node开启rabbitmq的web管理页面
- 4.6 查看rabbitmq插件是否加载成功（成功后前面有E）
- 4.7 访问：http://192.168.10.22:15672/
- 4.8 所有node修改guest密码为adminrabbitmq(默认用户为：guest 密码为：guest)
- 4.9 cont02添加一个openstack的用户，并设密码为adminopenstack。并设置权限成为管理员(成集群后，cont02和cont03上也可以登录)
- 4.10 发送cont01上的erlang.cookie到其他节点配置集群
- 4.11 cont01、cont03、mon01停止应用，并ram的方式加入cont01节点，之后重启应用（先查看cont01集群名字）
- 4.12 检测集群状态
- 4.13 为了保持rabbitmq的消息一致性，可以设置消息镜像模式策略 所有节点执行
- 4.14 RabbitMQ注意小计（故障恢复提示）：
- 5、部署PCS
- 5.1 安装pcs+pacemaker+corosync （cont01、cont02和 cont03）
- 5.2 设置集群密码，而且三个节点密码需一直为：pcs#typora
- 5.3 控制节点创建配置文件corosync.conf
- 5.4 配置集群，设置集群互相认证
- 5.5 配置节点认证
- 5.6 创建集群
- 5.7 启动集群并查看集群状态
- 5.8 检验corosync状态
- 5.9 创建集群VIP （预设VIP地址为192.168.10.20）
- 6、部署Haproxy
- 6.1 所有控制节点安装Haproxy，并保持相同的配置，对于配置文件，建议根据实际情况进行优化。
- 6.1.1安装并配置haproxy
- 6.2 配置Haproxy能监控Galera数据库集群
- 6.3 开启haproxy,并自启动（cont01、cont02、cont03）
- 6.4 登陆http://192.168.10.20:8789/ 检测状态 默认账户:admin 密码:admin
- 7、部署Keystone
- 7.1 控制节点安装Enable the OpenStack repository(cont01和cont02)
- 7.2 控制节点安装 SQL database(cont01和cont02)
- 7.3 安装Memcached (cont01、cont02和cont03)
- 7.4 configure the Identity service
- 7.5 Install and configure components
- 7.6 Configure the Apache HTTP server [cont0$表示cont01、cont02和cont03]
- 7.7 Create a domain, projects, users, and roles
1、前期准备
1.1 软硬件准备
控制主机三台（三网卡）+计算节点三台（双网卡、兼分布式存储OSD）+存储节点一台（双网卡部署MOD）
1.1.1组件
| 组件 | 版本 | 备注 |
|---|---|---|
| CentOS | CentOS Linux release 7.7.1908 (Core) | |
| Openstack | Rocky | |
| Ceph | ceph version 12.2.12 luminous (stable) |
1.1.2 网络拓扑（逻辑）
1.控制节点(cont01、cont02、cont03)运行MariaDB galeria,Haproxy,pacemaker,corosync,keystone, glance,horizon,nova&neutron&cinder管理相关组件和openstack相关的基础服务
2.计算节点(comp01、comp02、comp03)运行Nova-compute,neutron-openswitch-agent,ceph-osd等
3.存储控制节点(mon01)运行ceph-mon,ceph-mgr(如生产环境，建议也采用三个节点部署)
4.控制节点网络：
管理网络–>eth0:192.168.10.0/24 用于对openstack集群host os,api等网络管理
租户隧道VM网络–>eth1:10.10.10.0/24 用于集群内部的guest os之间的数据交换，采用vxlan/gre等方式
外部External网络–>eth2:主要用户计算节点访问internet/外部的floating ip
ps:此处的管理网络合并Management网络+API网络
5.计算节点网络：
管理网络–>eth0:192.168.10.0/24 用于对openstack集群host os,api等网络管理，
租户隧道网络–>eth1:10.10.10.0/24 用于集群内部的guest os之间的数据交换，采用vxlan/gre等方式
6.前端采用Haproxy做高可用，数据库采用MariaDB galeria，存储采用ceph分布式存储集群
7.无状态的服务，如xxx-api，采取active/active的模式运行；有状态的服务，如neturon-xxx-agent，cinder-volume等，建议采取active/passive的模式运行（因前端采用haproxy，客户端的多次请求可能会被转发到不同的控制节点，如果客户端请求被负载到无状态信息的控制节点，可能会导致操作请求失败）；自身具有集群机制的服务，如rabbitmq，memcached等采用本身的集群机制。
1.1.3 物理拓扑（部分参考）
1.2 配置hosts文件(集群所有电脑配置)
[root@cont02:/root]# vim /etc/hosts
192.168.10.21 cont01
192.168.10.22 cont02
192.168.10.23 cont03
192.168.10.20 VirtualIP
192.168.10.24 mon01
192.168.10.19 comp01
192.168.10.18 comp02
192.168.10.17 comp03
[root@cont02:/root]# scp /etc/hosts cont02:/etc/
[root@cont02:/root]# scp /etc/hosts cont03:/etc/
[root@cont02:/root]# scp /etc/hosts mon01:/etc/
[root@cont02:/root]# scp /etc/hosts comp01:/etc/
[root@cont02:/root]# scp /etc/hosts comp02:/etc/
[root@cont02:/root]# scp /etc/hosts comp03:/etc/
1.3 节点间设置免密互相登陆
[root@cont01:/root]# ssh-keygen
ssh-copy-id cont02
ssh-copy-id cont03
ssh-copy-id mon01
ssh-copy-id comp01
ssh-copy-id comp02
ssh-copy-id comp03
[root@cont02:/root]# ssh-keygen
ssh-copy-id cont01
ssh-copy-id cont03
ssh-copy-id mon01
ssh-copy-id comp01
ssh-copy-id comp02
ssh-copy-id comp03
[root@cont03:/root]# ssh-keygen
ssh-copy-id cont01
ssh-copy-id cont02
ssh-copy-id mon01
ssh-copy-id comp01
ssh-copy-id comp02
ssh-copy-id comp03
1.4 配置NTP服务
1.4.1 在控制节点进行配置（cont01、cont02、cont03）
[root@cont02:/root]# yum install system-config-date -y
[root@cont02:/root]# vim /etc/chrony.conf
3 server 0.centos.pool.ntp.org iburst
4 server 192.168.10.21 iburst
5 server 192.168.10.22 iburst
6 server 192.168.10.23 iburst
25 # Allow NTP client access from local network.
26 allow 192.168.0.0/16
[root@cont02:/root]# scp /etc/chrony.conf cont01:/etc/chrony.conf
[root@cont02:/root]# scp /etc/chrony.conf cont03:/etc/chrony.conf
[root@cont01:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@cont01:/root]# systemctl status chronyd
[root@cont02:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@cont02:/root]# systemctl status chronyd
[root@cont03:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@cont03:/root]# systemctl status chronyd
[root@cont01:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont02 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont03 0 7 0 - +0ns[ +0ns] +/- 0ns
^* 119.28.206.193 2 6 37 21 +197us[+1847us] +/- 55ms
[root@cont02:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont01 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont03 0 7 0 - +0ns[ +0ns] +/- 0ns
^* 162.159.200.123 3 6 37 51 +2273us[ +143us] +/- 121ms
[root@cont03:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont01 0 8 0 - +0ns[ +0ns] +/- 0ns
^? cont02 0 8 0 - +0ns[ +0ns] +/- 0ns
^? tick.ntp.infomaniak.ch 0 8 0 - +0ns[ +0ns] +/- 0ns
[root@cont02:/root]# ssh cont01 'date'; ssh cont03 'date' ; date
Tue Feb 11 20:32:16 CST 2020
Tue Feb 11 20:32:16 CST 2020
Tue Feb 11 20:32:16 CST 2020
1.4.2 在其他节点上安装NTP软件（mon01、comp01、comp02、comp03）
[root@mon01:/root]# yum install chrony -y
[root@comp01:/root]# yum install chrony -y
[root@comp02:/root]# yum install chrony -y
[root@comp03:/root]# yum install chrony -y
[root@mon01:/root]# vim /etc/chrony.conf
3 server 192.168.10.22 iburst
4 server 192.168.10.21 iburst
5 server 192.168.10.23 iburst
6 #server 3.centos.pool.ntp.org iburst
[root@comp01:/root]# vim /etc/chrony.conf
3 server 192.168.10.22 iburst
4 server 192.168.10.21 iburst
5 server 192.168.10.23 iburst
6 #server 3.centos.pool.ntp.org iburst
[root@comp02:/root]# vim /etc/chrony.conf
3 server 192.168.10.22 iburst
4 server 192.168.10.21 iburst
5 server 192.168.10.23 iburst
6 #server 3.centos.pool.ntp.org iburst
[root@comp03:/root]# vim /etc/chrony.conf
3 server 192.168.10.22 iburst
4 server 192.168.10.21 iburst
5 server 192.168.10.23 iburst
6 #server 3.centos.pool.ntp.org iburst
[root@mon01:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@mon01:/root]# systemctl status chronyd
[root@comp01:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@comp01:/root]# systemctl status chronyd
[root@comp02:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@comp02:/root]# systemctl status chronyd
[root@comp03:/root]# systemctl enable chronyd && systemctl restart chronyd
[root@comp03:/root]# systemctl status chronyd
[root@mon01:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont02 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont01 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont03 0 7 0 - +0ns[ +0ns] +/- 0ns
[root@mon01:/root]# timedatectl
Local time: Wed 2020-01-15 11:22:23 CST
Universal time: Wed 2020-01-15 03:22:23 UTC
RTC time: Wed 2020-01-15 03:22:24
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
[root@comp01:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont02 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont01 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont03 0 7 0 - +0ns[ +0ns] +/- 0ns
[root@comp01:/root]# timedatectl
Local time: Wed 2020-01-15 11:22:24 CST
Universal time: Wed 2020-01-15 03:22:24 UTC
RTC time: Wed 2020-01-15 03:22:24
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
[root@comp02:/root]# chronyc sources
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? cont02 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont01 0 7 0 - +0ns[ +0ns] +/- 0ns
^? cont03 0 7 0 - +0ns[ +0ns] +/- 0ns
[root@comp02:/root]# timedatectl
Local time: Wed 2020-01-15 11:22:24 CST
Universal time: Wed 2020-01-15 03:22:24 UTC
RTC time: Wed 2020-01-15 03:22:24
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
1.5 系统初始设置
1.5.1 所有节点关闭防火墙和selinux
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
1.5.2 所有节点编辑limits.conf和sysctl.conf文件
cat >>/etc/security/limits.conf<>/etc/sysctl.conf< 2、部署MariaDB Galeria集群
2.1 控制节点安装mariadb(cont01、cont02、cont03)
Galera是Galera Cluster，是一种为数据库设计的新型的、数据不共享的、高度冗余的高可用方案，galera mariadb就是集成了Galera插件的MariaDB集群，Galera本身是具有多主特性的，所以galera mariadb不是传统的主备模式的集群，而是多主节点架构。
2.1.1 控制节点添加mariadb-10.4.11的yum源
[root@cont01:/root]# vim /etc/yum.repos.d/mariadb.repo
[mariadb]
name = MariaDB
baseurl = http://mirror.netinch.com/pub/mariadb//mariadb-10.4.11/yum/centos7-amd64/
gpgcheck = 0
enabled = 1
[root@cont01:/root]# yum makecache
[root@cont02:/root]#
cat > /etc/yum.repos.d/mariadb.repo << EOF
[mariadb]
name = MariaDB
baseurl = http://mirror.netinch.com/pub/mariadb//mariadb-10.4.11/yum/centos7-amd64/
gpgcheck = 0
enabled = 1
EOF
[root@cont02:/root]# yum makecache
[root@cont02:/root]# scp /etc/yum.repos.d/mariadb.repo cont03:/etc/yum.repos.d/
[root@cont03:/root]# yum makecache
2.1.2 控制节点yum安装mariadb软件包并开启服务
[root@cont01:/root]# yum -y install MariaDB-server MariaDB-client galera-4 rsync xinetd wget ntpdate
[root@cont01:/root]# systemctl enable mariadb && systemctl start mariadb
[root@cont01:/root]# systemctl status mariadb
[root@cont02:/root]# yum -y install MariaDB-server MariaDB-client galera-4 rsync xinetd wget ntpdate
[root@cont02:/root]# systemctl enable mariadb && systemctl start mariadb
[root@cont02:/root]# systemctl status mariadb
[root@cont03:/root]# yum -y install MariaDB-server MariaDB-client galera-4 rsync xinetd wget ntpdate
[root@cont03:/root]# systemctl enable mariadb && systemctl start mariadb
[root@cont03:/root]# systemctl status mariadb
2.1.3 控制节点数据库安全设置、登陆数据库设置密码并授权
[root@cont02:/root]# mysql_secure_installation
注：Disallow root login remotely? [Y/n] n
密码：typora#2019
[root@cont02:/root]# mysql
MariaDB [(none)]> SET password=PASSWORD('typora#2019');
//密码设置过的可以省略
Query OK, 0 rows affected (0.002 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.002 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.002 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> exit
Bye
[root@cont01:/root]# mysql_secure_installation
注：Disallow root login remotely? [Y/n] n
密码：typora#2019
[root@cont01:/root]# mysql
MariaDB [(none)]> SET password=PASSWORD('typora#2019');
//密码设置过的可以省略
Query OK, 0 rows affected (0.012 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.002 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> exit
Bye
[root@cont03:/root]# mysql_secure_installation
注：Disallow root login remotely? [Y/n] n
密码：typora#2019
[root@cont03:/root]# mysql
MariaDB [(none)]> SET password=PASSWORD('typora#2019');
//密码设置过的可以省略
Query OK, 0 rows affected (0.009 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.002 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'localhost' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> grant all privileges on *.* to 'root'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.016 sec)
MariaDB [(none)]> grant all privileges on *.* to 'mysql'@'%' identified by 'typora#2019' with grant option;
Query OK, 0 rows affected (0.008 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> exit
Bye
2.1.4 配置MariaDB Galeria 集群 (控制节点全修改配置文件/etc/my.cnf.d/server.cnf)
[root@cont02:/root]# vim /etc/my.cnf.d/server.cnf
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Galera-related settings
#
[galera]
# Mandatory settings
##wsrep_on=ON #开启全同步复制模式
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.10.22,192.168.10.23,192.168.10.21"
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
##innodb_autoinc_lock_mode=2 #主键自增模式修改为交叉模式
##wsrep_slave_threads=8 #开启并行复制线程，根据CPU核数设置
#
# Allow server to accept connections on all interfaces.
#
bind-address=192.168.10.22
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0 #事务提交每隔1秒刷盘
##wsrep_certify_nonPK=ON #为没有显式申明主键的表生成一个用于certificationtest的主键，默认为ON
##query_cache_size=0 #关闭查询缓存
##wsrep_provider_options="gcache.size=4G" #同步复制缓冲池
wsrep_provider_options="gcache.size=1G"
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_node_name=cont02
wsrep_node_address=192.168.10.22
wsrep_sst_method=rsync
##innodb_buffer_pool_size=2G
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.4 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.4]
[root@cont01:/root]# vim /etc/my.cnf.d/server.cnf
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.10.21,192.168.10.22,192.168.10.23"
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
bind-address=192.168.10.21
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
wsrep_provider_options="gcache.size=1G"
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_node_name=cont01
wsrep_node_address=192.168.10.21
wsrep_sst_method=rsync
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.4 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.4]
[root@cont03:/root]# vim /etc/my.cnf.d/server.cnf
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.10.23,192.168.10.22,192.168.10.21"
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
bind-address=192.168.10.23
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
wsrep_provider_options="gcache.size=1G"
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_node_name=cont03
wsrep_node_address=192.168.10.23
wsrep_sst_method=rsync
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.4 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.4]
2.1.5 控制节点停止mariadb.service
[root@cont01:/root]# systemctl daemon-reload
[root@cont01:/root]# systemctl stop mariadb.service
[root@cont02:/root]# systemctl daemon-reload
[root@cont02:/root]# systemctl stop mariadb.service
[root@cont03:/root]# systemctl daemon-reload
[root@cont03:/root]# systemctl stop mariadb.service
2.1.6 启动初始化集群
[root@cont02:/root]# mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql --wsrep-new-cluster &
[root@cont03:/root]# systemctl start mariadb
[root@cont03:/root]# systemctl status mariadb
[root@cont03:/root]# systemctl enable mariadb
[root@cont01:/root]# systemctl start mariadb
[root@cont01:/root]# systemctl status mariadb
[root@cont01:/root]# systemctl enable mariadb
注：两个节点启动成功后，在重新启动第一个节点：（删除第一个节点数据）
[root@cont02:/root]# pkill -9 mysql
[1]+ Killed mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql --wsrep-new-cluster
[root@cont02:/root]# rm -rf /var/lib/mysql/*
[root@cont02:/root]# systemctl start mariadb.service
[root@cont02:/root]# systemctl enable mariadb.service
Created symlink from /etc/systemd/system/mysql.service to /usr/lib/systemd/system/mariadb.service.
Created symlink from /etc/systemd/system/mysqld.service to /usr/lib/systemd/system/mariadb.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@cont02:/root]# systemctl status mariadb.service
注意：每次在第一台集群上，需要使用下面的命令来启动：
galera_new_cluster
2.2 登录数据库，查看集群状态
[root@cont02:/root]# mysql -uroot -p"typora#2019"
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.4.11-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.002 sec)
MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.002 sec)
MariaDB [(none)]> show status like 'wsrep%';
+-------------------------------+----------------------------------------------------
| Variable_name | Value
+-------------------------------+----------------------------------------------------
| wsrep_local_state_uuid | a2b7231d-383a-11ea-b086-83d5778583b5
| wsrep_protocol_version | 10
| wsrep_last_committed | 5
| wsrep_replicated | 0
| wsrep_replicated_bytes | 0
| wsrep_repl_keys | 0
| wsrep_repl_keys_bytes | 0
| wsrep_repl_data_bytes | 0
| wsrep_repl_other_bytes | 0
| wsrep_received | 3
| wsrep_received_bytes | 256
| wsrep_local_commits | 0
| wsrep_local_cert_failures | 0
| wsrep_local_replays | 0
| wsrep_local_send_queue | 0
| wsrep_local_send_queue_max | 1
| wsrep_local_send_queue_min | 0
| wsrep_local_send_queue_avg | 0
| wsrep_local_recv_queue | 0
| wsrep_local_recv_queue_max | 1
| wsrep_local_recv_queue_min | 0
| wsrep_local_recv_queue_avg | 0
| wsrep_local_cached_downto | 5
| wsrep_flow_control_paused_ns | 0
| wsrep_flow_control_paused | 0
| wsrep_flow_control_sent | 0
| wsrep_flow_control_recv | 0
| wsrep_cert_deps_distance | 0
| wsrep_apply_oooe | 0
| wsrep_apply_oool | 0
| wsrep_apply_window | 0
| wsrep_commit_oooe | 0
| wsrep_commit_oool | 0
| wsrep_commit_window | 0
| wsrep_local_state | 4
| wsrep_local_state_comment | Synced
| wsrep_cert_index_size | 0
| wsrep_causal_reads | 0
| wsrep_cert_interval | 0
| wsrep_open_transactions | 0
| wsrep_open_connections | 0
| wsrep_incoming_addresses | AUTO,AUTO,AUTO
| wsrep_cluster_weight | 3
| wsrep_desync_count | 0
| wsrep_evs_delayed |
| wsrep_evs_evict_list |
| wsrep_evs_repl_latency | 0/0/0/0/0
| wsrep_evs_state | OPERATIONAL
| wsrep_gcomm_uuid | d5a23bb7-383a-11ea-b735-86c7a20615b4
| wsrep_applier_thread_count | 1
| wsrep_cluster_capabilities |
| wsrep_cluster_conf_id | 18446744073709551615
| wsrep_cluster_size | 3
| wsrep_cluster_state_uuid | a2b7231d-383a-11ea-b086-83d5778583b5
| wsrep_cluster_status | Primary
| wsrep_connected | ON
| wsrep_local_bf_aborts | 0
| wsrep_local_index | 0
| wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
| wsrep_provider_name | Galera
| wsrep_provider_vendor | Codership Oy
| wsrep_provider_version | 26.4.3(r4535)
| wsrep_ready | ON
| wsrep_rollbacker_thread_count | 1
| wsrep_thread_count | 2 +-------------------------------+---------------------------------------------
65 rows in set (0.002 sec)
MariaDB [(none)]> exit
Bye
[root@cont01:/var/lib/mysql]# cat grastate.dat gvwstate.dat
# GALERA saved state
version: 2.1
uuid: a2b7231d-383a-11ea-b086-83d5778583b5
seqno: -1
safe_to_bootstrap: 0
my_uuid: b3a02e5e-383a-11ea-ab01-07face985b92
#vwbeg
view_id: 3 ac080e34-383a-11ea-9d9d-22f77c5f5aa6 5
bootstrap: 0
member: ac080e34-383a-11ea-9d9d-22f77c5f5aa6 0
member: b3a02e5e-383a-11ea-ab01-07face985b92 0
member: d5a23bb7-383a-11ea-b735-86c7a20615b4 0
#vwend
[root@cont02:/var/lib/mysql]# cat grastate.dat gvwstate.dat
# GALERA saved state
version: 2.1
uuid: a2b7231d-383a-11ea-b086-83d5778583b5
seqno: -1
safe_to_bootstrap: 0
my_uuid: d5a23bb7-383a-11ea-b735-86c7a20615b4
#vwbeg
view_id: 3 ac080e34-383a-11ea-9d9d-22f77c5f5aa6 5
bootstrap: 0
member: ac080e34-383a-11ea-9d9d-22f77c5f5aa6 0
member: b3a02e5e-383a-11ea-ab01-07face985b92 0
member: d5a23bb7-383a-11ea-b735-86c7a20615b4 0
#vwend
[root@cont03:/var/lib/mysql]# cat grastate.dat gvwstate.dat
# GALERA saved state
version: 2.1
uuid: a2b7231d-383a-11ea-b086-83d5778583b5
seqno: -1
safe_to_bootstrap: 0
my_uuid: ac080e34-383a-11ea-9d9d-22f77c5f5aa6
#vwbeg
view_id: 3 ac080e34-383a-11ea-9d9d-22f77c5f5aa6 5
bootstrap: 0
member: ac080e34-383a-11ea-9d9d-22f77c5f5aa6 0
member: b3a02e5e-383a-11ea-ab01-07face985b92 0
member: d5a23bb7-383a-11ea-b735-86c7a20615b4 0
#vwend
2.3 每个控制节点/usr/bin 目录下面编辑心跳检测clustercheck脚本
[root@cont01:/root]# cd /usr/bin/
[root@cont01:/usr/bin]# wget https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck
[root@cont01:/usr/bin]# chmod +x /usr/bin/clustercheck
[root@cont01:/usr/bin]# ls -ld /usr/bin/clustercheck
-rwxr-xr-x. 1 root root 3882 Jan 15 19:19 /usr/bin/clustercheck
[root@cont02:/root]# cd /usr/bin/
[root@cont02:/usr/bin]# wget https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck
[root@cont02:/usr/bin]# chmod +x /usr/bin/clustercheck
[root@cont02:/usr/bin]# ls -ld /usr/bin/clustercheck
-rwxr-xr-x. 1 root root 3882 Jan 15 19:18 /usr/bin/clustercheck
[root@cont03:/root]# cd /usr/bin/
[root@cont03:/usr/bin]# wget https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck
[root@cont03:/usr/bin]# chmod +x /usr/bin/clustercheck
[root@cont03:/usr/bin]# ls -ld /usr/bin/clustercheck
-rwxr-xr-x. 1 root root 3882 Jan 15 19:20 /usr/bin/clustercheck
3、部署xinetd服务
3.1 控制节点安装xinetd服务，用于监控Galera服务(cont01、cont02、cont03)
yum install xinetd -y
3.2 控制节点创建xinetd配置文件
[root@cont01:/root]#
cat </etc/xinetd.d/mysqlchk
#default:on
##description:mysqlchk
service mysqlchk
{
disable = no
flags = REUSE
socket_type = stream
port = 9200
wait = no
user = nobody
server = /usr/bin/clustercheck
log_on_failure += USERID
only_from = 0.0.0.0/0
per_source = UNLIMITED
}
END
[root@cont01:/root]# scp /etc/xinetd.d/mysqlchk cont02:/etc/xinetd.d/
[root@cont01:/root]# scp /etc/xinetd.d/mysqlchk cont03:/etc/xinetd.d/
3.3 控制节点修改/etc/services
echo ‘mysqlchk 9200/tcp #mysqlchk’>>/etc/services
[root@cont02:/root]# vim /etc/services
转到第10101行，注释使用端口9200的服务，并为mysqlchk / clustercheck添加一行。
mysqlchk 9200/tcp # mysqlchk
#wap-wsp 9200/tcp # WAP connectionless session service
#wap-wsp 9200/udp # WAP connectionless session service
[root@cont01:/root]# vim /etc/services
转到第10101行，注释使用端口9200的服务，并为mysqlchk / clustercheck添加一行。
mysqlchk 9200/tcp # mysqlchk
#wap-wsp 9200/tcp # WAP connectionless session service
#wap-wsp 9200/udp # WAP connectionless session service
[root@cont03:/root]# vim /etc/services
转到第10101行，注释使用端口9200的服务，并为mysqlchk / clustercheck添加一行。
mysqlchk 9200/tcp # mysqlchk
#wap-wsp 9200/tcp # WAP connectionless session service
#wap-wsp 9200/udp # WAP connectionless session service
3.4 控制节点启动xinetd服务(cont01、cont02、cont03)
systemctl enable xinetd && systemctl start xinetd
systemctl status xinetd
3.5 故障场景的恢复
①首先判断当前数据库集群中是否有服务在启动着，如果有则直接启动服务即可，其中1个节点挂了，一般只需要重启A节点的服务即可。
② 如果当前所有节点的数据库服务都挂了，则需要找状态最新的那个节点让它携带–wsrep-new-cluster参数启动，启动起来之后其它节点直接启动服务即可。
查找最新节点策略：首先获取各节点的grastate.dat文件中的seqno值，值最大的那个就是最新的节点；如果所有节点的seqno都是-1，则去比较所有节点的gvwstate.dat文件中的my_uuid和view_id是否相等，相等的那个则作为第一个启动节点，第一个启动节点启动后，其它节点正常启动即可；如果依然未找到则需要人工干预来恢复了。//启动 mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql --wsrep-new-cluster &③如果以上两种方法全部无效。尝试采用下面操作(进过测试，此方法成功过)
第一步：数据库无法启动情况下把/var/lib/mysql下的几个文件：grastate.dat gvwstate.dat galera.cache mysql.sock统统删掉再重启
第二步：更改配置文件中的 “gcomm://” 是特殊的地址，仅仅是galera cluster初始化启动时候使用。如果集群启动以后，我们关闭了第一个节点，那么再次启动的时候必须先修改"gcomm://"为其他节点的集群地址，首次作为发起人则此处为空，不向任何人同步数据。
第三步：本机启动服务：systemctl start mariadb
第四部：启动集群：galera_new_cluster
第五步：启动其余节点上的mariadb: systemctl start mariadb
第六步：更改回 “gcomm://” 重启数据库systemctl restart mariadb
监测状态说明
集群完整性检查:
wsrep_cluster_state_uuid:在集群所有节点的值应该是相同的,有不同值的节点,说明其没有连接入集群.
wsrep_cluster_conf_id:正常情况下所有节点上该值是一样的.如果值不同,说明该节点被临时”分区”了.当节点之间网络连接恢复的时候应该会恢复一样的值.
wsrep_cluster_size:如果这个值跟预期的节点数一致,则所有的集群节点已经连接.
wsrep_cluster_status:集群组成的状态.如果不为”Primary”,说明出现”分区”或是”split-brain”状况.
节点状态检查:
wsrep_ready: 该值为ON,则说明可以接受SQL负载.如果为Off,则需要检查wsrep_connected.
wsrep_connected: 如果该值为Off,且wsrep_ready的值也为Off,则说明该节点没有连接到集群.(可能是wsrep_cluster_address或wsrep_cluster_name等配置错造成的.具体错误需要查看错误日志)
wsrep_local_state_comment:如果wsrep_connected为On,但wsrep_ready为OFF,则可以从该项查看原因.
复制健康检查:
wsrep_flow_control_paused:表示复制停止了多长时间.即表明集群因为Slave延迟而慢的程度.值为0~1,越靠近0越好,值为1表示复制完全停止.可优化wsrep_slave_threads的值来改善.
wsrep_cert_deps_distance:有多少事务可以并行应用处理.wsrep_slave_threads设置的值不应该高出该值太多.
wsrep_flow_control_sent:表示该节点已经停止复制了多少次.
wsrep_local_recv_queue_avg:表示slave事务队列的平均长度.slave瓶颈的预兆.
当前节点数据库状态
MariaDB [(none)]> show status like 'wsrep_local_state_comment';
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| wsrep_local_state_comment | Synced |
+---------------------------+--------+
MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.001 sec)
MariaDB [(none)]> show status like 'wsrep_cluster_state_uuid';
MariaDB [(none)]> show status like 'wsrep_cluster_status';
状态查询表：
Open :节点启动成功，尝试连接到集群
Primary :节点已处于集群中，在新节点加入时，选取donor进行数据库同步时会产生的状态
Joiner :节点处于等待接收或正在接收同步文件的状态
Joined :节点完成数据同步，但还有部分数据不是最新的，在追赶与集群数据一致的状态
Synced :节点正常提供服务的状态，表示当前节点数据状态与集群数据状态是一致的
Donor :表示该节点被选为Donor节点，正在为新加进来的节点进行全量数据同步，此时该节点对客户端不提供服务
4、部署rabbitmq消息队列
4.1、所有控制节点安装rabbitmq和erlang软件包（cont01、cont02、cont03、mon01）
//节点：cont01
[root@cont01:/root]# yum install epel-release -y
[root@cont01:/root]# yum install -y erlang rabbitmq-server.noarch
[root@cont01:/root]# systemctl enable rabbitmq-server && systemctl start rabbitmq-server
[root@cont01:/root]# systemctl status rabbitmq-server
[root@cont01:/root]# netstat -ntlpa | grep 5672
//节点：cont02
[root@cont02:/root]# yum install epel-release -y
[root@cont02:/root]# yum install -y erlang rabbitmq-server.noarch
[root@cont02:/root]# systemctl enable rabbitmq-server && systemctl start rabbitmq-server
[root@cont02:/root]# systemctl status rabbitmq-server
[root@cont02:/root]# netstat -ntlpa | grep 5672
//节点：cont03
[root@cont03:/root]# yum install epel-release -y
[root@cont03:/root]# yum install -y erlang rabbitmq-server.noarch
[root@cont03:/root]# systemctl enable rabbitmq-server && systemctl start rabbitmq-server
[root@cont03:/root]# systemctl status rabbitmq-server
[root@cont03:/root]# netstat -ntlpa | grep 5672
//节点：mon01
[root@mon01:/root]# yum install epel-release -y
[root@mon01:/root]# yum install -y erlang rabbitmq-server.noarch
[root@mon01:/root]# systemctl enable rabbitmq-server && systemctl start rabbitmq-server
[root@mon01:/root]# systemctl status rabbitmq-server
[root@mon01:/root]# netstat -ntlpa | grep 5672
4.2 查看集群状态
（提示：Rabbitmq的配置文件/etc/rabbitmq/rabbitmq.config）
[root@cont01:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont01 ...
[{nodes,[{disc,[rabbit@cont01]}]},
{running_nodes,[rabbit@cont01]},
{cluster_name,<<"rabbit@cont01">>},
{partitions,[]}]
...done.
[root@cont02:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont02 ...
[{nodes,[{disc,[rabbit@cont02]}]},
{running_nodes,[rabbit@cont02]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
[root@cont03:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont03 ...
[{nodes,[{disc,[rabbit@cont03]}]},
{running_nodes,[rabbit@cont03]},
{cluster_name,<<"rabbit@cont03">>},
{partitions,[]}]
...done.
[root@mon01:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@mon01 ...
[{nodes,[{disc,[rabbit@mon01]}]},
{running_nodes,[rabbit@mon01]},
{cluster_name,<<"rabbit@mon01">>},
{partitions,[]}]
...done.
//*注：partitions里如果有节点一般非正常，可以通过重启rabbitmq-server服务来恢复*//
4.3 编辑rabbitmq变量文件
[root@cont01:/root]# vim /etc/rabbitmq/rabbitmq-env.conf
##
RABBITMQ_NODE_PORT=5672
ulimit -S -n 4096
RABBITMQ_SERVER_ERL_ARGS="+K true +A30 +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}]"
RABBITMQ_NODE_IP_ADDRESS=192.168.10.21
[root@cont02:/root]# vim /etc/rabbitmq/rabbitmq-env.conf
##
RABBITMQ_NODE_PORT=5672
ulimit -S -n 4096
RABBITMQ_SERVER_ERL_ARGS="+K true +A30 +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}]"
RABBITMQ_NODE_IP_ADDRESS=192.168.10.22
[root@cont03:/root]# vim /etc/rabbitmq/rabbitmq-env.conf
##
RABBITMQ_NODE_PORT=5672
ulimit -S -n 4096
RABBITMQ_SERVER_ERL_ARGS="+K true +A30 +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}]"
RABBITMQ_NODE_IP_ADDRESS=192.168.10.23
[root@mon01:/root]# vim /etc/rabbitmq/rabbitmq-env.conf
##
RABBITMQ_NODE_PORT=5672
ulimit -S -n 4096
RABBITMQ_SERVER_ERL_ARGS="+K true +A30 +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}]"
RABBITMQ_NODE_IP_ADDRESS=192.168.10.24
4.4 查看rabbitmq插件
[root@cont01:/root]# /usr/lib/rabbitmq/bin/rabbitmq-plugins list
[ ] amqp_client 3.3.5
[ ] cowboy 0.5.0-rmq3.3.5-git4b93c2d
[ ] eldap 3.3.5-gite309de4
[ ] mochiweb 2.7.0-rmq3.3.5-git680dba8
[ ] rabbitmq_amqp1_0 3.3.5
[ ] rabbitmq_auth_backend_ldap 3.3.5
[ ] rabbitmq_auth_mechanism_ssl 3.3.5
[ ] rabbitmq_consistent_hash_exchange 3.3.5
[ ] rabbitmq_federation 3.3.5
[ ] rabbitmq_federation_management 3.3.5
[ ] rabbitmq_management 3.3.5
[ ] rabbitmq_management_agent 3.3.5
[ ] rabbitmq_management_visualiser 3.3.5
[ ] rabbitmq_mqtt 3.3.5
[ ] rabbitmq_shovel 3.3.5
[ ] rabbitmq_shovel_management 3.3.5
[ ] rabbitmq_stomp 3.3.5
[ ] rabbitmq_test 3.3.5
[ ] rabbitmq_tracing 3.3.5
[ ] rabbitmq_web_dispatch 3.3.5
[ ] rabbitmq_web_stomp 3.3.5
[ ] rabbitmq_web_stomp_examples 3.3.5
[ ] sockjs 0.3.4-rmq3.3.5-git3132eb9
[ ] webmachine 1.10.3-rmq3.3.5-gite9359c7
4.5 所有node开启rabbitmq的web管理页面
rabbitmq-plugins enable rabbitmq_management
systemctl restart rabbitmq-server
systemctl status rabbitmq-server
4.6 查看rabbitmq插件是否加载成功（成功后前面有E）
[root@cont01:/root]# /usr/lib/rabbitmq/bin/rabbitmq-plugins list
[e] amqp_client 3.3.5
[ ] cowboy 0.5.0-rmq3.3.5-git4b93c2d
[ ] eldap 3.3.5-gite309de4
[e] mochiweb 2.7.0-rmq3.3.5-git680dba8
[ ] rabbitmq_amqp1_0 3.3.5
[ ] rabbitmq_auth_backend_ldap 3.3.5
[ ] rabbitmq_auth_mechanism_ssl 3.3.5
[ ] rabbitmq_consistent_hash_exchange 3.3.5
[ ] rabbitmq_federation 3.3.5
[ ] rabbitmq_federation_management 3.3.5
[E] rabbitmq_management 3.3.5
[e] rabbitmq_management_agent 3.3.5
[ ] rabbitmq_management_visualiser 3.3.5
[ ] rabbitmq_mqtt 3.3.5
[ ] rabbitmq_shovel 3.3.5
[ ] rabbitmq_shovel_management 3.3.5
[ ] rabbitmq_stomp 3.3.5
[ ] rabbitmq_test 3.3.5
[ ] rabbitmq_tracing 3.3.5
[e] rabbitmq_web_dispatch 3.3.5
[ ] rabbitmq_web_stomp 3.3.5
[ ] rabbitmq_web_stomp_examples 3.3.5
[ ] sockjs 0.3.4-rmq3.3.5-git3132eb9
[e] webmachine 1.10.3-rmq3.3.5-gite9359c7
提示：rabbitmq: web默认端口15672 rabbitmq: server默认端口5672
4.7 访问：http://192.168.10.22:15672/
注：如果是阿里云服务器，需要打开15672端口.(后期可能有80端口被占用情况)
默认账户：guest/guest
4.8 所有node修改guest密码为adminrabbitmq(默认用户为：guest 密码为：guest)
[root@cont01:/root]# rabbitmqctl change_password guest adminrabbitmq
Changing password for user "guest" ...
...done.
[1]+ Done rabbitmqctl change_password guest admin
[root@cont02:/root]# rabbitmqctl change_password guest adminrabbitmq
Changing password for user "guest" ...
...done.
[1]+ Done rabbitmqctl change_password guest admin
[root@cont03:/root]# rabbitmqctl change_password guest adminrabbitmq
Changing password for user "guest" ...
...done.
[root@mon01:/root]# rabbitmqctl change_password guest adminrabbitmq
Changing password for user "guest" ...
...done.
[1]+ Done rabbitmqctl change_password guest admin
4.9 cont02添加一个openstack的用户，并设密码为adminopenstack。并设置权限成为管理员(成集群后，cont02和cont03上也可以登录)
[root@cont02:/root]# rabbitmqctl add_user openstack adminopenstack
Creating user "openstack" ...
...done.
[root@cont02:/root]# rabbitmqctl set_permissions openstack ".*" ".*" ".*"
Setting permissions for user "openstack" in vhost "/" ...
...done.
[root@cont02:/root]# rabbitmqctl set_user_tags openstack administrator
Setting tags for user "openstack" to [administrator] ...
...done.
[root@cont02:/root]# rabbitmqctl list_users
Listing users ...
guest [administrator]
openstack [administrator]
...done.
4.10 发送cont01上的erlang.cookie到其他节点配置集群
[root@cont02:/root]# rabbitmqctl status
Status of node rabbit@cont02 ...
[{pid,12512},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},
{webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},
{rabbit,"RabbitMQ","3.3.5"},
{os_mon,"CPO CXC 138 46","2.2.14"},
{inets,"INETS CXC 138 49","5.9.8"},
{mnesia,"MNESIA CXC 138 12","4.11"},
{amqp_client,"RabbitMQ AMQP Client","3.3.5"},
{xmerl,"XML parser","1.3.6"},
{sasl,"SASL CXC 138 11","2.3.4"},
{stdlib,"ERTS CXC 138 10","1.19.4"},
{kernel,"ERTS CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,41896520},
{connection_procs,5600},
{queue_procs,5600},
{plugins,385080},
{other_proc,13866040},
{mnesia,60888},
{mgmt_db,50256},
{msg_index,34528},
{other_ets,1114200},
{binary,74848},
{code,20229052},
{atom,711569},
{other_system,5358859}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"192.168.10.22"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,8331886592},
{disk_free_limit,50000000},
{disk_free,178624393216},
{file_descriptors,
[{total_limit,3996},
{total_used,3},
{sockets_limit,3594},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,190}]},
{run_queue,0},
{uptime,369}]
...done.
[root@cont02:/root]# scp /var/lib/rabbitmq/.erlang.cookie cont01:/var/lib/rabbitmq/
[root@cont02:/root]# scp /var/lib/rabbitmq/.erlang.cookie cont03:/var/lib/rabbitmq/
[root@cont02:/root]# scp /var/lib/rabbitmq/.erlang.cookie mon01:/var/lib/rabbitmq/
4.11 cont01、cont03、mon01停止应用，并ram的方式加入cont01节点，之后重启应用（先查看cont01集群名字）
[root@cont02:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont02 ...
[{nodes,[{disc,[rabbit@cont02]}]},
{running_nodes,[rabbit@cont02]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
[root@cont01:/root]# systemctl restart rabbitmq-server
[root@cont01:/root]# systemctl status rabbitmq-server
[root@cont01:/root]# rabbitmqctl stop_app
Stopping node rabbit@cont01 ...
...done.
[root@cont01:/root]# rabbitmqctl join_cluster --ram rabbit@cont02
Clustering node rabbit@cont01 with rabbit@cont02 ...
...done.
[root@cont01:/root]# rabbitmqctl start_app
Starting node rabbit@cont01 ...
...done.
[root@cont03:/root]# systemctl restart rabbitmq-server
[root@cont03:/root]# systemctl status rabbitmq-server
[root@cont03:/root]# rabbitmqctl stop_app
[root@cont03:/root]# rabbitmqctl join_cluster --ram rabbit@cont02
Clustering node rabbit@cont03 with rabbit@cont02 ...
...done.
[root@cont03:/root]# rabbitmqctl start_app
Starting node rabbit@cont03 ...
...done.
[root@mon01:/root]# systemctl restart rabbitmq-server
[root@mon01:/root]# systemctl status rabbitmq-server
[root@mon01:/root]# rabbitmqctl stop_app
Stopping node rabbit@mon01 ...
...done.
[root@mon01:/root]# rabbitmqctl join_cluster --ram rabbit@cont02
Clustering node rabbit@mon01 with rabbit@cont02 ...
...done.
[root@mon01:/root]# rabbitmqctl start_app
Starting node rabbit@mon01 ...
...done.
4.12 检测集群状态
[root@cont01:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont01 ...
[{nodes,[{disc,[rabbit@cont02]},
{ram,[rabbit@mon01,rabbit@cont03,rabbit@cont01]}]},
{running_nodes,[rabbit@mon01,rabbit@cont03,rabbit@cont02,rabbit@cont01]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
[root@cont02:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont02 ...
[{nodes,[{disc,[rabbit@cont02]},
{ram,[rabbit@mon01,rabbit@cont03,rabbit@cont01]}]},
{running_nodes,[rabbit@mon01,rabbit@cont03,rabbit@cont01,rabbit@cont02]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
[root@cont03:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@cont03 ...
[{nodes,[{disc,[rabbit@cont02]},
{ram,[rabbit@mon01,rabbit@cont03,rabbit@cont01,]}]},
{running_nodes,[rabbit@mon01,rabbit@cont01,rabbit@cont02,rabbit@cont03]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
[root@mon01:/root]# rabbitmqctl cluster_status
Cluster status of node rabbit@mon01 ...
[{nodes,[{disc,[rabbit@cont02]},
{ram,[rabbit@mon01,rabbit@cont03,rabbit@cont01]}]},
{running_nodes,[rabbit@cont01,rabbit@cont02,rabbit@cont03,rabbit@mon01]},
{cluster_name,<<"rabbit@cont02">>},
{partitions,[]}]
...done.
4.13 为了保持rabbitmq的消息一致性，可以设置消息镜像模式策略 所有节点执行
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
优化RabbitMQQ性能
在RabbitMQQ中，socket descriptors 和 File descriptors的默认配额并不大
修改配置
在其配置文件中添加如下参数，其值请根据实际情况进行调整：
vim /usr/lib/systemd/system/rabbitmq-erver.service
[Service]
LimitNOFILE = 16384
重启rabbitmq
systemctl daemon-reload
systemctl restart rabbitmq-server
//下面为引用参考文案，如下：(节后调整)
在为OpenStack的服务配置使用rabbitmq消息队列服务时，可以如下配置：
transport_url = rabbit://openstack:adminopenstack@cont01,openstack:adminopenstack@cont02,openstack:adminopenstack@cont03
可以看到这里的配置方式是将所有节点的rabbitmq服务以拼接方式拼在一起，当cont1节点上的消息队列服务不可用时可以将请求转发给cont2，再不行可以再转发给cont3节点.
同时应该配置如下参数：
rabbit_retry_interval=1
rabbit_retry_backoff=2
rabbit_max_retries=0
rabbit_durable_queues=true
rabbit_ha_queues=true
4.14 RabbitMQ注意小计（故障恢复提示）：
①保证集群中至少有一个磁盘类型的节点以防数据丢失，在更改节点类型时尤其要注意。
②若整个集群被停掉了，应保证最后一个down掉的节点被最先启动，若不能则要使用forget_cluster_node命令将其移出集群。
③若集群中节点几乎同时以不可控的方式down 了，此时再其中一个节点使用force_boot 命令重启节点。
④如果加入集群后，意外关闭等造成rabbitmq-server启动不成功，可以尝试一下步骤：/var/lib/rabbitmq/mnesia 目录下存在[email protected]、rabbit@localhost、rabbit@localhost-plugins-expand，删除这3项后，并且删除 /var/lib/rabbitmq/ 目录下 .erlang.cookie和erl_crash.dump 再使用systemctl start rabbitmq-server启动
5、部署PCS
5.1 安装pcs+pacemaker+corosync （cont01、cont02和 cont03）
所有控制节点安装pcs、pacemaker、corosync, pacemaker是资源管理器，corosync提供心跳机制。
[root@cont01:/root]# yum install -y lvm2 cifs-utils quota psmisc pcs pacemaker corosync fence-agents-all resource-agents crmsh
[root@cont02:/root]# yum install -y lvm2 cifs-utils quota psmisc pcs pacemaker corosync fence-agents-all resource-agents crmsh
[root@cont03:/root]# yum install -y lvm2 cifs-utils quota psmisc pcs pacemaker corosync fence-agents-all resource-agents crmsh
[root@cont01:/root]# systemctl enable pcsd corosync
[root@cont02:/root]# systemctl enable pcsd corosync
[root@cont03:/root]# systemctl enable pcsd corosync
[root@cont01:/root]# systemctl start pcsd && systemctl status pcsd
[root@cont02:/root]# systemctl start pcsd && systemctl status pcsd
[root@cont03:/root]# systemctl start pcsd && systemctl status pcsd
5.2 设置集群密码，而且三个节点密码需一直为：pcs#typora
[root@cont01:/root]# echo "pcs#typora" |passwd --stdin hacluster
[root@cont02:/root]# echo "pcs#typora" |passwd --stdin hacluster
[root@cont03:/root]# echo "pcs#typora" |passwd --stdin hacluster
5.3 控制节点创建配置文件corosync.conf
[root@cont02:/root]# cat </etc/corosync/corosync.conf
totem {
version: 2
secauth:off
cluster_name:openstack-cluster
transport:udpu
}
nodelist {
node {
ring0_addr:cont01
nodeid:1
}
node {
ring0_addr:cont02
nodeid:2
}
node {
ring0_addr:cont03
nodeid:3
}
}
logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
debug: off
}
quorum {
provider: corosync_votequorum
}
EOF
[root@cont02:/root]# scp /etc/corosync/corosync.conf cont01:/etc/corosync/
[root@cont02:/root]# scp /etc/corosync/corosync.conf cont03:/etc/corosync/
5.4 配置集群，设置集群互相认证
ssh-keygen
ssh-copy-id cont01
ssh-copy-id cont02
ssh-copy-id cont03
5.5 配置节点认证
[root@cont02:/root]# pcs cluster auth cont01 cont02 cont03 -u hacluster -p"pcs#typora"
cont02: Authorized
cont03: Authorized
cont01: Authorized
pcs cluster auth cont01 cont02 -u hacluster -p {password}
{password}表示为刚才设置的密码
5.6 创建集群
[root@cont02:/root]# pcs cluster setup --force --name openstack-cluster cont01 cont02 cont03
Destroying cluster on nodes: cont01, cont02, cont03...
cont02: Stopping Cluster (pacemaker)...
cont03: Stopping Cluster (pacemaker)...
cont01: Stopping Cluster (pacemaker)...
cont01: Successfully destroyed cluster
cont02: Successfully destroyed cluster
cont03: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'cont01', 'cont02', 'cont03'
cont01: successful distribution of the file 'pacemaker_remote authkey'
cont03: successful distribution of the file 'pacemaker_remote authkey'
cont02: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
cont01: Succeeded
cont02: Succeeded
cont03: Succeeded
Synchronizing pcsd certificates on nodes cont01, cont02, cont03...
cont02: Success
cont03: Success
cont01: Success
Restarting pcsd on the nodes in order to reload the certificates...
cont02: Success
cont03: Success
cont01: Success
5.7 启动集群并查看集群状态
[root@cont02:/root]# pcs cluster enable --all
cont01: Cluster Enabled
cont02: Cluster Enabled
cont03: Cluster Enabled
[root@cont02:/root]# pcs cluster start --all
cont01: Starting Cluster (corosync)...
cont02: Starting Cluster (corosync)...
cont03: Starting Cluster (corosync)...
cont01: Starting Cluster (pacemaker)...
cont03: Starting Cluster (pacemaker)...
cont02: Starting Cluster (pacemaker)...
[root@cont02:/root]# pcs cluster status
Cluster Status:
Stack: corosync
Current DC: cont03 (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum
Last updated: Wed Jan 15 21:44:10 2020
Last change: Wed Jan 15 21:43:53 2020 by hacluster via crmd on cont03
3 nodes configured
0 resources configured
PCSD Status:
cont02: Online
cont03: Online
cont01: Online
[root@cont02:/root]# ps aux | grep pacemaker
root 15586 0.0 0.0 132972 8700 ? Ss 21:43 0:00 /usr/sbin/pacemakerd -f
haclust+ 15587 0.1 0.0 136244 14620 ? Ss 21:43 0:00 /usr/libexec/pacemaker/cib
root 15588 0.0 0.0 136064 7664 ? Ss 21:43 0:00 /usr/libexec/pacemaker/stonithd
root 15589 0.0 0.0 98836 4372 ? Ss 21:43 0:00 /usr/libexec/pacemaker/lrmd
haclust+ 15590 0.0 0.0 128068 6620 ? Ss 21:43 0:00 /usr/libexec/pacemaker/attrd
haclust+ 15591 0.0 0.0 80508 3500 ? Ss 21:43 0:00 /usr/libexec/pacemaker/pengine
haclust+ 15592 0.0 0.0 140380 8260 ? Ss 21:43 0:00 /usr/libexec/pacemaker/crmd
root 15632 0.0 0.0 112712 960 pts/0 S+ 21:44 0:00 grep --color=auto pacemaker
5.8 检验corosync状态
（执行crm_verify -L -V时会报错）
[root@cont02:/root]# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
id = 192.168.10.22
status = ring 0 active with no faults
[root@cont02:/root]# corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.10.21)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.10.22)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(192.168.10.23)
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined
[root@cont02:/root]# pcs status corosync
Membership information
----------------------
Nodeid Votes Name
1 1 cont01
2 1 cont02 (local)
3 1 cont03
[root@cont02:/root]# crm_verify -L -V
error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
[root@cont02:/root]# pcs property set stonith-enabled=false
[root@cont02:/root]# pcs property set no-quorum-policy=ignore
[root@cont02:/root]# crm_verify -L -V
[root@cont02:/root]# pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000
[root@cont02:/root]# pcs property set cluster-recheck-interval=1000
检测配置（执行时会报错）
crm_verify -L -V
根据检测结果
pcs property set stonith-enabled=false #禁用STONITH
pcs property set no-quorum-policy=ignore #忽略仲裁
crm_verify -L -V #error 消失了
crm 交互式命令，比较好用
设置合适的输入处理历史记录及策略引擎生成的错误与警告
pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000
基于时间驱动的方式进行状态处理
pcs property set cluster-recheck-interval=1000
5.9 创建集群VIP （预设VIP地址为192.168.10.20）
[root@cont02:/root]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.10.20 cidr_netmask=32 op monitor interval=30s
6、部署Haproxy
6.1 所有控制节点安装Haproxy，并保持相同的配置，对于配置文件，建议根据实际情况进行优化。
6.1.1安装并配置haproxy
[root@cont01:/root]# yum -y install haproxy
[root@cont02:/root]# yum -y install haproxy
[root@cont03:/root]# yum -y install haproxy
[root@cont02:/root]# vim /etc/rsyslog.d/haproxy.conf
##配置HAProxy的日志
$ModLoad imudp
$UDPServerRun 514
$template Haproxy,"%msg%n"
local3.info -/var/log/haproxy.log;Haproxy
local3.notice -/var/log/haproxy-status.log;Haproxy
local3.*~
[root@cont02:/root]# scp /etc/rsyslog.d/haproxy.conf cont01:/etc/rsyslog.d/
[root@cont02:/root]# scp /etc/rsyslog.d/haproxy.conf cont03:/etc/rsyslog.d/
[root@cont01:/root]# systemctl enable haproxy.service
[root@cont01:/root]# systemctl restart rsyslog.service
[root@cont01:/root]# systemctl status rsyslog.service
[root@cont02:/root]# systemctl enable haproxy.service
[root@cont02:/root]# systemctl restart rsyslog.service
[root@cont02:/root]# systemctl status rsyslog.service
[root@cont03:/root]# systemctl enable haproxy.service
[root@cont03:/root]# systemctl restart rsyslog.service
[root@cont03:/root]# systemctl status rsyslog.service
[root@cont02:/root]# vim /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local3
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
spread-checks 3
tune.bufsize 32768
tune.maxrewrite 1024
tune.ssl.default-dh-param 2048
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option tcplog
option splice-auto
option http-server-close
# option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 20s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
# timeout http-keep-alive 10s
timeout check 10s
maxconn 8000
listen stats
bind 0.0.0.0:8789
mode http
stats enable
stats uri /
stats realm Haproxy\ Statistics
stats auth admin:admin
stats refresh 15s
stats show-node
stats show-legends
stats hide-version
listen dashboard_cluster
bind 192.168.10.20:8080
balance source
option tcpka
option httpchk
option tcplog
server cont01 192.168.10.21:80 check inter 2000 rise 2 fall 5
server cont02 192.168.10.22:80 check inter 2000 rise 2 fall 5
server cont03 192.168.10.23:80 check inter 2000 rise 2 fall 5
listen galera_cluster
bind 192.168.10.20:3307
balance source
hash-type consistent
mode tcp
option tcplog
option clitcpka
option httpchk
timeout client 28801s
timeout server 28801s
server cont01 192.168.10.21:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3 backup
server cont03 192.168.10.23:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3 backup
listen mq_cluster
bind 192.168.10.20:5672 tcp-ut 5s
mode tcp
option tcpka
balance roundrobin
server cont01 192.168.10.21:5672 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:5672 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:5672 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen keystone_admin_cluster
bind 192.168.10.20:5001
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httplog
option httpclose
option forwardfor
server cont01 192.168.10.21:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen keystone_public_internal_cluster
bind 192.168.10.20:5001
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httplog
option httpclose
option forwardfor
server cont01 192.168.10.21:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:5000 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen glance_registry_cluster
bind 192.168.10.20:9192
timeout server 30m
server cont01 192.168.10.21:9191 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:9191 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:9191 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen glance_api_cluster
bind 192.168.10.20:9293
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk /version
option httplog
option httpclose
timeout server 30m
server cont01 192.168.10.21:9292 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:9292 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:9292 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen nova_ec2_api_cluster
bind 192.168.10.20:9773
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
option httplog
option httpclose
timeout server 600s
server cont01 192.168.10.21:8773 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8773 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8773 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen nova_compute_api_cluster
bind 192.168.10.20:9774
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
option httplog
option httpclose
timeout server 600s
server cont01 192.168.10.21:8774 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8774 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8774 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen nova_metadate_api_cluster
bind 192.168.10.20:9775
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
option httplog
option httpclose
timeout server 600s
server cont01 192.168.10.21:8775 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8775 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8775 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen nova_vncproxy_cluster
bind 192.168.10.20:6081
http-request set-header X-Forwarded-Proto https if { ssl_fc }
server cont01 192.168.10.21:6080 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:6080 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:6080 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen nova_placement_cluster
bind 192.168.10.20:9778
http-request set-header X-Forwarded-Proto https if { ssl_fc }
server cont01 192.168.10.21:8778 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8778 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8778 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen neutron_api_cluster
bind 192.168.10.20:9997
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
option httplog
option httpclose
server cont01 192.168.10.21:9696 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:9696 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:9696 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen cinder_api_cluster
bind 192.168.10.20:9776
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
option httplog
option httpclose
server cont01 192.168.10.21:8776 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8776 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8776 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
listen ceilometer_api_cluster
bind 192.168.10.20:9777
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option httplog
option httpclose
server cont01 192.168.10.21:8777 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont02 192.168.10.22:8777 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
server cont03 192.168.10.23:8777 check inter 10s fastinter 2s downinter 2s rise 3 fall 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
#frontend main *:5000
# acl url_static path_beg -i /static /images /javascript /stylesheets
# acl url_static path_end -i .jpg .gif .png .css .js
# use_backend static if url_static
# default_backend app
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
# balance roundrobin
# server static 127.0.0.1:4331 check
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
#backend app
# balance roundrobin
# server app1 127.0.0.1:5001 check
# server app2 127.0.0.1:5002 check
# server app3 127.0.0.1:5003 check
# server app4 127.0.0.1:5004 check
[root@cont02:/root]# scp /etc/haproxy/haproxy.cfg cont01:/etc/haproxy/
[root@cont02:/root]# scp /etc/haproxy/haproxy.cfg cont03:/etc/haproxy/
6.2 配置Haproxy能监控Galera数据库集群
在控制节点三台MariaDB上执行下列操作
[root@cont0$:/root]# mysql
MariaDB [(none)]> use mysql;
MariaDB [mysql]> grant process on *.* to 'clustercheckuser'@'localhost' identified by 'clustercheckpassword!';
MariaDB [mysql]> grant process on *.* to 'clustercheckuser'@'%' identified by 'clustercheckpassword!';
MariaDB [mysql]> flush privileges;
MariaDB [mysql]> exit
[root@cont0$:/root]# cat < /etc/sysconfig/clustercheck
MYSQL_USERNAME="clustercheckuser"
MYSQL_PASSWORD="clustercheckpassword!"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"
EOF
[root@cont0$:/root]# vim /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1
fs.file-max = 655350
net.ipv4.ip_local_port_range = 1025 65000
[root@cont0$:/root]# sysctl -p
//注：重启systemctl restart haproxy.service 后，192.168.10.20:3307端口才可以登陆
[root@cont03:/root]# mysql -h192.168.10.20 -P3307 -uroot -p"typora#2019"
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 54
Server version: 10.4.11-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant process on *.* to 'clustercheckuser'@'localhost' identified by 'clustercheckpassword!';
Query OK, 0 rows affected (0.108 sec)
MariaDB [(none)]> grant process on *.* to 'clustercheckuser'@'%' identified by 'clustercheckpassword!';
Query OK, 0 rows affected (0.086 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.091 sec)
MariaDB [(none)]> exit
Bye
6.3 开启haproxy,并自启动（cont01、cont02、cont03）
systemctl restart haproxy.service
systemctl enable haproxy
systemctl status haproxy
[root@cont01:/root]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 40
Percona XtraDB Cluster Node is synced.
[root@cont02:/root]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 40
Percona XtraDB Cluster Node is synced.
[root@cont03:/root]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 40
Percona XtraDB Cluster Node is synced.
注：三台机子中，1台状态为active,另外两台为backup
6.4 登陆http://192.168.10.20:8789/ 检测状态 默认账户:admin 密码:admin
7、部署Keystone
7.1 控制节点安装Enable the OpenStack repository(cont01和cont02)
[root@cont01:/root]# yum install centos-release-openstack-rocky -y
[root@cont01:/root]# yum install python-openstackclient -y
[root@cont01:/root]# yum install openstack-selinux openstack-utils -y
[root@cont02:/root]# yum install centos-release-openstack-rocky -y
[root@cont02:/root]# yum install python-openstackclient -y
[root@cont02:/root]# yum install openstack-selinux openstack-utils -y
[root@cont03:/root]# yum install centos-release-openstack-rocky -y
[root@cont03:/root]# yum install python-openstackclient -y
[root@cont03:/root]# yum install openstack-selinux openstack-utils -y
注：安装openstack命令工具 openstack-utils，后期则可以使用openstack-config 命令配置openstack
7.2 控制节点安装 SQL database(cont01和cont02)
[root@cont01:/root]# yum install mariadb mariadb-server python2-PyMySQL -y
[root@cont01:/root]# vim /etc/my.cnf.d/server.cnf
##
[mysqld]
bind-address = 192.168.10.21
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
[root@cont02:/root]# vim /etc/my.cnf.d/server.cnf
# this is only for the mysqld standalone daemon
[mysqld]
bind-address = 192.168.10.22
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
[root@cont03:/root]# vim /etc/my.cnf.d/server.cnf
##
[mysqld]
bind-address = 192.168.10.23
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
[root@cont01:/root]# systemctl restart mariadb.service
[root@cont01:/root]# systemctl status mariadb.service
[root@cont02:/root]# systemctl restart mariadb.service
[root@cont02:/root]# systemctl status mariadb.service
[root@cont03:/root]# systemctl restart mariadb.service
[root@cont03:/root]# systemctl status mariadb.service
//show status like 'wsrep_cluster_size';
7.3 安装Memcached (cont01、cont02和cont03)
[root@cont01:/root]# yum install memcached python-memcached -y
[root@cont01:/root]# vim /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1,::1,cont01"
[root@cont01:/root]# systemctl enable memcached.service && systemctl start memcached.service
[root@cont01:/root]# systemctl status memcached.service
[root@cont02:/root]# yum install memcached python-memcached -y
[root@cont02:/root]# vim /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1,::1,cont02"
[root@cont02:/root]# systemctl enable memcached.service && systemctl start memcached.service
[root@cont02:/root]# systemctl status memcached.service
[root@cont03:/root]# yum install memcached python-memcached -y
[root@cont03:/root]# vim /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1,::1,cont03"
[root@cont03:/root]# systemctl enable memcached.service && systemctl start memcached.service
[root@cont03:/root]# systemctl status memcached.service
7.4 configure the Identity service
[root@cont01:/root]# mysql -uroot -p"typora#2019"
MariaDB [(none)]> CREATE DATABASE keystone;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| keystone |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'KEYSTONE_typora';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'KEYSTONE_typora';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> exit
Bye
由于做了MariaDB集群，所有cont02和cont03上自动同步了数据库。同步数据库和创建用户只需要在一个节点上操作就可以。
[root@cont02:/root]# mysql -uroot -p"typora#2019"
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| keystone |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.03 sec)
MariaDB [(none)]> exit
Bye
[root@cont03:/root]# mysql -uroot -p"typora#2019"
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| keystone |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.03 sec)
MariaDB [(none)]> exit
Bye
7.5 Install and configure components
[root@cont01:/root]# yum install openstack-keystone httpd mod_wsgi -y
[root@cont02:/root]# yum install openstack-keystone httpd mod_wsgi -y
[root@cont03:/root]# yum install openstack-keystone httpd mod_wsgi -y
[root@cont01:/root]# cd /etc/keystone/
[root@cont01:/etc/keystone]# ls
default_catalog.templates keystone.conf keystone-paste.ini logging.conf policy.json sso_callback_template.html
[root@cont01:/etc/keystone]# cp -p keystone.conf{,.bak}
[root@cont01:/etc/keystone]# vim /etc/keystone/keystone.conf
551 [cache]
552 backend = oslo_cache.memcache_pool
553 enabled = true
554 memcache_servers = cont01:11211,cont02:11211,cont03:11211
723 [database]
724 connection = mysql+pymysql://keystone:KEYSTONE_typora@VirtualIP:3307/keystone
2805 [token]
2828 provider = fernet
[root@cont02:/root]# cd /etc/keystone/
[root@cont02:/etc/keystone]# cp -p keystone.conf{,.bak}
[root@cont02:/etc/keystone]# vim /etc/keystone/keystone.conf
551 [cache]
552 backend = oslo_cache.memcache_pool
553 enabled = true
554 memcache_servers = cont01:11211,cont02:11211,cont03:11211
723 [database]
724 connection = mysql+pymysql://keystone:KEYSTONE_typora@VirtualIP:3307/keystone
2805 [token]
2828 provider = fernet
[root@cont03:/root]# cd /etc/keystone/
[root@cont03:/etc/keystone]# cp -p keystone.conf{,.bak}
[root@cont03:/etc/keystone]# vim /etc/keystone/keystone.conf
551 [cache]
552 backend = oslo_cache.memcache_pool
553 enabled = true
554 memcache_servers = cont01:11211,cont02:11211,cont03:11211
723 [database]
724 connection = mysql+pymysql://keystone:KEYSTONE_typora@VirtualIP:3307/keystone
2805 [token]
2828 provider = fernet
//注：查看keystone配置文件
[root@cont01:/root]# egrep -v "^#|^$" /etc/keystone/keystone.conf
[root@cont02:/etc/keystone]# su -s /bin/sh -c "keystone-manage db_sync" keystone
[root@cont01:/etc/keystone]# su -s /bin/sh -c "keystone-manage db_sync" keystone
[root@cont03:/etc/keystone]# su -s /bin/sh -c "keystone-manage db_sync" keystone
注意：如果出现同步失败，请执行后面步骤（参考pip）
[root@cont02:/root]# mysql -uroot -p"typora#2019"
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| keystone |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> use keystone;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [keystone]> show tables;
+-----------------------------+
| Tables_in_keystone |
+-----------------------------+
| access_token |
| application_credential |
| application_credential_role |
| assignment |
| config_register |
| consumer |
| credential |
| endpoint |
| endpoint_group |
| federated_user |
| federation_protocol |
| group |
| id_mapping |
| identity_provider |
| idp_remote_ids |
| implied_role |
| limit |
| local_user |
| mapping |
| migrate_version |
| nonlocal_user |
| password |
| policy |
| policy_association |
| project |
| project_endpoint |
| project_endpoint_group |
| project_tag |
| region |
| registered_limit |
| request_token |
| revocation_event |
| role |
| sensitive_config |
| service |
| service_provider |
| system_assignment |
| token |
| trust |
| trust_role |
| user |
| user_group_membership |
| user_option |
| whitelisted_config |
+-----------------------------+
44 rows in set (0.00 sec)
MariaDB [keystone]> exit
Bye
//或者使用下列方法数据库进行数据库连接测试
[root@cont02:/root]# mysql -ukeystone -p"KEYSTONE_typora" -P3307 -hVirtualIP -e "use keystone;show tables;"
//初始化fernet令牌库
[root@cont02:/etc/keystone]# keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
[root@cont02:/etc/keystone]# keystone-manage credential_setup --keystone-user keystone --keystone-group keystone
//创建keystone用户，初始化serveice实体和endpoint api端点
注：创建keystone服务实体和身份认证服务，以下三种类型分别为公共的、内部的、管理的,<1>在endpoint表增加3个服务实体的API端点,<2>在local_user表中创建admin用户,<3>在project表中创建admin和Default项目（默认域）<4>在role表创建3种角色，admin，member和reader <5>在service表中创建identity服务
[root@cont01:/root]# keystone-manage bootstrap --bootstrap-password admin \
> --bootstrap-admin-url http://VirtualIP:5001/v3/ \
> --bootstrap-internal-url http://VirtualIP:5001/v3/ \
> --bootstrap-public-url http://VirtualIP:5001/v3/ \
> --bootstrap-region-id RegionOne
[root@cont02:/root]# keystone-manage bootstrap --bootstrap-password admin \
> --bootstrap-admin-url http://VirtualIP:5001/v3/ \
> --bootstrap-internal-url http://VirtualIP:5001/v3/ \
> --bootstrap-public-url http://VirtualIP:5001/v3/ \
> --bootstrap-region-id RegionOne
[root@cont03:/root]# keystone-manage bootstrap --bootstrap-password admin \
> --bootstrap-admin-url http://VirtualIP:5001/v3/ \
> --bootstrap-internal-url http://VirtualIP:5001/v3/ \
> --bootstrap-public-url http://VirtualIP:5001/v3/ \
> --bootstrap-region-id RegionOne
[root@cont02:/etc/keystone]# scp -r credential-keys/ fernet-keys/ cont01:/etc/keystone/
[root@cont02:/etc/keystone]# scp -r credential-keys/ fernet-keys/ cont03:/etc/keystone/
[root@cont01:/etc/keystone]#chown keystone:keystone /etc/keystone/credential-keys/ -R
[root@cont01:/etc/keystone]#chown keystone:keystone /etc/keystone/fernet-keys/ -R
[root@cont03:/etc/keystone]#chown keystone:keystone /etc/keystone/credential-keys/ -R
[root@cont03:/etc/keystone]#chown keystone:keystone /etc/keystone/fernet-keys/ -R
注：假如出现同步失败的话，需要执行如下操作（参考pip）
##下载pip
wget https://bootstrap.pypa.io/get-pip.py
##安装pip
python get-pip.py
##查看版本
pip -V
##查看安装路径
find / -name pip
##软链接
####ln -s -f /usr/lib/python2.7/site-packages/pip /usr/bin/pip
pip uninstall urllib3 -y
pip uninstall chardet -y
pip install requests
####hash -r
python get-pip.py
[root@cont03:/root]# pip install requests
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: requests in /usr/lib/python2.7/site-packages (2.19.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<2.8,>=2.5 in /usr/lib/python2.7/site-packages (from requests) (2.5)
Collecting urllib3<1.24,>=1.21.1
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
|████████████████████████████████| 143kB 85kB/s
Installing collected packages: urllib3
Found existing installation: urllib3 1.10.2
Uninstalling urllib3-1.10.2:
Successfully uninstalled urllib3-1.10.2
Successfully installed urllib3-1.23
7.6 Configure the Apache HTTP server [cont0$表示cont01、cont02和cont03]
[root@cont0$:/root]# vim /etc/httpd/conf/httpd.conf
ServerName VirtualIP
//上一步骤为在httpd.conf文件中添加ServerName VirtualIP
[root@cont0$:/root]# ln -s /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/
[root@cont0$:/root]# ll /etc/httpd/conf.d/
total 16
-rw-r--r--. 1 root root 2926 Aug 8 19:41 autoindex.conf
-rw-r--r--. 1 root root 366 Aug 8 19:42 README
-rw-r--r--. 1 root root 1252 Aug 6 21:44 userdir.conf
-rw-r--r--. 1 root root 824 Aug 6 21:44 welcome.conf
lrwxrwxrwx. 1 root root 38 Jan 16 23:00 wsgi-keystone.conf -> /usr/share/keystone/wsgi-keystone.conf
[root@cont0$:/root]# systemctl enable httpd.service && systemctl start httpd.service
[root@cont0$:/root]# systemctl status httpd.service
[root@cont0$:/root]# vim openrc
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_AUTH_URL=http://VirtualIP:5001/v3
export OS_IDENTITY_API_VERSION=3
[root@cont0$:/root]# source openrc
[root@cont0$:/root]# openstack user list
+----------------------------------+-------+
| ID | Name |
+----------------------------------+-------+
| 02c1960ba4c44f46b7152c0a7e52fdba | admin |
+----------------------------------+-------+
[root@cont0$:/root]# vim admin-openrc
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://VirtualIP:5001/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2
[root@cont03:/root]# vim demo-openrc
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=myproject
export OS_USERNAME=myuser
export OS_PASSWORD=myuser
export OS_AUTH_URL=http://VirtualIP:5001/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2
注：查看是否已经设置开启自启动
[root@cont01:/root]# systemctl list-unit-files | grep httpd
httpd.service enabled
7.7 Create a domain, projects, users, and roles
注：由于是集群，所以只要在一个控制节点上创建即可
[root@cont02:/root]# openstack domain create --description "An Example Domain" example
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | An Example Domain |
| enabled | True |
| id | f46ed6dd1c554eddb5175c265c6c98d4 |
| name | example |
| tags | [] |
+-------------+----------------------------------+
[root@cont02:/root]# openstack project create --domain default --description "Service Project" service
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Service Project |
| domain_id | default |
| enabled | True |
| id | b7877e02531e4eb7a368aa5c44e295e9 |
| is_domain | False |
| name | service |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+
[root@cont02:/root]# openstack project list
+----------------------------------+---------+
| ID | Name |
+----------------------------------+---------+
| b7877e02531e4eb7a368aa5c44e295e9 | service |
| eff859cedf434117b226d20f64d40e22 | admin |
+----------------------------------+---------+
[root@cont01:/root]# openstack project list
+----------------------------------+---------+
| ID | Name |
+----------------------------------+---------+
| b7877e02531e4eb7a368aa5c44e295e9 | service |
| eff859cedf434117b226d20f64d40e22 | admin |
+----------------------------------+---------+
[root@cont03:/root]# openstack project list
+----------------------------------+---------+
| ID | Name |
+----------------------------------+---------+
| b7877e02531e4eb7a368aa5c44e295e9 | service |
| eff859cedf434117b226d20f64d40e22 | admin |
+----------------------------------+---------+
[root@cont02:/root]# openstack project create --domain default --description "Demo Project" myproject
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Demo Project |
| domain_id | default |
| enabled | True |
| id | 9e73d22ab12f45cd99c72bc5c04ff961 |
| is_domain | False |
| name | myproject |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+
[root@cont02:/root]# openstack user create --domain default --password=typora# myuser
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 61c06b9891a64e68b87d84dbcec5e9ac |
| name | myuser |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
[root@cont02:/root]# openstack role create myrole
[root@cont02:/root]# openstack role add --project myproject --user myuser myrole
[root@cont02:/root]# unset OS_AUTH_URL OS_PASSWORD
[root@cont02:/root]# openstack --os-auth-url http://VirtualIP:5001/v3 \
> --os-project-domain-name Default --os-user-domain-name Default \
> --os-project-name admin --os-username admin token issue
Password:typora# //输入密码：typora#
The request you have made requires authentication. (HTTP 401) (Request-ID: req-6117e2db-4501-4202-b794-918f61d0643e)
[root@cont02:/root]# openstack --os-auth-url http://VirtualIP:5001/v3 \
> --os-project-domain-name Default --os-user-domain-name Default \
> --os-project-name admin --os-username admin token issue
+------------+------------------------------------------------------------------------
| Field | Value --------------------------------------------------------------------------------+
| expires | 2020-01-16T17:13:23+0000
| id | gAAAAABeIIujGF9vkLny2yr73nYj2WEpHzS8w6nxf2eq-8VG9J3g1so_k7NesxKQjrgeoLLPcZS9UeA7ETwjGboGTRAoeXZRCDffQQFOA_ywTIcMHSRTKRHEUsf949SPxSWk6eOHjcncpFrW5gt0GF-dKBbOgqiaW3mdxEnO2mitZwjAiu-v07A |
| project_id | eff859cedf434117b226d20f64d40e22
| user_id | 02c1960ba4c44f46b7152c0a7e52fdba ------------------------------------------------------------------------+
[root@cont02:/root]# . admin-openrc
[root@cont02:/root]# openstack token issue
+------------+------------------------------------------------------------------------
| Field | Value
+------------+------------------------------------------------------------------------
| expires | 2020-01-16T16:27:37+0000
| id | gAAAAABeIIDpJl7tk9hlti4O1pEst-wGQeeehdN4e2OzRPSy-YBb6dYcwZknlnLdj9Cjb5rzSYZ1gGLcvONHh4z6ot2UiXdPcuwlkJ-HOyzGMFyAHr3jyUSRr-5lB-o8E05PRvJq5u4NHSunG5mW-D3J-ila-dfUJ2IsOtu8MmE90pi0sEuK8eI |
| project_id | eff859cedf434117b226d20f64d40e22
| user_id | 02c1960ba4c44f46b7152c0a7e52fdba ---------------------------+
//测试是否可以使用admin账户进行登陆认证，请求认证令牌
[root@cont01:/root]# openstack --os-auth-url http://VirtualIP:5001/v3 --os-project-domain-name Default --os-user-domain-name Default --os-project-name admin --os-username admin token issue
待续，详见Centos 7 部署 OpenStack_Rocky版高可用集群3-2