Corosync与pacemaker安装:
1.环境说明
(1).操作系统
CentOS 6.5 X86_64位系统
(2).软件环境
**corosync-1.4.1-17.el6.x86_64
**pacemaker-1.1.10-14.el6.x86_64
**crmsh-1.2.6-4.el6.x86_64.rpm
**pssh-2.3.1-2.el6.x86_64.rpm
**drbd-8.4.3-33.el6.x86_64.rpm
**drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
(3).拓扑环境
节点数:2 分别为:node1 node2
node1:172.16.100.6 node2:172.16.100.7 9 TestHost:172.16.100.88
拓扑结构如下图所示:
实现配置过程如下:
1、准备工作
为了配置一台Linux主机成为HA的节点，通常需要做出如下的准备工作：
1)所有节点的主机名称和对应的IP地址解析服务可以正常工作，且每个节点的主机名称需要跟"uname -n“命令的结果保持一致；因此，需要保证两个节点上的/etc/hosts文件均为下面的内容：
# vim /etc/hosts 172.16.100.6 node1.samlee.com node1 172.16.100.7 node2.samlee.com node2
为了使得重新启动系统后仍能保持如上的主机名称，还分别需要在各节点执行类似如下的命令：
Node1配置:
# sed -i 's@\(HOSTNAME=\).*@\1node1.samlee.com@g' /etc/sysconfig/network # hostname node1.samlee.com
Node2配置：
# sed -i 's@\(HOSTNAME=\).*@\1node2.samlee.com@g' /etc/sysconfig/network # hostname node2.samlee.com
2)设定两个节点可以基于密钥进行ssh通信，这可以通过如下的命令实现：
Node1配置:
# ssh-keygen -t rsa -P '' # ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2 # ssh node2.samlee.com 'date';date
Node2配置:
# ssh-keygen -t rsa -P '' # ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1 # ssh node1.samlee.com 'date';date
3)设置5分钟自动同步时间(node1、node2都需要配置)
# crontab -e */5 * * * * /sbin/ntpdata 172.16.100.10 &> /dev/null
2、安装配置Corosync集群管理工具
1)安装Corosync工具及pacemaker(yum方式)
# yum -y install corosync # yum -y install pacemaker
安装crmsh(rpm方式)
RHEL自6.4起不再提供集群的命令行配置工具crmsh，转而使用pcs；如果你习惯了使用crm命令，可下载相关的程序包自行安装即可。crmsh依赖于pssh，因此需要一并下载。
# cd /root/corosync_packages/ # yum -y --nogpgcheck localinstall crmsh*.rpm pssh*.rpm
2)配置corosync(操作在node1.samlee.com上执行)
# cd /etc/corosync/
# cp corosync.conf.example corosync.conf
# vim corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: on --开启认证功能
threads: 0 --CPU个数
interface {
ringnumber: 0
bindnetaddr: 172.16.0.0 --集群节点运行所在的网络地址
mcastaddr: 226.96.6.17 --组播传输地址
mcastport: 5405 --心跳信息检测端口
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
##设置随corosync启动的服务
service {
ver: 0
name: pacemaker
}
##ais运行身份设定
aisexec {
user: root
group: root
}
并设定此配置文件中 bindnetaddr后面的IP地址为你的网卡所在网络的网络地址，我们这里的两个节点在172.16.0.0网络，因此这里将其设定为172.16.0.0；如下
bindnetaddr: 172.16.0.0
3)生成节点间通信时所用到的认证密钥文件:
# corosync-keygen 如果随机数不够的话需要需要登录状态狂敲键盘
如发现corosync生成/etc/corosync/autokey需要敲随机数，可以使用以下方式解决(仅测试环境参考使用):
# mv /dev/random /dev/h # ln /dev/urandom /dev/random # corosync-keygen # rm -rf /dev/random # mv /dev/h /dev/random
4)将corosync.conf和authkey复制至node2：
# scp -p corosync.conf authkey node2:/etc/corosync/
5)分别在node1、node2两个节点中创建corosync生成的日志所在的目录
# mkdir /var/log/cluster # ssh node2 'mkdir /var/log/cluster'
6)启动corosync服务
# service corosync start # ssh node2 '/etc/init.d/corosync start'
7)查询Corosync集群状态
# crm status Last updated: Thu Aug 18 23:22:20 2016 Last change: Thu Aug 18 22:49:11 2016 via crmd on node2.samlee.com Stack: classic openais (with plugin) Current DC: node2.samlee.com - partition with quorum Version: 1.1.10-14.el6-368c726 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ node1.samlee.com node2.samlee.com ]
以上为corosync集群配置所有过程。
3、安装配置drbd共享存储管理工具
(1)下载rpm安装包直接安装即可:
# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
(2)配置/etc/drbd.d/global-common.conf
# vim /etc/drbd.d/global_common.conf
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
#wfc-timeout 120;
#degr-wfc-timeout 120;
}
disk {
on-io-error detach;
#fencing resource-only;
}
net {
cram-hmac-alg "sha1";
shared-secret "mydrbdlab";
}
syncer {
rate 1000M;
}
}
(3)创建共享存储分区大小为5G(node1,node2都必须执行分区)。
# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (7675-15665, default 7675):
Using default value 7675
Last cylinder, +cylinders or +size{K,M,G} (7675-15665, default 15665): +5G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
#重新读取内核分区
# partx -a /dev/sda
# cat /proc/partitions
major minor #blocks name
8 0 125829120 sda
8 1 204800 sda1
8 2 61440000 sda2
8 3 5248836 sda3
(4)定义资源文件/etc/drbd.d/mystore.res,内容如下:
# vim /etc/drbd.d/mystore.res
resource mystore {
on node1.samlee.com {
device /dev/drbd0;
disk /dev/sda3;
address 172.16.100.6:7789;
meta-disk internal;
}
on node2.samlee.com {
device /dev/drbd0;
disk /dev/sda3;
address 172.16.100.7:7789;
meta-disk internal;
}
}
以上文件在两个节点上必须相同，因此，可以基于ssh将刚才配置的文件全部同步至另外一个节点。
# scp /etc/drbd.d/* node2:/etc/drbd.d/
(5)在两个节点上初始化已定义的资源并启动服务：
1)初始化资源，在Node1和Node2上分别执行：
# drbdadm create-md mystore
2)启动服务，在Node1和Node2上分别执行：
# service drbd start
3)查看启动状态：
# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5248636
也可以使用drbd-overview命令来查看：
# drbd-overview 0:mystore/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
从上面的信息中可以看出此时两个节点均处于Secondary状态。于是，我们接下来需要将其中一个节点设置为Primary。在要设置为Primary的节点上执行如下命令：
# drbdadm primary --force mystore
注： 也可以在要设置为Primary的节点上使用如下命令来设置主节点：
# drbdadm -- --overwrite-data-of-peer primary mystore
(5)创建文件系统
文件系统的挂载只能在Primary节点进行，因此，也只有在设置了主节点后才能对drbd设备进行格式化,下面的操作在node1.samlee.com上完成：
# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate C r----- # mke2fs -t ext4 /dev/drbd0 # mkdir /mydata # mount /dev/drbd0 /mydata
(6)创建mysql用户及mysql数据存储目录规划
# groupadd -r -g 3306 mysql # useradd -r -g 3306 -d /mydata/data/ -s /sbin/nologin mysql --以上mysql用户node1和node2节点都要创建 # mkdir /mydata/data # chown mysql.mysql /mydata/data/
考虑生产环境数据安全,可使用ACL权限控制:
# setfacl -m u:mysql:rwx /mydata/data/ # getfacl /mydata/data/ getfacl: Removing leading '/' from absolute path names # file: mydata/data/ # owner: mysql # group: mysql user::rwx user:mysql:rwx group::r-x mask::rwx other::r-x
挂载好drbd资源后及mysql用户目录环境准备后下面开始安装配置mysql：
(1).安装开发环境及开发包组:
# yum -y groupinstall "Development Tools" # yum -y groupinstall "Development Libraries" # yum -y install pcre-devel # yum -y install ncurses-devel # yum -y install openssl-devel
(2).安装夸平台编译器cmake
# tar xf cmake-2.8.8.tar.gz # cd cmake-2.8.8 # ./configure # make && make install
(3).编译安装mysql-5.5.33
# tar xf mysql-5.5.33.tar.gz # cd mysql-5.5.33 # cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \ > -DMYSQL_DATADIR=/mydata/data \ > -DSYSCONFDIR=/etc \ > -DWITH_INNOBASE_STORAGE_ENGINE=1 \ > -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ > -DWITH_BLACKHOLE_STORAGE_ENGINE=1 \ > -DWITH_READLINE=1 \ > -DWITH_SSL=system \ > -DWITH_ZLIB=system \ > -DWITH_LIBWRAP=0 \ > -DMYSQL_UNIX_ADDR=/tmp/mysql.sock \ > -DDEFAULT_CHARSET=utf8 \ > -DDEFAULT_COLLATION=utf8_general_ci # make && make install
(4).数据库服务器初始化(数据初始化创建系统数据库，在主节点上创建一次即可)
# cd /usr/local/mysql/ # chown -R :mysql /usr/local/mysql/ #赋予目录用户组为mysql #运行./script/mysql_install_db 创建系统库 # scripts/mysql_install_db --user=mysql --datadir=/mydata/data/ # cp support-files/my-large.cnf /etc/my.cnf #复制配置文件 # vim/etc/my.cnf #编辑配置文件 在[mysqld]下添加此项: datadir = /mydata/data innodb_file_per_table = on # cp support-files/mysql.server /etc/init.d/mysqld #复制启动脚本 # chkconfig --add mysqld #加入服务列表 # service mysqld start
导出头文件,库文件和man文档:
# ln -sv /usr/local/mysql/include/ /usr/include/mysql # vim /etc/ld.so.conf.d/mysql.conf 添加此行: /usr/local/mysql/lib/ # ldconfig # ldconfig -p | grep mysql # vim /etc/man.config 添加此行: MANPATH /usr/local/mysql/man # man mysqld
(5)配置PATH环境变量
# vim /etc/profile.d/mysql.sh export PATH=$PATH:/usr/local/mysql/bin # chmod +x /etc/profile.d/mysql.sh # . /etc/profile.d/mysql.sh
(6)测试成功如下:
node1:
[root@node1 ~]# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate C r----- /mydata ext4 5.0G 168M 4.6G 4% [root@node1 ~]# ll /mydata/data/ total 28700 -rw-rw---- 1 mysql mysql 18874368 Aug 19 01:03 ibdata1 -rw-rw---- 1 mysql mysql 5242880 Aug 19 01:03 ib_logfile0 -rw-rw---- 1 mysql mysql 5242880 Aug 19 01:03 ib_logfile1 drwx------ 2 mysql root 4096 Aug 19 01:00 mysql -rw-rw---- 1 mysql mysql 107 Aug 19 01:03 mysql-bin.000001 -rw-rw---- 1 mysql mysql 19 Aug 19 01:03 mysql-bin.index -rw-r----- 1 mysql root 1900 Aug 19 01:03 node1.samlee.com.err -rw-rw---- 1 mysql mysql 6 Aug 19 01:03 node1.samlee.com.pid drwx------ 2 mysql mysql 4096 Aug 19 01:00 performance_schema drwx------ 2 mysql root 4096 Aug 19 01:00 test 连接登陆mysql #mysql 创建一个远程用户： mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'172.16.%.%' IDENTIFIED BY 'redhat'; 重读授权表 mysql> FLUSH PRIVILEGES;
将node1停止mysql服务降级为备节点:
# service mysqld stop # umount /mydata/ # drbdadm secondary mystore # drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
提升node2为主节点启动mysql服务:
# drbdadm primary mystore # mount /dev/drbd0 /mydata/ # service mysqld start
测试完成后，停止所有服务，将所有节点降级为备节点状态:
# service mysqld stop # umount /mydata/ # drbdadm secondary mystore # drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
停止所有节点drbd服务，drbd服务必须加入服务列表中：
# chkconfig --add drbd # chkconfig --list drbd drbd 0:off 1:off 2:on 3:on 4:on 5:on 6:off # chkconfig drbd off # service drbd stop
配置drbd+mysql资源高可用配置如下：
crm(live)configure# property stonith-enabled=false crm(live)configure# property no-quorum-policy=ignore crm(live)configure# rsc_defaults resource-stickiness=100 crm(live)configure# verify crm(live)configure# commit crm(live)configure# primitive mysqlvip ocf:heartbeat:IPaddr params ip=172.16.100.66 nic=eth0 cidr_netmask=16 crm(live)configure# verify crm(live)configure# primitive mysqlstore ocf:linbit:drbd params drbd_resource=mystore op monitor role=Master interval=30s timeout=30s op monitor role=Slave interval=60s timeout=20s op start timeout=240s op stop timeout=100s crm(live)configure# verify crm(live)configure# master ms_mysqlstore mysqlstore meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify="true" crm(live)configure# verify crm(live)configure# primitive mysqlfs ocf:heartbeat:Filesystem params device=/dev/drbd0 directory=/mydata fstype=ext4 op monitor interval=30s timeout=40s op start timeout=60s op stop timeout=60s on-fail=restart crm(live)configure# verify crm(live)configure# primitive mysqlserver lsb:mysqld crm(live)configure# verify crm(live)configure# colocation mysqlfs_with_ms_mysqlstore_master inf: mysqlfs ms_mysqlstore:Master crm(live)configure# verify crm(live)configure# colocation mysqlserver_with_mysqlfs inf: mysqlfs mysqlserver crm(live)configure# verify crm(live)configure# colocation mysqlvip_with_mysqlserver inf: mysqlvip mysqlserver crm(live)configure# verify crm(live)configure# order mysqlfs_after_ms_mysqlstore_master mandatory: ms_mysqlstore:promote mysqlfs:start crm(live)configure# verify crm(live)configure# order mysqlserver_after_ms_mysqlstore inf: mysqlfs mysqlserver crm(live)configure# verify crm(live)configure# commit # crm configure show node node1.samlee.com node node2.samlee.com primitive mysqlfs ocf:heartbeat:Filesystem \ params device="/dev/drbd0" directory="/mydata" fstype="ext4" \ op monitor interval="30s" timeout="40s" \ op start timeout="60s" interval="0" \ op stop timeout="60s" on-fail="restart" interval="0" primitive mysqlserver lsb:mysqld primitive mysqlstore ocf:linbit:drbd \ params drbd_resource="mystore" \ op monitor role="Master" interval="30s" timeout="30s" \ op monitor role="Slave" interval="60s" timeout="20s" \ op start timeout="240s" interval="0" \ op stop timeout="100s" interval="0" primitive mysqlvip ocf:heartbeat:IPaddr \ params ip="172.16.100.66" nic="eth0" cidr_netmask="16" ms ms_mysqlstore mysqlstore \ meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" colocation mysqlfs_with_ms_mysqlstore_master inf: mysqlfs ms_mysqlstore:Master colocation mysqlserver_with_mysqlfs inf: mysqlfs mysqlserver colocation mysqlvip_with_mysqlserver inf: mysqlvip mysqlserver order mysqlfs_after_ms_mysqlstore_master inf: ms_mysqlstore:promote mysqlfs:start order mysqlserver_after_ms_mysqlstore inf: mysqlfs mysqlserver property $id="cib-bootstrap-options" \ dc-version="1.1.10-14.el6-368c726" \ cluster-infrastructure="classic openais (with plugin)" \ expected-quorum-votes="2" \ stonith-enabled="false" \ no-quorum-policy="ignore" rsc_defaults $id="rsc-options" \ resource-stickiness="100"
查询集群状态信息:
# crm status Last updated: Fri Aug 19 02:03:41 2016 Last change: Fri Aug 19 02:00:03 2016 via cibadmin on node1.samlee.com Stack: classic openais (with plugin) Current DC: node2.samlee.com - partition with quorum Version: 1.1.10-14.el6-368c726 2 Nodes configured, 2 expected votes 5 Resources configured Online: [ node1.samlee.com node2.samlee.com ] mysqlvip (ocf::heartbeat:IPaddr): Started node1.samlee.com Master/Slave Set: ms_mysqlstore [mysqlstore] Masters: [ node1.samlee.com ] Slaves: [ node2.samlee.com ] mysqlfs (ocf::heartbeat:Filesystem): Started node1.samlee.com mysqlserver (lsb:mysqld): Started node1.samlee.com
通过客户连接测试如下:
# mysql -h172.16.100.66 -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.5.33-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema| | mysql | | performance_schema| | test | +--------------------+ 4 rows in set (0.00 sec)