corosync+pacemaker+RA实现mysql高可用

操作目的：

基于Corosync和NFS服务器实现MySQL的高可用

规划：

准备工作：根据规划图，设置三台主机的主机名和地址
（这里不再给出具体过程）

一、配置nfs服务器

#fdisk /dev/sda
创建一个大小为20G 类型为8e的分区 本文为/dev/sda5

#partprobe /dev/sda
#pvcreate /dev/sda5
#vgcreate myvg /dev/sda5
#lvcreate -L 10G -n mydata myvg
#mke2fs -j -L MYDATA /dev/myvg/mydata
#mkdir /mydata
#vim /etc/fstab
添加
LABEL=MYDATA            /mydata                 ext3    defaults        0 0
#mount -a

添加用户
# groupadd -g 306 -r mysql
# useradd -g mysql -r -u 306 -s /sbin/nologin mysql
# id mysql
uid=306(mysql) gid=306(mysql) groups=306(mysql) context=root:system_r:unconfined_t:SystemLow-SystemHigh

让mysql用户对/mydata目录有读写的权限
# chown -R mysql:mysql /mydata/

通过nfs共享/mydata
#vim /etc/exports
/mydata         172.16.220.11(rw,no_root_squash) 172.16.220.12（rw,no_root_squash)

#service nfs start
# showmount -e localhost
Export list for localhost:/mydata 172.16.220.12,172.16.220.11

二、配置mysql的高可用

1  配置node1 node2的时间同步和ssh互联

node1:
#hwclock -s
#ssh-keygen -i rsa
#vim /etc/hosts
172.16.220.11 node1
172.16.220.12 node2
# ssh-copy-id -i .ssh/id_rsa.pub root@node2
# ssh node2 'ifconfig'

node2:
#hwclock -s
#ssh-keygen -i rsa
#vim /etc/hosts
172.16.220.11 node1
172.16.220.12 node2
# ssh-copy-id -i .ssh/id_rsa.pub root@node1
# ssh node1 'ifconfig'

 

2  添加用户

node1 node2:
# groupadd -r -g 306 mysql
# useradd -g mysql -u 306 -r mysql
# mkdir /mydata
# mount -t nfs 172.16.220.22:/mydata /mydata/
# ls /mydata/
lost+found

验证mysql用户是否对/mydata目录有写的权限
# su - mysql
su: warning: cannot change directory to /home/mysql: No such file or directory
-bash-3.2$ cd /mydata
-bash-3.2$ mkdir data   //创建这个目录，作为mysql的数据目录
-bash-3.2$ ls
data  lost+found

 

3  安装mysql

node1 :
安装 mysql-5.5.22-linux2.6-i686.tar.gz

# tar xf mysql-5.5.22-linux2.6-i686.tar.gz -C /usr/local/
# cd /usr/local/
# ln -sv mysql-5.5.22-linux2.6-i686 mysql
#cd mysql

初始化mysql
#chown -R mysql:mysql .
# scripts/mysql_install_db --user=mysql --datadir=/mydata/data/


提供配置文件
# cp /usr/local/mysql/support-files/my-large.cnf /etc/my.cnf
#vim /etc/my.cnf
[mysqld]中修改
thread_concurrency = 2
datadir = /mydata/data

提供脚本
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
# chkconfig --add mysqld
# service mysqld start
Starting MySQL........                                     [  OK  ]
# /usr/local/mysql/bin/mysql   连上mysql



停止服务
#service mysqld stop
# chkconfig mysqld off
# chkconfig --list mysqld      //保证都是off

node2:
安装 mysql-5.5.22-linux2.6-i686.tar.gz

# tar xf mysql-5.5.22-linux2.6-i686.tar.gz -C /usr/local/
# cd /usr/local/
# ln -sv mysql-5.5.22-linux2.6-i686 mysql
#cd mysql
#chown -R root:mysql
# cp /usr/local/mysql/support-files/my-large.cnf /etc/my.cnf
#vim /etc/my.cnf
[mysqld]中修改
thread_concurrency = 2
datadir = /mydata/data
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
# chkconfig --add mysqld
# service mysqld start
Starting MySQL........                                     [  OK  ]
# /usr/local/mysql/bin/mysql   连上mysql
#service mysqld stop
# chkconfig mysqld off
# chkconfig --list mysqld      //保证都是off

到此两个mysql服务器可以实现nfs的共享存储了

 

4  添加集群服务

准备的包：
 

corosync-1.2.7-1.1.el5.i386.rpm
corosynclib-1.2.7-1.1.el5.i386.rpm

pacemaker-1.1.5-1.1.el5.i386.rpm
pacemaker-libs-1.1.5-1.1.el5.i386.rpm

heartbeat-3.0.3-2.3.el5.i386.rpm
heartbeat-libs-3.0.3-2.3.el5.i386.rpm

cluster-glue-1.0.6-1.6.el5.i386.rpm
cluster-glue-libs-1.0.6-1.6.el5.i386.rpm

perl-TimeDate-1.16-5.el5.noarch.rpm
libesmtp-1.0.4-5.el5.i386.rpm
resource-agents-1.0.4-1.1.el5.i386.rpm

下载到node1 node2 的/root目录下

4.1安装corosync

node1 node2:
#yum -y --nogpgcheck localinstall *.rpm

 

4.2 corosync的配置文件

node1:
# cd /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
#vim /etc/corosync.conf
修改内容如下：
secauth: on
bindnetaddr: 172.16.0.0
mcastaddr: 226.99.1.1
 to_logfile: yes
 to_syslog: no

接着编辑添加如下内容：
service {
        ver: 0
        name: pacemaker
        use_mgmtd: yes
}

aisexec {
        user: root
        group: root
}

生成节点间通信时用到的认证密钥文件：
# corosync-keygen        //会自动的在/etc/corosync目录下生成一个key文件，是二进制格式的，权限为400

将corosync和authkey复制至node2:
# scp -p authkey corosync.conf node2:/etc/corosync/

分别为两个节点创建corosync生成的日志所在的目录：
# mkdir  /var/log/cluster
# ssh node2 'mkdir  /var/log/cluster'

启动服务
# service corosync start
# ssh node2 '/etc/init.d/corosync start'

查看工作状态
# crm status
============
Last updated: Tue Aug 21 09:07:33 2012
Stack: openais
Current DC: node1 - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node1 node2 ]

 

三、配置集群资源

node1:（集群资源的配置，在一个节点上完成即可）

1 禁用stonith
# crm configure property stonith-enabled=false

2 修改忽略quorum不能满足的集群状态检查：
# crm configure property no-quorum-policy=ignore

3 为资源指定默认黏性值：
# crm configure rsc_defaults resource-stickiness=100

4 配置资源（资源的添加，在哪个节点上都可以）

1）添加访问mysql的Ip
# crm configure primitive myip ocf:heartbeat:IPaddr params ip=172.16.220.21

2）添加挂载目录的资源
 

现在node1 node2上卸载挂载的目录
#umount /mydata

#crm
crm(live)configure# primitive mynfs ocf:heartbeat:Filesystem params device="172.16.220.22:/mydata" directory="/mydata" fstype="nfs" op start timeout=60s op stop timeout=60s

资源的工作状态
# crm status
============
Last updated: Tue Aug 21 09:46:30 2012
Stack: openais
Current DC: node1 - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ node1 node2 ]

 myip (ocf::heartbeat:IPaddr): Started node1
 mynfs (ocf::heartbeat:Filesystem): Started node2

此时可以测试：
mynfs资源在node2上，查看即可
# ls /mydata/data
ibdata1      ib_logfile1  mysql-bin.000001  mysql-bin.000003  node1.err  performance_schema
ib_logfile0  mysql        mysql-bin.000002  mysql-bin.index   node2.err  test
有数据文件了，此时在node1上是没有/data目录的

让node2变为被动节点
#crm node standby
# crm status
============
Last updated: Tue Aug 21 09:51:52 2012
Stack: openais
Current DC: node1 - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Node node2: standby
Online: [ node1 ]

 myip (ocf::heartbeat:IPaddr): Started node1
 mynfs (ocf::heartbeat:Filesystem): Started node1
此时node1上查看
#ls /mydata/data
ibdata1      ib_logfile1  mysql-bin.000001  mysql-bin.000003  node1.err  performance_schema
ib_logfile0  mysql        mysql-bin.000002  mysql-bin.index   node2.err  test

 

3）添加mysqld服务

（mysqld 一定要和myip mynfs在一起，并且mysqld要晚于mynfs和myip）
# crm
crm(live)# configure
crm(live)configure# primitive mysqld lsb:mysqld
定义排列约束
crm(live)configure# colocation mysqld_with_mynfs_with_myip inf: mysqld mynfs myip

定义顺序约束
crm(live)configure# order mysqld_after_myip mandatory: myip mysqld
crm(live)configure# order mysqld_after_mynfs mandatory: mynfs mysqld:start
定义完顺序之后，查看xml
crm(live)configure# show xml 要确保是order是如下内容
<rsc_order first="mynfs" id="mysqld_after_mynfs" score="INFINITY" then="mysqld" then-action="start"/>
<rsc_order first="myip" id="mysqld_after_myip" score="INFINITY" then="mysqld"/>

提交配置
crm(live)configure# commit

#crm status
Online: [ node1 node2 ]

 myip (ocf::heartbeat:IPaddr): Started node1
 mynfs (ocf::heartbeat:Filesystem): Started node1
 mysqld (lsb:mysqld): Started node1
此时可以测试以下，mysqld在node1，就在node1上测试
#/usr/local/mysql/bin/mysql
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+           证明资源生效了

 

 建立一个远程连接的用户
 

mysql> grant all on *.* to root@'%' identified by 'redhat';
mysql> flush privileges;

 

远程连接172.16.220.21的mysql服务如图

 

 

 

操作过程中遇到的问题
 

1 添加mynfs资源时
Failed actions:
    mynfs_start_0 (node=node1, call=5, rc=5, status=complete): not installed
    mynfs_start_0 (node=node2, call=4, rc=5, status=complete): not installed
第一次使用
# crm configure primitive mynfs ocf:heartbeat:Filesystem params device="172.16.220.22/mydata" directory="/mydata" fstype="nfs" op start timeout=60s op stop timeout=60s
来添加的，出现了如上错误

第二次使用
#crm
crm(live)configure# primitive mynfs ocf:heartbeat:Filesystem params device="172.16.220.22:/mydata" directory="/mydata" fstype="nfs" op start timeout=60s op stop timeout=60s

工作正常了 诡异啊
2 添加进了mgmtd服务，但是5560服务还是没有启动。解决一下

3 可以远程连接所在节点的mysql，但是不能通过vip连接，怎么原因？
解决：地址冲突，其他主机占用了这个vip（一定要保证vip，虚拟ip没有被其他主机占用）