构建heartbeat+mysql+master+ldirectord构建高可用的MYSQL主主复制
对于对数据库要求比较高的业务，我们可以用MYSQL 主主复制来保证业务的稳定性，同时我们又要确保业务切换的的自动化，无手动干预。就得借组heartbeat 和ldirectord来实现。
两台主机：第一步：MYSQL AA复制，第二步：高可用，第三步：负载均衡和状态检测.
环境准备：
node1主机：Centos5.5 eth0:192.168.112.129 eth1:192.168.198.128
node2主机：Centos5.5 eth0:192.168.112.130 eth1:192.168.198.129
vip节点：192.168.112.131
以下操作在2台机器同时执行：
[root@Jiechao ~]# /usr/sbin/ntpdate pool.ntp.org
30 Dec 03:39:40 ntpdate[5099]: step time server 61.153.197.226 offset 2592175.431928 sec
[root@Jiechao ~]# date
Fri Dec 30 03:39:42 CST 2011
[root@Jiechao ~]# echo '*/10 * * * * /usr/sbin/ntpdate pool.ntp.org'>>/var/spool/cron/root
[root@Jiechao ~]# /etc/init.d/iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
[root@Jiechao ~]# chkconfig --level 3 iptables off
[root@Jiechao ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
两台主机共同执行操作：
ifconfig eth0:0 192.168.112.131 netmask 255.255.255.255 up #创建VIP节点
在2台主机安装：
安装软件
[root@node1 ~]# yum -y install mysql mysql-server perl-DBI perl-DBD-MySQL ##节约时间我全部用yum来安装软件包
在node1上操作：
[root@node1 ~]# /etc/init.d/mysqld restart
[root@node1 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.77 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> grant replication slave,file on *.* to 'repl1'@'192.168.112.130' identified by '123456';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
在node2上操作：
[root@node2 ~]# /etc/init.d/mysqld restart
[root@node2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.77 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> grant replication slave,file on *.* to 'repl2'@'192.168.112.129' identified by '123456';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
然后2台主机都停止MYSQL 服务器。
[root@node1 ~]# /etc/init.d/mysqld stop
Stopping MySQL: [ OK ]
[root@node2 ~]# /etc/init.d/mysqld stop
Stopping MySQL: [ OK ]
在node1主机上操作：
[root@node1 ~]# vi /etc/my.cnf 添加：
log-bin=mysql-bin
server-id= 1
binlog-do-db=test
binlog-ignore-db=mysql
replicate-do-db=test
replicate-ignore-db=mysql
log-slave-updates
slave-skip-errors=all
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=1
在node2主机上操作：
[root@node2 ~]# vi /etc/my.cnf 添加：
log-bin=mysql-bin
server-id= 2
binlog-do-db=test
binlog-ignore-db=mysql
replicate-do-db=test
replicate-ignore-db=mysql
log-slave-updates
slave-skip-errors=all
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=2
重启2台mysql数据库：
[root@node1 ~]# /etc/init.d/mysqld restart
Stopping MySQL: [FAILED]
Starting MySQL: [ OK ]
[root@node2 ~]# /etc/init.d/mysqld restart
Stopping MySQL: [FAILED]
Starting MySQL: [ OK ]
在node1主机上操作：
[root@node1 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> flush tables with read lock\G
Query OK, 0 rows affected (0.00 sec)
mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 98
Binlog_Do_DB: test
Binlog_Ignore_DB: mysql
1 row in set (0.00 sec)
在node2主机上操作：
[root@node2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> flush tables with read lock\G
Query OK, 0 rows affected (0.00 sec)
mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 98
Binlog_Do_DB: test
Binlog_Ignore_DB: mysql
1 row in set (0.00 sec)
在node1主机上操作：
mysql> change master to
-> master_host='192.168.112.130',
-> master_user='repl2',
-> master_password='123456',
-> master_log_file='mysql-bin.000001',
-> master_log_pos=98;
Query OK, 0 rows affected (0.05 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
在node2主机上操作：
mysql> change master to
-> master_host='192.168.112.129',
-> master_user='repl1',
-> master_password='123456',
-> master_log_file='mysql-bin.000001',
-> master_log_pos=98;
Query OK, 0 rows affected (0.05 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
接下来：查看各自机器上的IO进程和 SLAVE进程是否都开启。
node1主机：
mysql> show processlist\G
*************************** 1. row ***************************
Id: 2
User: root
Host: localhost
db: NULL
Command: Query
Time: 0
State: NULL
Info: show processlist
*************************** 2. row ***************************
Id: 3
User: system user
Host:
db: NULL
Command: Connect
Time: 83
State: Waiting for master to send event
Info: NULL
*************************** 3. row ***************************
Id: 4
User: system user
Host:
db: NULL
Command: Connect
Time: 67
State: Has read all relay log; waiting for the slave I/O thread to update it
Info: NULL
*************************** 4. row ***************************
Id: 5
User: repl1
Host: 192.168.112.130:49409
db: NULL
Command: Binlog Dump
Time: 19
State: Has sent all binlog to slave; waiting for binlog to be updated
Info: NULL
4 rows in set (0.00 sec)
node2主机：
mysql> show processlist\G
*************************** 1. row ***************************
Id: 2
User: root
Host: localhost
db: NULL
Command: Query
Time: 0
State: NULL
Info: show processlist
*************************** 2. row ***************************
Id: 3
User: repl2
Host: 192.168.112.129:48953
db: NULL
Command: Binlog Dump
Time: 99
State: Has sent all binlog to slave; waiting for binlog to be updated
Info: NULL
*************************** 3. row ***************************
Id: 4
User: system user
Host:
db: NULL
Command: Connect
Time: 67
State: Waiting for master to send event
Info: NULL
*************************** 4. row ***************************
Id: 5
User: system user
Host:
db: NULL
Command: Connect
Time: 51
State: Has read all relay log; waiting for the slave I/O thread to update it
Info: NULL
4 rows in set (0.00 sec)
然后：释放掉各自的锁，然后进行插数据测试。
在node1上操作：
mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)
mysql> use test;
Database changed
mysql> create table test
-> (id int not null auto_increment primary key,
-> str varchar(255) not null) engine myisam;
Query OK, 0 rows affected (0.06 sec)
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test |
+----------------+
1 row in set (0.00 sec)
mysql> insert into test(str) values
-> ('This is a master to master test table');
Query OK, 1 row affected (0.02 sec)
mysql> select * from test;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
+----+---------------------------------------+
1 row in set (0.00 sec)
在node2查看，是否有test这个表，然后在node2插入数据，测试，看node1是否存在该数据。
[root@node2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test |
+----------------+
1 row in set (0.00 sec)
mysql> select * from test;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
+----+---------------------------------------+
1 row in set (0.00 sec)
mysql> insert test(str) values('This is a test 2');
Query OK, 1 row affected (0.04 sec)
mysql> select * from test;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
| 2 | This is a test 2 |
+----+---------------------------------------+
2 rows in set (0.00 sec)
返回node1主机，查看是否存在node2刚刚插入的数据结果？
[root@node1 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test |
+----------------+
1 row in set (0.00 sec)
mysql> select * from test;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
| 2 | This is a test 2 |
+----+---------------------------------------+
2 rows in set (0.00 sec)
呵呵，是有结果的，证明MySQL-Master+Master已经OK了，接下来，准备搭建heartbeat和ldirectord。
在node1和node2主机上安装heartbeat高可用软件。注：heartbeat需要安装两次，可能是软件本身的Bug问题吧。
[root@node1 ~]# yum -y install heartbeat
Total 450 kB/s | 3.2 MB 00:07
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : heartbeat-pils 1/4
Installing : heartbeat-stonith 2/4
Installing : PyXML 3/4
useradd: user hacluster exists
error: %pre(heartbeat-2.1.3-3.el5.centos.i386) scriptlet failed, exit status 9 看到2处的吧
error: install: %pre scriptlet failed (2), skipping heartbeat-2.1.3-3.el5.centos 看到2处的吧
Installed:
heartbeat.i386 0:2.1.3-3.el5.centos
Dependency Installed:
PyXML.i386 0:0.8.4-4.el5_4.2 heartbeat-pils.i386 0:2.1.3-3.el5.centos
heartbeat-stonith.i386 0:2.1.3-3.el5.centos
Complete!
在重新安装一次：
[root@node1 ~]# yum -y install heartbeat
Total download size: 1.7 M
Downloading Packages:
heartbeat-2.1.3-3.el5.centos.i386.rpm | 1.7 MB 00:03
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : heartbeat 1/1
Installed:
heartbeat.i386 0:2.1.3-3.el5.centos
Complete!
这下好了吧。
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
在node1上操作：
#接下来拷贝 cp /usr/share/doc/heartbeat-2.1.3/ 下面的 ha.cf,authkeys,haresources 到/etc/ha.d 目录下面
[root@node1 ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d
[root@node1 ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d
[root@node1 ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d
[root@node1 ~]# cd /etc/ha.d
[root@node1 ha.d]# chmod 600 authkeys
[root@node1 ha.d]# cat authkeys
#
# Authentication file. Must be mode 600
#
#
# Must have exactly one auth directive at the front.
# auth send authentication using this method-id
#
# Then, list the method and key that go with that method-id
#
# Available methods: crc sha1, md5. Crc doesn't need/want a key.
#
# You normally only have one authentication method-id listed in this file
#
# Put more than one to make a smooth transition when changing auth
# methods and/or keys.
#
#
# sha1 is believed to be the "best", md5 next best.
#
# crc adds no security, except from packet corruption.
# Use only on physically secure networks.
#
auth 1 把原先的#去掉
1 crc 把原先的#去掉
#2 sha1 HI!
#3 md5 Hello!
[root@node1 ha.d]# vi ha.cf
debugfile /var/log/ha-debug 把原先的#去掉
logfile /var/log/ha-log 把原先的#去掉
logfacility local0 把原先的#去掉
keepalive 2 把原先的#去掉
deadtime 10 把原先的#去掉
warntime 5 把原先的#去掉
initdead 120 把原先的#去掉
ucast eth1 192.168.192.129 //这里是对方的ip 主要用于HA的相互检测网络接口要注意了，我用的是专用接口，node2主机改为192.168.192.128
auto_failback on 把原先的#去掉
watchdog /dev/watchdog //软件狗 把原先的#去掉
node node1 把原先的#去掉
node node2 把原先的#去掉
ping 192.168.192.1 把原先的#去掉
respawn hacluster /usr/lib/heartbeat/ipfail 把原先的#去掉
apiauth ipfail gid=haclient uid=hacluster 把原先的#去掉
[root@node1 ha.d]# vi haresources 添加：
node1 IPaddr::192.168.112.131/24/eth0:1 ldirectord
然后在scp到node2主机上：
[root@node1 ha.d]# scp -r /etc/ha.d 192.168.112.130:/etc
The authenticity of host '192.168.112.130 (192.168.112.130)' can't be established.
RSA key fingerprint is dd:73:2c:cf:3c:03:99:3c:95:f5:02:3c:b2:59:43:ac.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.112.130' (RSA) to the list of known hosts.
reverse mapping checking getaddrinfo for bogon failed - POSSIBLE BREAK-IN ATTEMPT!
[email protected]'s password:
README.config 100% 692 0.7KB/s 00:00
harc 100% 745 0.7KB/s 00:00
Delay 100% 910 0.9KB/s 00:00
Xinetd 100% 666 0.7KB/s 00:00
db2 100% 760 0.7KB/s 00:00
portblock 100% 739 0.7KB/s 00:00
LinuxSCSI 100% 1091 1.1KB/s 00:00
ServeRAID 100% 1012 1.0KB/s 00:00
IPaddr2 100% 1825 1.8KB/s 00:00
LVSSyncDaemonSwap 100% 4137 4.0KB/s 00:00
IPaddr 100% 2273 2.2KB/s 00:00
WAS 100% 1295 1.3KB/s 00:00
IPv6addr 100% 622 0.6KB/s 00:00
LVM 100% 790 0.8KB/s 00:00
Raid1 100% 1160 1.1KB/s 00:00
SendArp 100% 1563 1.5KB/s 00:00
OCF 100% 2923 2.9KB/s 00:00
WinPopup 100% 1166 1.1KB/s 00:00
MailTo 100% 1125 1.1KB/s 00:00
ids 100% 3424 3.3KB/s 00:00
apache 100% 828 0.8KB/s 00:00
AudibleAlarm 100% 805 0.8KB/s 00:00
IPsrcaddr 100% 1391 1.4KB/s 00:00
ICP 100% 951 0.9KB/s 00:00
hto-mapfuncs 100% 2325 2.3KB/s 00:00
Filesystem 100% 1903 1.9KB/s 00:00
ha.cf 100% 11KB 10.6KB/s 00:00
ask_resources 100% 1383 1.4KB/s 00:00
ip-request-resp 100% 1088 1.1KB/s 00:00
hb_takeover 100% 373 0.4KB/s 00:00
ip-request 100% 1313 1.3KB/s 00:00
status 100% 133 0.1KB/s 00:00
haresources 100% 5956 5.8KB/s 00:00
authkeys 100% 643 0.6KB/s 00:00
shellfuncs 100% 7184 7.0KB/s 00:00
提示输入对方root用户的密码，全部转输完以后，注意修改/etc/ha.d/ha.cf 中的ucast eth1 192.168.192.129 为对方的IP也就是node1主机的ETH1的IP 改为ucast eth1 192.168.192.128
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
配置负载和状态检测
两台主机分别安装ldirectord,在配置。
[root@node1 ~]# yum -y install heartbeat-ldirectord
[root@node1 ~]# cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /etc/ha.d/
[root@node1 ha.d]# vi ldirectord.cf
#Sample configuration for a MySQL virtual service. 找到这行，修改以下配置。
virtual = 192.168.112.131:3306
real=192.168.112.129:3306 gate 10
real=192.168.112.130:3306 gate 10
# fallback=127.0.0.1:3306
service=mysql
scheduler=rr
# #persistent=600
# #netmask=255.255.255.255
protocol=tcp
# checktype=negotiate
login="jc"
passwd="jc"
database="test"
request="SELECT * FROM jc"
在两边的MYSQL test数据库中建立 jc这个表，然后给予这两个IP访问的权限，我是直接用%来表示.
mysql> grant all privileges on test.jc to 'jc'@'%' identified by 'jc';
Query OK, 0 rows affected (0.09 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
两边启动heartbeat 观察日志，用ipvsadm –Ln 查看状态，然后用客户端连接测试
[root@node1 ha.d]# /etc/init.d/heartbeat restart
Stopping High-Availability services:
[ OK ]
Waiting to allow resource takeover to complete:
[ OK ]
Starting High-Availability services:
2011/12/30_05:13:42 INFO: Resource is stopped
[ OK ]
[root@node2 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
2011/12/30_05:13:26 INFO: Resource is stopped
mknod: `/dev/watchdog': File exists
[ OK ]
[root@node1 ~]# /etc/init.d/ldirectord start
Starting ldirectord... success
然后我们查看下结果：2台主机的VIP是否已经接管了？
[root@node1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.112.131:3306 rr
-> 192.168.112.129:3306 Local 10 0 0
-> 192.168.112.130:3306 Route 10 0 0
[root@node2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.131:3306 rr
-> 192.168.112.129:3306 Route 0 0 0
-> 192.168.112.130:3306 Local 0 0 0
现在开始测试：
在node1，用jc用户进入，创建一个jc表。
[root@node1 ~]# mysql -ujc -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 45
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> create table jc
-> (id int not null auto_increment primary key,
-> str varchar(255) not null) engine myisam;
Query OK, 0 rows affected (0.04 sec)
mysql> insert into jc(str) values
-> ('This is a master to master test table');
Query OK, 1 row affected (0.04 sec)
然后我停掉Node2的heartbeat和ldirectord。
[root@node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@node2 ~]# /etc/init.d/ldirectord stop
Stopping ldirectord... success
[root@node2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 41
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| jc |
| test |
+----------------+
2 rows in set (0.00 sec)
mysql> select *from jc;
Empty set (0.00 sec)
mysql> select *from jc;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
+----+---------------------------------------+
1 row in set (0.00 sec)
OK，我看到结果了，我们在node2插入一个数据，把node1的heartbeat和ldirectord停掉，看下node1是否有结果了。
[root@node2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 50
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> insert jc(str) values('This is a test 2');
Query OK, 1 row affected (0.02 sec)
mysql> select *from jc;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
| 2 | This is a test 2 |
+----+---------------------------------------+
2 rows in set (0.00 sec)
mysql> select *from jc;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
| 2 | This is a test 2 |
+----+---------------------------------------+
2 rows in set (0.00 sec)
好，我返回node1主机，先把把node1的heartbeat和ldirectord停掉，看下node1是否有结果了。
[root@node1 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
logd is already stopped
[root@node1 ~]# /etc/init.d/ldirectord stop
Stopping ldirectord... success
[root@node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 62
Server version: 5.0.77-log Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| jc |
| test |
+----------------+
2 rows in set (0.00 sec)
mysql> select *from jc;
+----+---------------------------------------+
| id | str |
+----+---------------------------------------+
| 1 | This is a master to master test table |
| 2 | This is a test 2 |
+----+---------------------------------------+
2 rows in set (0.00 sec)
很好，我们在看下node1停掉了，node2的VIP是否接管了MYSQL两台主主的主机？
[root@node2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.131:3306 rr
-> 192.168.112.129:3306 Route 0 0 0
-> 192.168.112.130:3306 Local 0 0 0
最后我在设置mysql，heartbeat,ldirectord开机自动启动。
[root@node1 ~]# chkconfig heartbeat on
[root@node1 ~]# chkconfig mysqld on
[root@node1 ~]# chkconfig ldirectord on
[root@node2 ~]# chkconfig heartbeat on
[root@node2 ~]# chkconfig mysqld on
[root@node2 ~]# chkconfig ldirectord on