heartbeat+DRBD+mysql高可用集群实战
四台主机
主机名 IP地址 用途
dbm128 172.16.0.128 heartbeat+DRBD+MYSQL的primary节点
dbm129 172.16.0.129 heartbeat+DRBD+mysql 的secondary节点
dbs130 172.16.0.130 slave(mater_host=172.16.0.233)
dbs131 172.16.0.131 slave(mater_host=172.16.0.233)
配置前的准备
1.设置hostname及解析
编辑/etc/hosts文件，添加如下内容：
172.16.0.128 dbm128
172.16.0.129 dbm129
2.新增加一块/dev/sdb5硬盘存放DRBD数据（两台都增加）
命令如下：
[root@dbm128 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 6527.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Partition number (1-4): 1
First cylinder (1-6527, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-6527, default 6527):
Using default value 6527
Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (1-6527, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-6527, default 6527):
Using default value 6527
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
3.DRBD的部署（两台机器都这么做）
yum -y install gcc gcc-c++ kernel* 安装好后重启一下，免得后面出错
#wget http://oss.linbit.com/drbd/8.4/drbd-8.4.3.tar.gz
# tar zxvf drbd-8.4.3.tar.gz
#cd drbd-8.4.3
#./configure --prefix=/ --with-km
#make
#make install
4. 加载安装DRDB模块
#modprobe drbd
机器重启后DRBD模块消失，需要写入/etc/rc.local开机自动执行
通过lsmod来检查是否已经执行成功
[root@dbm128 drbd-8.4.3]# lsmod | grep drbd
drbd 327968 0
5.编辑配置文件（两台配置一样dbm128，dbm129）
如果后面有问题要把LINUX防火墙关闭，或者把端口增加过去
[root@dbm128 drbd-8.4.3]# cat /etc/drbd.conf
global {usage-count yes;}
common{syncer{rate 200M;}}
resource r0{
protocol C;
net{
cram-hmac-alg sha1;
shared-secret "FooFunFactory";
}
disk{
on-io-error detach;
fencing resource-only;
}
startup{
wfc-timeout 120;
degr-wfc-timeout 120;
}
device /dev/drbd1;
on dbm128{
disk /dev/sdb5;
address 172.16.0.128:7789;
meta-disk internal;
}
on dbm129{
disk /dev/sdb5;
address 172.16.0.129:7789;
meta-disk internal;
}
}
6.启动DRBD(两台操作)
可通过如下命令启动DRBD
[root@dbm128 drbd-8.4.3]# drbdadm create-md all
[root@dbm128 drbd-8.4.3]# /etc/init.d/drbd start
[root@dbm128 drbd-8.4.3]# drbdadm -- --overwrite-data-of-peer primary all （这一步只在主服务器上做）
在主机上查看DRBD状态
[root@dbm128 ~]# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm128, 2014-07-18 23:01:15
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-
ns:45789492 nr:0 dw:0 dr:45794304 al:0 bm:2794 lo:0 pe:3 ua:5 ap:0 ep:1 wo:f oos:6639292
[================>...] sync'ed: 87.4% (6480/51196)M
finish: 0:06:46 speed: 16,336 (18,732) K/sec
以上数据正在同步。下面同步完成
[root@dbm128 ~]# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm128, 2014-07-18 23:01:15
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:52426428 nr:0 dw:0 dr:52426428 al:0 bm:3200 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
在从机上查看DRBD状态
[root@dbm129 ~]# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm129, 2014-07-18 23:14:34
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:52426428 dw:52426428 dr:0 al:0 bm:3200 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
在执行过程中，如果出现上述信息，代表操作成功
7.格式化DRBD文件系统（只在主机上做）
[root@dbm128 ~]# mkfs.ext3 /dev/drbd1
注意开机不要自启动DRBD服务，因为要通过heartbeat来启动DRBD
8.挂载DRBD分区
[root@dbm128 ~]# mkdir /data
[root@dbm128 ~]# mount /dev/drbd1 /data
[root@dbm128 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
92G 4.9G 82G 6% /
/dev/sda1 99M 35M 59M 38% /boot
tmpfs 501M 0 501M 0% /dev/shm
/dev/drbd1 50G 180M 47G 1% /data
后面就把mysql数据目录放在/data上
9.安装MYSQL数据库（四台都要安装）
安装所需要系统库相关库文件
[root@dbm128 ~]#yum -y install gcc gcc-c++ gcc-g77 autoconf automake zlib* fiex* libxml* ncurses-devel libmcrypt* libtool-ltdl-devel* cmake
[root@dbm128 ~]# groupadd mysql
[root@dbm128 ~]# useradd -r -g mysql mysql
[root@dbm128 ~]# tar zxvf mysql-5.5.25.tar.gz
[root@dbm128 ~]# cd mysql-5.5.25
[root@dbm128 mysql-5.5.25]# cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql/ -DMYSQL_DATADIR=/opt/mysql/data -DMYSQL_UNIX_ADDR=/opt/mysql/data/mysqld.sock -DWITH_INNOBASE_STORAGE_ENGINE=1 -DENABLED_LOCAL_INFILE=1 -DMYSQL_TCP_PORT=3306 -DEXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DMYSQL_UNIX_ADDR=/opt/mysql/data/mysql.sock -DMYSQL_USER=mysql -DWITH_DEBUG=0
[root@dbm128 mysql-5.5.25]# make && make install
赋予相关的权限
[root@dbm128 mysql]# chown -R mysql:mysql /opt/mysql
[root@dbm128 mysql]# /opt/mysql/scripts/mysql_install_db --user=mysql --basedir=/opt/mysql --datadir=/opt/mysql/data
mysql配置文件
[root@dbm128 mysql]# cp /opt/mysql/support-files/mysql.server /etc/init.d/mysql
[root@dbm128 mysql]# cp /opt/mysql/support-files/my-large.cnf /etc/my.cnf
[root@dbm128 mysql]# chmod +x /etc/init.d/mysql
dbm128上操作就可以，因为会同步过去所以dbm129不用操作
[root@dbm128 mysql]# cp -r /opt/mysql/data /data/mysql
[root@dbm128 mysql]# chown -R mysql:mysql /data/mysql
[root@dbm128 mysql]# vi /etc/init.d/mysql (两台要修改dbm128,dbm129)
找到datadir=/opt/mysql/data 修改成datadir=/data/mysql
[root@dbm128 mysql]#/etc/init.d/mysql start 启动（dbm129不用去启动到时做heartbeat采用自动启动）
[root@dbm128 mysql]# /opt/mysql/bin/mysqladmin -u root password '123456' （三台都操作，dbm129不用操作已经同步过去了）
10.安装heartbeat(两台机器都安装)
要先安装DRBD,然后才能安装hearbeat
安装命令如下：
[root@dbm128 ~]# yum -y install heartbeat heartbeat-devel heartbeat-stonith heartbeat-pils
接下来配置文件
[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/ha.cf
[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/haresources
[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/authkeys
[root@dbm128 ~]# chmod 600 /etc/ha.d/authkeys
修改ha.cf
[root@dbm128 ~]# cat /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
deadtime 15
warntime 5
initdead 30
ucast eth0 172.16.0.129 (修改对方的节点IP)
ping_group group 172.16.0.128 172.16.0.129
auto_failback on (从节点修改为off)
node dbm128
node dbm129
respawn hacluster /usr/lib64/heartbeat/ipfail
respawn hacluster /usr/lib64/heartbeat/dopd
apiauth ipfail gid=haclient uid=hacluster
apiauth dopd gid=haclient uid=hacluster
增加到后面cat /etc/ha.d/authkeys
auth 1
1 sha1 HA_JACKBILLOW
增加到后面 cat /etc/ha.d/haresources
dbm128 drbddisk::r0 Filesystem::/dev/drbd1::/data mysql IPaddr::172.16.0.233/24/eth0
上面dbm128主机，如果dbm129修改本主机名。
启动heartbeat服务：
[root@dbm128 ~]/etc/init.d/heartbeat start
[root@dbm129 ~]/etc/init.d/heartbeat start
配置heartbeat 在启动级别3中自动启动:
[root@dbm128 ~]chkconfig --level 35 heartbeat on
[root@dbm129 ~]chkconfig --level 35 heartbeat on
观察heartbeat 启动日志是否正常
[root@dbm128 ~]# tail -f /var/log/ha-log
IPaddr[4782]: 2014/07/21_16:48:15 INFO: Success
heartbeat[4147]: 2014/07/21_16:48:57 info: dbm129 wants to go standby [foreign]
heartbeat[4147]: 2014/07/21_16:48:57 info: standby: acquire [foreign] resources from dbm129
heartbeat[5029]: 2014/07/21_16:48:57 info: acquire local HA resources (standby).
ResourceManager[5042]: 2014/07/21_16:48:57 info: Acquiring resource group: dbm128 drbddisk::r0 Filesystem::/dev/drbd1::/data mysql IPaddr::172.16.0.233/24/eth0
Filesystem[5083]: 2014/07/21_16:48:58 INFO: Running OK
IPaddr[5158]: 2014/07/21_16:48:58 INFO: Running OK
heartbeat[5029]: 2014/07/21_16:48:58 info: local HA resource acquisition completed (standby).
heartbeat[4147]: 2014/07/21_16:48:58 info: Standby resource acquisition done [foreign].
heartbeat[4147]: 2014/07/21_16:48:58 info: remote resource transition completed.
测试一下，看172.16.0.233是VIP，在两台主机之间漂移。
11.主从复制的配置
修改dbs130,dbs131主的my.cnf的server-id=1，每个同步服务都必须设定一个唯一的编号。
在Master（这里为dbm128机器）上增加一个用于复制的账号：
mysql>GRANT REPLICATION SLAVE ON *.* TO 'repl'@'172.16.0.%' IDENTIFIED BY '123456';
mysql>FLUSH REPLICATION;
mysql> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000018 | 107 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
设置Slave 主机
修改my.cnf的server-id,内容如下：
dbs130主机
server-id=3
dbs130主机
server-id=4
开启Master与Slave的同步
在Slave上执行如下命令
mysql>CHANGE MASTER TO MASTER_HOST='172.16.0.233',
->MASTER_USER='repl',
->MASTER_PASSWORD='123456',
->MASTER_LOG_FILE='mysql-bin.000018',
->MASTER_LOG_POS=107;
之后执行
mysql>slave start;
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.16.0.233
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000018
Read_Master_Log_Pos: 107
Relay_Log_File: dbs130-relay-bin.000007
Relay_Log_Pos: 253
Relay_Master_Log_File: mysql-bin.000018
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 107
Relay_Log_Space: 556
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
1 row in set (0.00 sec)
在其他几台slave机器上也执行上面的命令