Redis+Gearmand实现MySQL数据自动同步

一、Redis介绍

---

redis 是一个高性能的 key-value 数据库。 redis 的出现,很大程度补偿了memcached 这类 keyvalue 存储的不足,在部分场合可以对关系数据库起到很好的补充作用。它提供了 Python,Ruby,Erlang,PHP 客户端,使用很方便。Redis 的所有数据都是保存在内存中,然后不定期的通过异步方式保存到磁盘上(这称为“半持久化模式”);也可以把每一次数据变化都写入到一个 append only file(aof)里面(这称为“全持久化模式”)。

二、实验环境

---

selinux iptables off
操作系统：Red Hat Enterprise Linux Server release 6.5 (Santiago)
百度网盘链接：https://pan.baidu.com/s/1NJiPYioi_843WfqMAI4PsA 密码: wdf8

主机名	IP	安装服务	功能说明
server1	10.10.10.1	PHP、Nginx	基础环境：LNMP
server2	10.10.10.2	redis	缓存数据
server3	10.10.10.3	Mysql	通过触发器实现数据自动更新

三、安装PHP

---

为了方便搭建，这里我们使用RPM安装，编译安装博客：https://blog.csdn.net/Dream_ya/article/details/81585830 ！！！

1、RPM安装PHP

[root@server1 ~]# ls
php-5.3.3-38.el6.x86_64.rpm         php-devel-5.3.3-38.el6.x86_64.rpm  php-mbstring-5.3.3-38.el6.x86_64.rpm
php-cli-5.3.3-38.el6.x86_64.rpm     php-fpm-5.3.3-38.el6.x86_64.rpm    php-mysql-5.3.3-38.el6.x86_64.rpm
php-common-5.3.3-38.el6.x86_64.rpm  php-gd-5.3.3-38.el6.x86_64.rpm     php-pdo-5.3.3-38.el6.x86_64.rpm

[root@server1 ~]# yum install -y *

2、配置php.ini

[root@server1 ~]# vim /etc/php.ini

3、启动PHP

[root@server1 ~]# /etc/init.d/php-fpm start

4、查看所支持的模块

[root@server1 ~]# php -m|grep mysql
mysql
mysqli
pdo_mysql

四、安装Nginx

---

1、下载Nginx

RPM包下载地址：http://nginx.org/packages/rhel/6/x86_64/RPMS/

[root@server1 ~]#wget http://nginx.org/packages/rhel/6/x86_64/RPMS/nginx-1.14.0-1.el6.ngx.x86_64.rpm

2、安装Nginx

[root@server1 ~]# rpm -ivh nginx-1.14.0-1.el6.ngx.x86_64.rpm 

3、配置nginx

[root@server1 ~]# vim /etc/nginx/conf.d/default.conf     ###其配置文件是在/etc/nginx/nginx.conf，从此文件指定到default.conf

4、PHP发布页面

[root@server1 ~]# vim /usr/share/nginx/html/index.php

phpinfo()
?>

5、启动Nginx

[root@server1 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@server1 ~]# nginx

6、查看结果

http://10.10.10.1/               ###浏览器

7、使用用phpize添加外挂模块

（1）编译加载

[root@server1 ~]# yum install -y unzip gcc
[root@server1 ~]# unzip /root/phpredis-master.zip
[root@server1 ~]# cd /root/phpredis-master 
[root@server1 phpredis-master]# phpize 
Configuring for:
PHP Api Version:         20090626
Zend Module Api No:      20090626
Zend Extension Api No:   220090626

[root@server1 phpredis-master]# ./configure 
[root@server1 phpredis-master]# make && make install
[root@server1 phpredis-master]# vim /etc/php.d/redis.ini
extension=redis.so

（2）查看是否加载成功

[root@server1 ~]# php -m |grep redis
redis

[root@server1 ~]# /etc/init.d/php-fpm restart

五、配置Redis（server2）

---

1、安装Redis

[root@server2 ~]# wget http://download.redis.io/releases/redis-4.0.8.tar.gz
[root@server2 ~]# yum install -y gcc
[root@server2 ~]# tar xf /root/redis-4.0.8.tar.gz
[root@server2 ~]# cd /root/redis-4.0.10
[root@server2 redis-4.0.8]# make && make install
[root@server2 redis-4.0.8]# cd utils/
[root@server2 utils]# ./install_server.sh            ###一直按回车即可

2、配置6379.conf（server2）

[root@server2 ~]# vim /etc/redis/6379.conf

3、查看结果

[root@server2 ~]# /etc/init.d/redis_6379 restart

[root@server2 ~]# netstat -lntup|grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      4639/redis-server 0 

六、配置MySQL（server3）

---

1、安装MySQL

[root@server3 ~]# yum install -y mysql-server
[root@server3 ~]# /etc/init.d/mysqld restart

2、登陆数据库创建test、修改权限

[root@server3 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use test;
mysql> CREATE TABLE `test` (`id` int(7) NOT NULL AUTO_INCREMENT, `name` char(8) DEFAULT NULL, PRIMARY KEY (`id`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;
mysql> INSERT INTO `test` VALUES (1,'dream'),(2,'test2'),(3,'test3'),(4,'test4'),(5,'test5'),(6,'test6'),(7,'test7'),(8,'test8'),(9,'test9');
mysql> select * from test;
+----+-------+
| id | name  |
+----+-------+
|  1 | dream |
|  2 | test2 |
|  3 | test3 |
|  4 | test4 |
|  5 | test5 |
|  6 | test6 |
|  7 | test7 |
|  8 | test8 |
|  9 | test9 |
+----+-------+
9 rows in set (0.00 sec)

mysql> grant all on test.* to redis@'%' identified by '1';
mysql> flush privileges;

4、设置PHP页面

[root@server1 ~]# vim /usr/share/nginx/html/test.php

        $redis = new Redis();
        $redis->connect('10.10.10.2',6379) or die ("could net connect redis server");
        $query = "select * from test";
        for ($key = 1; $key < 10; $key++)
        {
                if (!$redis->get($key))
                {
                        //数据库连接权限密码
                        $connect = mysql_connect('10.10.10.3','redis','1');
                        mysql_select_db(test);
                        $result = mysql_query($query);
                        //如果没有找到$key,将查询sql结果缓存到redis
                        while ($row = mysql_fetch_assoc($result))
                        {
                                $redis->set($row['id'],$row['name']);
                        }
                        $myserver = 'mysql';
                        break;
                }
                else
                {
                        $myserver = "redis";
                        $data[$key] = $redis->get($key);
                }
        }

        echo $myserver;
        echo "
";
        for ($key = 1; $key < 10; $key++)
        {
                echo "number is $key";

                echo "
";

                echo "name is $data[$key]";

                echo "
";
        }
?>

5、进行测试

（1）测试页面

http://10.10.10.1/test.php

再次刷新：

（2）修改MySQL表内容

[root@server3 ~]# mysql
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> update test set name='test1' where id=1;
Query OK, 1 row affected (0.05 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from test;
+----+-------+
| id | name  |
+----+-------+
|  1 | test1 |
|  2 | test2 |
|  3 | test3 |
|  4 | test4 |
|  5 | test5 |
|  6 | test6 |
|  7 | test7 |
|  8 | test8 |
|  9 | test9 |
+----+-------+

http://10.10.10.1/test.php

我们可以发现页面并不会进行触发，因此我们下面介绍可以自动页面触发！！！

七、配置自动触发

---

官网链接：http://gearman.org/
client ->r/w redis -> mysql

1、Gearmand介绍

（1）Gearman 是一个支持分布式的任务分发框架

• Gearman Job Server: Gearman 核心程序,需要编译安装并以守护进程形式运行在后台。

• Gearman Client:可以理解为任务的请求者。

• Gearman Worker:任务的真正执行者,一般需要自己编写具体逻辑并通过守护进程方式运行,Gearman Worker接收到Gearman Client传递的任务内容后,会按顺序处理。

（2）原理

下面要编写的mysql触发器,就相当于Gearman的客户端。修改表,插入表就相当于直接下发任务。然后通过lib_mysqludf_json UDF库函数将关系数据映射为JSON格式,然后在通过gearman-mysql-udf插件将任务加入到Gearman的任务队列中,最后通过redis_worker.php,也就是 Gearman的worker端来完成redis数据库的更新。

2、安装gearmand

[root@server1 ~]# ls /mnt
libevent-1.4.13-4.el6.x86_64.rpm        libevent-headers-1.4.13-4.el6.noarch.rpm  libevent-devel-1.4.13-4.el6.x86_64.rpm  
libgearman-1.1.8-2.el6.x86_64.rpm         libyaml-0.1.3-4.el6_6.x86_64.rpm libevent-doc-1.4.13-4.el6.noarch.rpm 
libgearman-devel-1.1.8-2.el6.x86_64.rpm

[root@server1 ~]# yum install -y /mnt/lib*
[root@server1 ~]# yum install -y /root/gearmand-1.1.8-2.el6.x86_64.rpm 
[root@server1 ~]# /etc/init.d/gearmand restart

3、添加PHP插件gearman

[root@server1 ~]# tar xf gearman-1.1.2.tgz
[root@server1 ~]# cd gearman-1.1.2
[root@server1 gearman-1.1.2]# phpize 
Configuring for:
PHP Api Version:         20090626
Zend Module Api No:      20090626
Zend Extension Api No:   220090626

[root@server1 gearman-1.1.2]# ./configure
[root@server1 gearman-1.1.2]# make
[root@server1 gearman-1.1.2]# make install
[root@server1 gearman-1.1.2]# vim /etc/php.d/gearman.ini
extension=gearman.so

4、查看模块是否加载成功

[root@server1 gearman-1.1.2]# /etc/init.d/php-fpm reload
[root@server1 gearman-1.1.2]# php -m |grep gearman
gearman

5、编写worker.php

[root@server1 ~]# vim /root/worker.php     ###IP为redis的IP

$worker = new GearmanWorker();
$worker->addServer();
$worker->addFunction('syncToRedis', 'syncToRedis');
$redis = new Redis();
$redis->connect('10.10.10.2', 6379);
while($worker->work());
function syncToRedis($job)
{
        global $redis;
        $workString = $job->workload();
        $work = json_decode($workString);
        if(!isset($work->id)){
                return false;
        }
        $redis->set($work->id, $work->name);
}
?>

[root@server1 ~]# nohup php worker.php &          ###打入后台运行
[1] 5818

6、安装lib_mysqludf_json-master

[root@server3 mnt]# yum install -y unzip gcc mysql-devel       ###mysql-devel是依赖
[root@server3 mnt]# ls
gearman-mysql-udf-0.6.tar.gz  lib_mysqludf_json-master.zip

[root@server3 mnt]# unzip lib_mysqludf_json-master.zip 
[root@server3 mnt]# cd /mnt/lib_mysqludf_json-master
[root@server3 lib_mysqludf_json-master]# gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

6、配置lib_mysqludf_json.so模块

lib_mysqludf_json UDF库函数将关系数据映射为 JSON 格式。通常把数据库中的数据映射为 JSON 格式,是通过程序来转换的，官网查看链接。

（1）查看MySQL模块位置

[root@server3 ~]# mysql
mysql> show global variables like 'plugin_dir';

（2）拷贝lib_mysqludf_json.so模块到MySQL模块位置

[root@server3 ~]# cp /mnt/lib_mysqludf_json-master/lib_mysqludf_json.so /usr/lib64/mysql/plugin/

7、安装gearman-mysql-udf

查看链接：https://launchpad.net/gearman-mysql-udf
插件功能：用来管理以及调用Gearman的分布式的队列！！！

[root@server3 ~]# ls
libevent-1.4.13-4.el6.x86_64.rpm
libevent-devel-1.4.13-4.el6.x86_64.rpm
libevent-doc-1.4.13-4.el6.noarch.rpm
libevent-headers-1.4.13-4.el6.noarch.rpm
libgearman-1.1.8-2.el6.x86_64.rpm
libgearman-devel-1.1.8-2.el6.x86_64.rpm
libyaml-0.1.3-4.el6_6.x86_64.rpm
redis-4.0.10
redis-4.0.10.tar.gz

[root@server3 ~]# yum install -y *
[root@server3 mnt]# tar xf /mnt/gearman-mysql-udf-0.6.tar.gz 
[root@server3 mnt]# cd gearman-mysql-udf-0.6
[root@server3 gearman-mysql-udf-0.6]# ./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
[root@server3 gearman-mysql-udf-0.6]# make && make install

8、注册UDF函数

[root@server3 ~]# mysql
mysql> CREATE FUNCTION json_object RETURNS STRING SONAME 'lib_mysqludf_json.so';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE FUNCTION gman_do_background RETURNS STRING SONAME 'libgearman_mysql_udf.so';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME 'libgearman_mysql_udf.so';
Query OK, 0 rows affected (0.00 sec)

mysql> select * from mysql.func;
+--------------------+-----+-------------------------+----------+
| name               | ret | dl                      | type     |
+--------------------+-----+-------------------------+----------+
| json_object        |   0 | lib_mysqludf_json.so    | function |
| gman_do_background |   0 | libgearman_mysql_udf.so | function |
| gman_servers_set   |   0 | libgearman_mysql_udf.so | function |
+--------------------+-----+-------------------------+----------+
3 rows in set (0.00 sec)

mysql> SELECT gman_servers_set('10.10.10.1:4730');        ###指定gearman的服务信息，这里为LNMP（server1）
+-------------------------------------+
| gman_servers_set('10.10.10.1:4730') |
+-------------------------------------+
| 10.10.10.1:4730                     |
+-------------------------------------+
1 row in set (0.00 sec)

9、编写mysql触发器

[root@server3 ~]# vim test.sql
use test;
DELIMITER $$
CREATE TRIGGER datatoredis AFTER UPDATE ON test FOR EACH ROW BEGIN
    SET @RECV=gman_do_background('syncToRedis', json_object(NEW.id as `id`, NEW.name as `name`));
  END$$
DELIMITER ;

[root@server3 ~]# mysql < test.sql

10、查看触发器

mysql> SHOW TRIGGERS FROM test;
mysql> show triggers from test\G;

11、测试

注意：要看下server1的：nohup php worker.php &是否还在后台运行！！！

mysql> use test;
Database changed

mysql> set @RECV = 1;
Query OK, 0 rows affected (0.00 sec)

mysql> select @RECV;
+-------+
| @RECV |
+-------+
|     1 |
+-------+
1 row in set (0.00 sec)

mysql> update test set name='dreamya' where id=1;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0

mysql> select @RECV;
+-------------+
| @RECV       |
+-------------+
| H:server1:1 |
+-------------+
1 row in set (0.00 sec)

mysql> select @RECV;
+-------------+
| @RECV       |
+-------------+
| H:server1:1 |
+-------------+
1 row in set (0.00 sec)

mysql> select * from test;
+----+---------+
| id | name    |
+----+---------+
|  1 | dreamya |
|  2 | test2   |
|  3 | test3   |
|  4 | test4   |
|  5 | test5   |
|  6 | test6   |
|  7 | test7   |
|  8 | test8   |
|  9 | test9   |
+----+---------+

mysql> update test set name = 'dreamya' where id = 1;