Gearman实现Mysql到Redis的数据同步

Gearman实现Mysql到Redis的数据同步

Gearman 是一个支持分布式的任务分发框架:
Gearman Job Server: Gearman 核心程序,需要编译安装并以守护进程形式运行在后台。
Gearman Client:可以理解为任务的请求者。
Gearman Worker:任务的真正执行者,一般需要自己编写具体逻辑并通过守护进程方式
运行,Gearman Worker 接收到 Gearman Client 传递的任务内容后,会按顺序处理。

大致流程:

1)下面要编写的 mysql 触发器,就相当于 Gearman 的客户端。

2)修改表,插入表就相当于直接下发任务。

3)再通过 lib_mysqludf_json UDF 库函数将关系数据映射为 JSON 格式

4)再通过 gearman-mysql-udf 插件将任务加入到 Gearman 的任务队列中

5)最后通过redis_worker.php,也就是 Gearman 的 worker 端来完成 redis 数据库的更新。

配置过程：

1、安装gearmand并开启服务

[root@server1 ~]# cd redis/
[root@server1 redis]# yum install -y gearmand-1.1.8-2.el6.x86_64.rpm libgearman-* libevent-*
[root@server1 redis]# /etc/init.d/gearmand start 

示图：安装过程存在依赖性，同步安装

[root@server1 redis]# vim worker.php    ##Master的ip
[root@server1 redis]# cp worker.php /usr/local/

2、编译、安装gearman

[root@server1 redis]# tar zxf gearman-1.1.2.tgz
[root@server1 redis]# cd gearman-1.1.2 
[root@server1 gearman-1.1.2]# phpize 
[root@server1 gearman-1.1.2]# ./configure 
[root@server1 gearman-1.1.2]# make && make install 
[root@server1 gearman-1.1.2]# cd /etc/php.d
[root@server1 php.d]# cp redis.ini gearman.ini
[root@server1 php.d]# vim gearman.ini 
extension=gearman.so
[root@server1 php.d]# /etc/init.d/php-fpm reload 
[root@server1 php.d]# php -m     
gearman 
mysql
redis

3、生成lib_mysqludf_json.so模块

[root@server1 redis]# scp lib_mysqludf_json-master.zip gearman-mysql-udf-0.6.tar.gz server3:
[root@server1 redis]# scp libevent-* server3:
[root@server1 redis]# scp libgearman-* server3:
[root@server3 ~]# yum install -y mysql-devel
[root@server3 ~]# yum install -y gcc
[root@server3 ~]# unzip lib_mysqludf_json-master.zip 
[root@server3 ~]# cd lib_mysqludf_json-master
[root@server3 lib_mysqludf_json-master]# ls
lib_mysqludf_json.c     lib_mysqludf_json.so   README.md
lib_mysqludf_json.html  lib_mysqludf_json.sql
[root@server3 lib_mysqludf_json-master]# gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c 
[root@server3 lib_mysqludf_json-master]# cp lib_mysqludf_json.so /usr/lib64/mysql/plugin/

4、注册 json 函数

[root@server3 ~]# /etc/init.d/mysqld start
[root@server3 ~]# mysql
mysql> show global variables like 'plugin_dir';
+---------------+-------------------------+
| Variable_name | Value                   |
+---------------+-------------------------+
| plugin_dir    | /usr/lib64/mysql/plugin |
+---------------+-------------------------+
mysql> create function json_object returns string soname 'lib_mysqludf_json.so';
mysql> select * from mysql.func;
+-------------+-----+----------------------+----------+
| name        | ret | dl                   | type     |
+-------------+-----+----------------------+----------+
| json_object |   0 | lib_mysqludf_json.so | function |
+-------------+-----+----------------------+----------+

5、注册UDF函数

[root@server3 ~]# yum install -y libevent-*
[root@server3 ~]# yum install -y libgearman-* 
[root@server3 ~]# tar zxf gearman-mysql-udf-0.6.tar.gz 
[root@server3 ~]# cd gearman-mysql-udf-0.6
[root@server3 gearman-mysql-udf-0.6]# ./configure --libdir=/usr/lib64/mysql/plugin/
[root@server3 gearman-mysql-udf-0.6]# make && make install 

6、指定 gearman 的服务信息

[root@server3 ~]# mysql
mysql> create function gman_do_background returns string soname 'libgearman_mysql_udf.so';
mysql> create function gman_servers_set returns string soname 'libgearman_mysql_udf.so';
mysql> select * from mysql.func;
+--------------------+-----+-------------------------+----------+
| name               | ret | dl                      | type     |
+--------------------+-----+-------------------------+----------+
| json_object        |   0 | lib_mysqludf_json.so    | function |
| gman_do_background |   0 | libgearman_mysql_udf.so | function |
| gman_servers_set   |   0 | libgearman_mysql_udf.so | function |
+--------------------+-----+-------------------------+----------+
mysql> select gman_servers_set('172.25.51.1:4730');
+--------------------------------------+
| gman_servers_set('172.25.51.1:4730') |
+--------------------------------------+
| 172.25.51.1:4730                     |
+--------------------------------------+

7、编写 mysql 触发器

[root@server3 ~]# vim test.sql 
[root@server3 ~]# mysql < test.sql 
mysql> show triggers from test;
[root@server3 ~]# mysql
mysql> show triggers from test;
+-------------+--------+-------+----------------------------------------------------------------------------------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+
| Trigger     | Event  | Table | Statement                                                                                                      | Timing | Created | sql_mode | Definer        | character_set_client | collation_connection | Database Collation |
+-------------+--------+-------+----------------------------------------------------------------------------------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+
| datatoredis | UPDATE | test  | BEGIN
    SET @RECV=gman_do_background('syncToRedis', json_object(NEW.id as `id`, NEW.name as `name`)); 
  END | AFTER  | NULL    |          | root@localhost | latin1               | latin1_swedish_ci    | latin1_swedish_ci  |
+-------------+--------+-------+----------------------------------------------------------------------------------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+

示图：test数据库内容

8、测试：

server1：开启nginx、php-fpm

server3：更改test数据库信息

[root@server3 ~]# mysql
mysql> use test
mysql> update test set name='hhh' where id=1;

客户端：