时间过得真快,6月份已经过去一半了,令人感慨万千;最近,相信大家都看了热点新闻,前几天某山事件看得老哥愤怒不已,多年扫黑除恶还是有漏网之鱼!这些被抓的黑恶势力,希望能够重判!重判!重判!
老哥能做的就是声援了,接着,重操旧业,还是继续更新文章,今天我们来学习一下Docker环境下,搭建Mysql8.0以及使用Canal实现与Redis的同步
环境:需要提前安装好Docker,Docker 安装 以及卸载
1、新建一个mysql文件夹
[root@VM-4-12-centos opt]# mkdir mysql
[root@VM-4-12-centos opt]# cd mysql/
[root@VM-4-12-centos mysql]# pwd
/opt/mysql
2、新建配置、数据、日志文件夹
[root@VM-4-12-centos mysql]# mkdir conf data logs
3、拉取mysql 8.0.29,如果不知道拉取哪个版本,可以参考DockHub,上面有版本还有具体参考的命令,DockHub -Mysql远程图书馆
[root@VM-4-12-centos mysql]# docker pull mysql:8.0.29
3A、启动mysql,老哥解释一下这些命令
\ 换行
-p 服务器端口:docker 里面mysql 的端口映射
-v 是数据卷挂载,将当前文件夹路径和docker里面mysql默认路径绑定,$PWD 当前文件夹
-e配置环境变量 密码,
–privileged 拥有root权限
-d 后台运行
docker run \
-p 3306:3306 \
--name mysql \
-v $PWD/conf:/etc/mysql/conf.d \
-v $PWD/logs:/logs \
-v $PWD/data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=root \
--privileged \
-d \
mysql:8.0.29
5、查看运行状态
[root@VM-4-12-centos mysql]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3933ae0a3984 mysql:8.0.29 "docker-entrypoint.s…" 10 minutes ago Up 10 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
6、我们登录mysql容器,然后新建一个test数据库
[root@VM-4-12-centos mysql]# docker exec -it mysql bash
root@e54909746c9a:/# mysql -uroot -p
mysql> create database test;
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
7、创建一个用户,稍后用于Canal连接Mysql8.0启动的用户,这里需要注意一下,MySQL8.0.16版本中新增了一个system_user帐户类型,如果没有加入一下参数,Root无法授权用户,会报错 ERROR 1227 (42000): Access denied; you need (at least one of) the SYSTEM_USER privilege(s) for this operation
mysql> grant system_user on *.* to 'root';
依次执行如下命令,一条一条执行
use mysql;
create user Jessica@'%' identified by 'root';
grant all privileges on *.* to Jessica@'%' with grant option;
ALTER USER 'Jessica'@'%' IDENTIFIED WITH mysql_native_password BY 'root';
flush privileges;
8、然后在 mysql/conf文件下,新建配置文件,增加我们需要同步的数据库
先退出Mysql容器
mysql> exit
Bye
root@e54909746c9a:/# exit
exit
[root@VM-4-12-centos mysql]# vim conf/my.cnf
[mysqld]
skip-name-resolve
character_set_server=utf8
datadir=/opt/mysql/data
server-id=1000
log-bin=/var/lib/mysql/mysql-bin
#数据库的名称
binlog-do-db=test
重启Mysql容器
[root@VM-4-12-centos mysql]# docker restart mysql
9、这里注意。有个巨大的坑,Docker重启Mysql 8.0后,有可能会导致Mysql 普通用户和新建test数据库丢失,所以,需要重新执行6、7步骤
查看运行的容器和Mysql日志,没有问题
[root@VM-4-12-centos mysql]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e54909746c9a mysql:8.0.29 "docker-entrypoint.s…" 21 minutes ago Up 58 seconds 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
[root@VM-4-12-centos mysql]# docker logs mysql
10、接下来是安装Canal,在安装之前,我们需要了解一下,Canal能够帮助我们做什么,它能够伪装成一个mysql的从节点,然后把数据"偷偷地"发送给Redis,完成数据同步
11、另外,我们还需要创建一个网络,将MySQL、Canal、Redis放到同一个Docker网络中
[root@VM-4-12-centos mysql]# docker network create MysqlConnectRedis
12、然后,让mysql加入这个网络:
[root@VM-4-12-centos mysql]# docker network connect MysqlConnectRedis mysql
13、然后开始我们正式安装Canal,到Docker图书馆找到最新版本,Canal 版本选择
[root@VM-4-12-centos mysql]# docker pull canal/canal-server:v1.1.6
14、云服务器开放端口11111,启动Canal,然后老哥需要解释一下,这些命令的意思,注意:mysql:3306 等同于我们Linux的连接IP,Docker内部连接,可以直接使用容器的名字当做 IP
-e canal.destinations=test \ 你要连接的数据库
-e canal.instance.master.address=你的公网IP地址:3306 \
–network MysqlConnectRedis \ 你要连接的Docker网络
docker run -p 11111:11111 --name canal \
-e canal.destinations=test \
-e canal.instance.master.address=你的公网IP地址:3306 \
-e canal.instance.dbUsername=Jessica \
-e canal.instance.dbPassword=root \
-e canal.instance.connectionCharset=UTF-8 \
-e canal.instance.tsdb.enable=true \
-e canal.instance.gtidon=false \
-e canal.instance.filter.regex=test\\..* \
--network MysqlConnectRedis \
-d canal/canal-server:v1.1.6
15、查看Docker 运行的容器
[root@VM-4-12-centos mysql]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ab533135c42 canal/canal-server:v1.1.6 "/alidata/bin/main.s…" 7 seconds ago Up 5 seconds 9100/tcp, 11110/tcp, 11112/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp
75ececbfe672 mysql:8.0.29 "docker-entrypoint.s…" 32 minutes ago Up 29 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
16、启动是成功了,但是我怎么知道有没有连接到Mysql呢? 我们可以进入Canal容器查看,出现Successfuly,说明成功连接到Mysql
[root@VM-4-12-centos mysql]# docker exec -it canal bash
[root@dbd46e6efaa3 admin]# cat canal-server/logs/test/test.log
17、无需操作,如果我们的binlog日志有点多了,我们可以先停止Canal,然后连接上Mysql,到里面重置binlog初始位置,最后重启Canal
[root@VM-4-12-centos mysql]# docker exec -it mysql bash
mysql> reset master ;
mysql> show master status ;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 | 157 | test | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
[root@VM-4-12-centos mysql]# docker restart canal
18、然后我们安装一下Redis,Docker 安装Redis教程
19、最后,我们用java代码去实现Mysql 8.0 到Redis的同步
canal的依赖
top.javatool
canal-spring-boot-starter
1.2.1-RELEASE
pom.xml
org.springframework.boot
spring-boot-starter-data-redis
org.apache.commons
commons-pool2
2.8.1
org.springframework.boot
spring-boot-starter-web
mysql
mysql-connector-java
runtime
8.0.27
org.projectlombok
lombok
true
cn.hutool
hutool-all
5.7.17
com.baomidou
mybatis-plus-boot-starter
3.4.3
org.redisson
redisson
3.13.6
com.github.ben-manes.caffeine
caffeine
2.8.0
top.javatool
canal-spring-boot-starter
1.2.1-RELEASE
org.apache.maven.plugins
maven-compiler-plugin
3.6.0
1.8
application.yml
server:
port: 80
spring:
application:
name: redis-demo
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://101.34.7.236:3306/test?useSSL=false&serverTimezone=UTC
username: root
password: root
redis:
host: 101.34.7.236
port: 6379
lettuce:
pool:
max-active: 10
max-idle: 10
min-idle: 1
time-between-eviction-runs: 10s
jackson:
default-property-inclusion: non_null # JSON处理时忽略非空字段
mybatis-plus:
type-aliases-package: com.dowhere.pojo # 别名扫描包
logging:
level:
com.dowhere: debug
canal:
destination: test
server: 101.34.7.236:11111
pojo
@Data
@TableName("tb_user")
public class User {
private Long id;
private String userName;
}
config
@Configuration
public class CacheConfig {
@Bean
public Cache getCache(){
return Caffeine.newBuilder().
expireAfterWrite(Duration.ofSeconds(15))
.build();
}
}
处理类
import cn.hutool.json.JSONUtil;
import com.dowhere.pojo.User;
import com.github.benmanes.caffeine.cache.Cache;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import top.javatool.canal.client.annotation.CanalTable;
import top.javatool.canal.client.handler.EntryHandler;
/**
* @author: Jessica
* @create: 2022-06-16 13:47
* @desc:
**/
@CanalTable("tb_user")
@Component
public class UserHandler implements EntryHandler {
/**
* Redis 缓存
*/
@Autowired
private StringRedisTemplate redisTemplate;
/**
* Jvm缓存
*/
@Autowired
private Cache userCache;
@Override
public void insert(User user) {
userCache.put(user.getId(),user);
this.saveUser(user);
}
@Override
public void update(User before, User after) {
userCache.put(after.getId(),after);
this.saveUser(after);
}
@Override
public void delete(User user) {
userCache.invalidate(user.getId());
this.deleteUserById(user.getId());
}
public void saveUser(User user){
redisTemplate.opsForValue().set("user:id:"+user.getId(),JSONUtil.toJsonStr(user));
}
public void deleteUserById(Long id){
redisTemplate.delete("user:id:"+id);
}
}
20、然后我们在Mysql容器,新建一张表
[root@VM-4-12-centos mysql]# docker exec -it mysql bash
root@e54909746c9a:/# mysql -uroot -p
mysql> use test
mysql> CREATE TABLE `tb_user` (
-> `id` int NOT NULL AUTO_INCREMENT,
-> `userName` varchar(255) DEFAULT NULL,
-> PRIMARY KEY (`id`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
21、启动IDEA,然后我们插入3条数据
mysql> insert into tb_user(id,username) values (null,'Jessica');
mysql> insert into tb_user(id,username) values (null,'Jack');
mysql> insert into tb_user(id,username) values (null,'tom');
22、查看IDEA控制台,发现消息同步过来了
23、查看Redis,没有任何问题
mysql> delete from tb_user where id=3;
Query OK, 1 row affected (0.00 sec)