docker安装mysql、logstash同步数据到es

docker安装mysql、logstash同步数据到es

一、安装mysql

• 1.拉取镜像

docker pull mysql:5.7.26

• 2.启动

docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d -i -p 3306:3306 mysql:5.7.26

• 3.授予远程访问权限

docker exec -it mysql bash
mysql -uroot -p123456

use mysql;
select host,user from user;
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
flush privileges;

• 4.开启mysql binlog

docker exec -it mysql bash
apt-get update
apt-get install vim
vim etc/mysql/mysql.conf.d/mysqld.cnf

在mysqld配置项下添加如下几行内容

log-bin=/var/lib/mysql/mysql-bin
server-id=1
binlog-format=ROW

mysqld.cnf

重启mysql。

• 5.查看binlog是否开启，数据库客户端执行如下命令

show variables like '%log_bin%';

log_bin

• 6.创建mysql账户，用于让canal同步数据

use mysql;
create user canal identified by 'canal';
grant select,replication slave,replication client on *.* to 'canal'@'%';
flush privileges;

• 7.创建测试用数据库

create database test;
use test;
CREATE TABLE `user` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL,
  `age` int(11) NOT NULL,
  `createtime` datetime NOT NULL,
  `lastupdatetime` datetime NOT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
INSERT INTO `test`.`user`(`name`, `age`, `createtime`, `lastupdatetime`) VALUES ('jack', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');
INSERT INTO `test`.`user`( `name`, `age`, `createtime`, `lastupdatetime`) VALUES ( 'William', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');

二、安装logstash

• 1.拉取镜像

docker pull logstash:7.17.0

• 2.启动

docker run -d -p 5044:5044 -p 9600:9600 -it --name logstash  logstash:7.17.0

• 3.修改配置文件

docker exec -it -u root logstash bash
apt-get update
apt-get install vim

修改logstash.yml

vim config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.url: http://192.168.0.103:9200

修改log4j2.properties

vim config/log4j2.properties

在末尾增加如下两项

logger.elasticsearchoutput.name = logstash.outputs.elasticsearch
logger.elasticsearchoutput.level = debug

vim config/pipelines.yml

相关值修改如下

- pipeline.id: logstash  
path.config: "/usr/share/logstash/config/*.conf"  
pipeline.workers: 3

修改mysql.conf，没有的话会创建

vim config/mysql.conf

将如下内容粘贴进去

input {
   stdin {
   }
   jdbc {
     jdbc_connection_string => "jdbc:mysql://192.168.0.103:3306/test"
     jdbc_user => "canal"
     jdbc_password => "canal"
     jdbc_driver_library => "/usr/share/logstash/config/mysql-connector-java-5.1.27-bin.jar" # 这个jar文件需要下载
     jdbc_driver_class => "com.mysql.jdbc.Driver"
     statement => "SELECT * FROM user"
     type => "project"
   }
 }
 
 filter {
   json {
     source => "message"
     remove_field => ["message"]
   }
 }
 
 output {
   elasticsearch {
     hosts => "http://192.168.0.103:9200"
     index => "project"
     document_id => "%{id}"
   }
 
   stdout {
     codec => json_lines
   }
 }

mysql-connector-java下载

apt-get install wget
wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.27.tar.gz
tar zxvf mysql-connector-java-5.1.27.tar.gz

将解压后的.jar文件放到config目录下即可。

重启logstash。

• 4.执行加载配置项

docker exec -it logstash bash
bin/logstash -f config/mysql.conf

kibana

注：Logstash could not be started because there is already another instance usining...如果报这个错误，就进入logstash终端，找到data目录，ls -all 展示所有文件目录，把里面的.lock文件删除即可。

• 5.测试

查询

命中结果

其他文章请看：
docker安装kibana
docker安装elasticsearch