elasticsearch 多表同步
使用logstash-6.2.4和logstash-input-jdbc插件实现mysql数据同步到Elasticsearch
环境:
Linux
Elasticsearch-6.2.4
logstash-6.2.4 安装ruby环境 gem-2.7.6
logstash-input-jdbc-4.3.3
安装:
1.
下载6.2.4版本的,tar.GZ架包
上传至Linux服务器安装目录下，这里是/opt/elasticsearch/logstash-6.2.2
tar -xvf logstash-6.2.4.tar.gz
测试是否安装成功:
在logstash的bin目录下运行
./logstash -e ‘input { stdin { } } output { stdout {} }’
2.
判断是否安装gem
gem –v
提示非法命令则未安装
安装gem
sudo yum install gem
ruby仓库镜像替换淘宝
gem sources --add https://ruby.taobao.org/ --remove https://rubygems.org/
3.
修改Gemfile的数据源地址
logstash 文件下的配置Gemfile，修改 source 的值 为： “https://ruby.taobao.org”
修改 Gemfile.lock配置，找到 remote 修改它的值为： https://ruby.taobao.org
4.
到logstash的bin目录下运行下面的命令，查看可用的插件
./logstash-plugin list --verbose
安装
./logstash-plugin install logstash-input-jdbc
注意:
.conf,.sql要放到logstash-6.2.4\bin下面
修改Gemfile里面sources为https://ruby.taobao.org/(注意前面只有一个空格否则配置失败)
修改Gemfile.lock里面remote 为https://ruby.taobao.org/(注意前面只有一个空格否则配置失败)
./logstash -e "input { stdin { } } output { stdout {} }" 查看logstash是否成功
./logstash-plugin install logstash-input-jdbc 安装logstash-input-jdbc插件
或者sudo bin/logstash-plugin install logstash-input-jdbc 安装logstash-input-jdbc插件
如果报system错误 则给logstash文件赋权限chmod 777
bin/logstash -f mysql.conf (Linux).执行多表同步
如果报错误则bin/logstash -f mysql.conf --path.data=/root/
或者./logstash -f config-mysql --path.data=/var/log
多表导入同步Elasticsearch:
.conf文件:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://dev.yonyouccs.com:3001/test"
jdbc_user => "root"
jdbc_password => "root"
jdbc_driver_library => "D:/software/logstash-6.2.2/logstash-6.2.2/mysql/mysql-connector-java-5.1.40.jar"//引用mysql架包
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "D:/software/logstash-6.2.2/logstash-6.2.2/mysql/jdbc.sql"
schedule => "* * * * *"
type => "user"//多表的时候要指定类型
}
jdbc {
jdbc_connection_string => "jdbc:mysql://dev.yonyouccs.com:3001/test"
jdbc_user => "root"
jdbc_password => "root"
jdbc_driver_library => "D:/software/logstash-6.2.2/logstash-6.2.2/mysql/mysql-connector-java-5.1.40.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
parameters => {"number" => "200"}
statement_filepath => "D:/software/logstash-6.2.2/logstash-6.2.2/mysql/jdbc-seckill.sql"//指定一个sql文件
schedule => "* * * * *"
type => "kill"
}
}
filter {//监听
json {
source => "message"
remove_field => ["message"]
}
}
output {
if[type] == "user" {//根据类型判断
elasticsearch {
hosts => ["localhost:9200"]//导入哪个es
index => "index_user"//设置其索引
document_id => "%{id}"//设置其ID一般为表内ID
}
}
if[type] == "kill" {
elasticsearch {
hosts => ["localhost:9200"]
index => "index_kill"
document_id => "%{id}"
}
}
stdout {
codec => json_lines
}
}
POST /index_js/_search 查询是否同步
