Mysql数据准实时同步到Elasticsearch

一. 摘要

Elasticsearch作为大数据场景下的搜索和分析的引擎，广泛的应用于实时数据的分析场景。本文介绍如何通过logstash_input_jdbc插件将mysql数据准实时的同步于Elasticsearch。

二. 前期准备

• 安装Elasticsearch（本文采用的版本为2.4.6）
• 安装Mysql
• 安装Logstash（本文采用的版本为6.1.1）

由于本文主要介绍如何Mysql数据向Elasticsearch同步实操步骤，前期准备工作请参阅其他文章。

三. 实验步骤

3.1 安装logstash_input_jdbc插件

进入logstash bin目录，执行./logstash-plugin list --group input，查看是否已安装logstash_input_jdbc插件

chenyaleideMacBook-Pro:bin lay$ ./logstash-plugin list --group input
logstash-input-beats
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jdbc
logstash-input-kafka
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix

未安装插件的情况下执行./logstash-plugin install logstash-input-jdbc

3.2 安装logstash-output-elasticsearch插件

进入logstash bin目录，执行./logstash-plugin list --group output，查看是否已安装logstash_output_elasticsearch插件

chenyaleideMacBook-Pro:bin lay$ ./logstash-plugin list --group output
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs

未安装插件的情况下执行./logstash-plugin install logstash_output_elasticsearch

3.3 创建mysql测试数据

本文为方便起见，创建teacher与student两张表，旨在展示如何同时同步多张表数据。

create table teacher (         
id varchar(10),     
first_name varchar(20),     
last_name varchar(20),     
age int(10),     
about varchar(100),     
interests varchar(100),     
updatetime timestamp null default current_timestamp on update current_timestamp );

create table student (         
id varchar(10),     
first_name varchar(20),     
last_name varchar(20),     
age int(10),     
about varchar(100),     
interests varchar(100),     
updatetime timestamp null default current_timestamp on update current_timestamp );

3.4 创建logstash作业文件

在/bin目录下新建jdbc.conf文件，当然你可以在其他目录创建。
输入以下内容：

input {
    jdbc {
      # mysql jdbc connection string to our backup databse
      jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/estest"
      # the user we wish to excute our statement as
      jdbc_user => "root"
      jdbc_password => "123456"
      # the path to our downloaded jdbc driver
      jdbc_driver_library => "/Users/lay/.m2/repository/mysql/mysql-connector-java/5.1.37/mysql-connector-java-5.1.37.jar"
      # the name of the driver class for mysql
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      jdbc_default_timezone => "Asia/Shanghai"
      #可以将sql单独提成一个文件，写法如下
      #statement_filepath => "jdbc.sql"
      statement => "select * from teacher where update_time > :sql_last_value"
      use_column_value => false
      clean_run => true
      last_run_metadata_path => "./teacher_last_run"
      schedule => "* * * * *"
      type => "jdbc"
      #通用属性tags
      tags => ["teacher"]
   }
   jdbc {
      # mysql jdbc connection string to our backup databse
      jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/estest"
      # the user we wish to excute our statement as
      jdbc_user => "root"
      jdbc_password => "123456"
      # the path to our downloaded jdbc driver
      jdbc_driver_library => "/Users/lay/.m2/repository/mysql/mysql-connector-java/5.1.37/mysql-connector-java-5.1.37.jar"
      # the name of the driver class for mysql
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      jdbc_default_timezone => "Asia/Shanghai"
      #statement_filepath => "jdbc.sql"
      statement => "select * from student where update_time > :sql_last_value"
      use_column_value => false
      clean_run => true
      last_run_metadata_path => "./student_last_run"
      schedule => "* * * * *"
      type => "jdbc"
      tags => ["student"]
   }
}

output {
    if "teacher" in [tags]{

      elasticsearch{
        hosts => "127.0.0.1:9200"
        index => "teacher"
        document_id => "%{id}"
      }
    }else if "student" in [tags]{
      elasticsearch{
        hosts => "127.0.0.1:9200"
        index => "student"
        document_id => "%{id}"
      }
    }
    stdout {
        codec => json_lines
    }
}

3.5 同步数据

启动logstash服务，执行如下命令：

logstash -f jdbc.conf

后台启动命令：

nohup ./logstash -f jdbc.conf  > logstash.log &

数据库插入数据：

INSERT INTO student(id,first_name,last_name,age,about,interests) VALUES('001','John','Smith', 25, 'I love to go rock climbing','[ "sports", "music" ]'); 
INSERT INTO student(id,first_name,last_name,age,about,interests) VALUES('002','Jane','Smith', 32, 'I like to collect rock albums','[ "music" ]'); 
INSERT INTO student(id,first_name,last_name,age,about,interests) VALUES('003','Douglas','Fir', 35, 'I like to build cabinets','[ "forestry" ]'); 

INSERT INTO teacher(id,first_name,last_name,age,about,interests) VALUES('001','John','Smith', 25, 'I love to go rock climbing','[ "sports", "music" ]'); 
INSERT INTO teacher(id,first_name,last_name,age,about,interests) VALUES('002','Jane','Smith', 32, 'I like to collect rock albums','[ "music" ]'); 
INSERT INTO teacher(id,first_name,last_name,age,about,interests) VALUES('003','Douglas','Fir', 35, 'I like to build cabinets','[ "forestry" ]'); 

3.6 结果验证

查询Elasticsearch中是否存在mysql的插入数据，至此数据同步完成。