【ELK7.4.2】MAC+ES+logstash+MySql同步更新数据

ES版本：7.4.2

logstash版本：7.4.2

MySql版本：8.0.18

Java数据库驱动版本：8.0.18

MySql建表语句如下（帮初学者省点时间）：

    CREATE DATABASE testyy;
    use testyy;
    CREATE TABLE IF NOT EXISTS `runoob_tbl`(
   `runoob_id` INT UNSIGNED AUTO_INCREMENT,
   `runoob_title` VARCHAR(100) NOT NULL,
   `runoob_author` VARCHAR(40) NOT NULL,
   `submission_date` DATE,
   PRIMARY KEY ( `runoob_id` )
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
    INSERT INTO runoob_tbl
    (runoob_title, runoob_author, submission_date)
    VALUES
    ("学习 PHP", "菜鸟教程", NOW());
    INSERT INTO runoob_tbl
    (runoob_title, runoob_author, submission_date)
    VALUES
    ("学习 MySQL", "菜鸟教程", NOW());
    INSERT INTO runoob_tbl
    (runoob_title, runoob_author, submission_date)
    VALUES
    ("JAVA 教程", "RUNOOB.COM", '2016-05-06');
    INSERT INTO runoob_tbl
    (runoob_title, runoob_author, submission_date)
    VALUES
    ("JAVA", "ABC.COM", '2017-05-06');
    INSERT INTO runoob_tbl
    (runoob_title, runoob_author, submission_date)
    VALUES
    ("python", "yyg.COM", '2018-05-06');
     INSERT INTO runoob_tbl
     (runoob_title, runoob_author, submission_date)
    VALUES
    ("vs2010", "nihao.COM", '2019-05-06');
     INSERT INTO runoob_tbl
     (runoob_title, runoob_author, submission_date)
    VALUES
    ("matlab", "阳阳.COM", '2016-2-06');
     INSERT INTO runoob_tbl
     (runoob_title, runoob_author, submission_date)
    VALUES
    ("有一个", "光.COM", '2012-05-06');

在...../logstash-7.4.2/bin/文件夹下建两个文件：1、sqlconfig.conf ；2、jdbc.sql。

1、sqlconfig.conf

input {
  stdin {}
  jdbc {
    jdbc_driver_library => "/##youpath##/logstash-7.4.2/logstash-core/lib/jars/mysql-connector-java-8.0.18.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/testyy"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    statement_filepath => "/##youpath##/logstash-7.4.2/bin/jdbc.sql"
    type => "testyy"
  }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
    elasticsearch {
        index => "testyyy"
        document_type => "%{type}"
        hosts => "localhost:9200"
        document_id => "%{runoob_id}"
    }
    stdout {
        codec => json_lines
    }
}

 2、jdbc.sql

SELECT runoob_id,runoob_title,runoob_author,submission_date from runoob_tbl

在 ...../logstash-7.4.2/bin/文件夹下运行：./logstash -f sqlconfig.conf

此时MySql数据库中的数据读入到命令框和ES中，如下为结果图：

结果图1:

结果图2:

需要注意的问题：

【1】MySql 8.0以上版本需要注意：The name of the class that implements java.sql.Driver in MySQL Connector/J has changed from com.mysql.jdbc.Driver to com.mysql.cj.jdbc.Driver. The old class name has been deprecated.

【2】将驱动即mysql-connector-java-8.0.18.jar拷贝到 logstash目录\logstash-core\lib\jars 下；

【3】JAVA数据库驱动下载地址。

附上多表更新代码：

input {
  stdin {}
  jdbc {
    jdbc_driver_library => "/youpath/logstash-7.4.2/logstash-core/lib/jars/mysql-connector-java-8.0.18.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/testyy"
    jdbc_user => "root"
    jdbc_password => "password"
    record_last_run =>true
    lowercase_column_names => false
    use_column_value => true
    tracking_column => "runoob_id"
    tracking_column_type =>"numeric"
    last_run_metadata_path =>"/youpath/logstash-7.4.2/run_metadata/my_info"
    clean_run =>"false"
    schedule => "* * * * *"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    statement_filepath => "/youpath/logstash-7.4.2/bin/jdbc.sql"
    type => "testyy"
  }
  jdbc{
    type => "testyy_article"
    jdbc_driver_library => "/youpath/logstash-7.4.2/logstash-core/lib/jars/mysql-connector-java-8.0.18.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/testyy"
    jdbc_user => "root"
    jdbc_password => "password"
    record_last_run =>true
    lowercase_column_names =>false
    use_column_value =>true
    tracking_column =>"update_time"
    tracking_column_type =>"timestamp"
    last_run_metadata_path =>"/youpath/logstash-7.4.2/run_metadata/my_info_article"
    clean_run =>"false"
    schedule => "* * * * *"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    statement_filepath => "/youpath/logstash-7.4.2/bin/jdbc1.sql"
  }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
    if [type] == "testyy" {
        elasticsearch {
        index => "testyyy"
        document_type => "%{type}"
        hosts => "localhost:9200"
        document_id => "%{runoob_id}"
    }
    }
    if [type] == "testyy_article"{
        elasticsearch {
        index => "testyy_art"
        document_type => "%{type}"
        hosts => "localhost:9200"
        document_id => "%{id}"
    }

    }
    
    stdout {
        codec => json_lines
    }
}

需要注意的是：ES7.0以上版本严格按照单索引单type，如果MySql数据表中的tables属性不一致是无法索引到一个index中的。

参考链接：

https://www.jianshu.com/p/e359880541b8