CentOS7安装logstash-6.1.1

logstash安装

下载rpm包

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.rpm

安装：

rpm -ivh logstash-6.1.1.rpm

安装完成后，使用whereis logstash查看文件安装位置：

• 配置文件目录/etc/logstash
• 安装主目录/usr/share/logstash

ruby安装及gem源配置

由于logstash是用ruby语言开发，因此需要预装环境。¹
安装rvm：

gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
\curl -sSL https://get.rvm.io | bash -s stable

安装完成后会出现如下日志：

Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.
  * Please do NOT forget to add your users to the rvm group.
     The installer no longer auto-adds root or users to the rvm group. Admins must do this.
     Also, please note that group memberships are ONLY evaluated at login time.
     This means that users must log out then back in before group membership takes effect!
Thanks for installing RVM 
Please consider donating to our open collective to help us maintain RVM.

根据提示执行：

source /etc/profile.d/rvm.sh

列出已知Ruby版本：

rvm list known

# 2.4.6
rvm install 2.4.6

修改rubygem为国内源：

gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
gem sources -l
# output: https://gems.ruby-china.com
# 确保只有 gems.ruby-china.com

修改logstash配置文件

sudo vim /usr/share/logstash/Gemfile

#将source修改为https://gems.ruby-china.com/
source "https://gems.ruby-china.com/"

测试

logstash预装了很多插件，可以使用命令/usr/share/logstash/bin/logstash-plugin list --installed --verbose查看，这里使用logstash-input-jdbc (4.3.2)测试。

添加mysql连接jar包

wget http://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar -P /usr/share/logstash

编写配置文件

vim logstash-input-jdbc-mysql.conf 

input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.46.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://xx.xx.xx.xx:3306/youwant_dev?autoReconnect=true&autoReconnectForPools=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false"
    jdbc_user => "swb@test"
    jdbc_password => "Swb@test!"

    # sql 语句文件
    statement => "SELECT * from t_operation_log where  operation_time > :sql_last_value"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    type => "jdbc"
    tracking_column => "operation_time"
    # 不使用全小写，否则会造成字段映射不上的问题
    lowercase_column_names => false
    use_column_value => false
    # 设置监听间隔  各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
    schedule => "* * * * *"
        # 设置时区
    jdbc_default_timezone =>"Asia/Shanghai"
  }
}

output {
  stdout {
    codec => json_lines
  }
  elasticsearch {
    hosts=> "localhost:9200"
    index => "t_operation_log"
    document_type => "swb"
    # 表示取mysql表中的id,可防止因时区未设置导致的重复数据录入
    document_id  => "%{id}"
  }
}

运行

# 测试配置文件正确性
/usr/share/logstash/bin/logstash -t -f /usr/share/logstash/logstash-input-jdbc-mysql.conf

# 配置文件正确则可执行
/usr/share/logstash/bin/logstash -f /usr/share/logstash/logstash-input-jdbc-mysql.conf

# 正常的输出
# Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
# Configuration OK

此处测试使用单数据源，多数据源可以参考使用logstash-input-jdbc同步MySQL数据到Elasticsearch²

查看日志

tail -f /var/log/logstash/logstash-plain.log

问题

有时候多次运行同一配置文件，会出现无法启动的问题，因为Logstash是默认只支持单实例运行的，有两个解决办法，一是关掉已在运行的³，二是开启多实例

参考

• http://gems.ruby-china.com/
• https://ruby-china.org/wiki/rvm-guide
• 使用logstash-6.2.2和logstash-input-jdbc插件实现mysql数据同步到Elasticsearch
• logstash-input-jdbc插件配置细节
• logstash 多实例运行

---

1. 貌似还有用java重写的logstash，有空可以试试为什么用java重写logstash ↩︎

2. 使用logstash-input-jdbc同步MySQL数据到Elasticsearch ↩︎

3. ps -ef | grep logstash，找到占用的PID，然后kill -9 $PID ↩︎