Logstash安装教程

安装

1. 官网下载压缩包
   https://www.elastic.co/cn/products/elasticsearch
2. 解压

tar -xzf logstash-7.5.1.tar.gz

3. 新建logstash.conf文件，用于配置输入、过滤规则、输出信息

[root@m5315392269-5 logstash]# cd logstash-7.5.1/
[root@m5315392269-5 logstash-7.5.1]# touch logstash.conf

#logstash.conf文件中的内容
input {
  beats {
    type => "log"
    port => "5044" #开始本机的5044端口，监听
  }
}

filter{
  mutate{
    split=>["message","|"]
      add_field => {
        "log_date" => "%{[message][0]}"
      }
      add_field => {
        "log_level" => "%{[message][1]}"
      }
      add_field => {
        "log_thread" => "%{[message][2]}"
      }
      add_field => {
        "log_class" => "%{[message][3]}"
      }
      add_field => {
        "log_content" => "%{[message][4]}"
      }

      remove_field => ["message"]
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["10.237.79.147:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"
  }
}

4. 启动logstash

[root@m5315392269-5 logstash-7.5.1]# ./bin/logstash -f logstash.conf

Logstash配置文件介绍

1. 设置输入
2. 设置过滤规则
   add_field:在事件中加入一个filed
   tags：添加一个任意的数字作为当前事件的标签，这个标签对我们后续的工作会有帮助
   参考教程链接：关于logstash导入es库的配置文件的一些理解
   
   mutate过滤器：
   mutate插件用于集成至Logstash上过滤日志数据，即在数据写入ES前进行数据清洗和数据格式化。
   split函数（mutate中常用函数）：将包含特定分隔符的字符串拆分成数组
   参考教程链接：ELK 系列六、logstash 的mutate过滤器功能介绍
3. 设置输出

遇到问题

（1）LogStash 错误：Logstash could not be started because there is already another instance usin

Sending Logstash logs to /usr/local/logstash/logstash-6.5.0/logs which is now configured via log4j2.properties
[2018-11-20T12:23:45,931][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-20T12:23:46,088][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2018-11-20T12:23:46,130][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

原因及解决办法
之前运行的instance有缓冲，保存在path.data里面有.lock文件（path默认指logstash解压后的目录），删除掉即可。

data$ ls -alh
总用量 20K
drwxr-sr-x  4 tingshuo staff 4.0K 11月 20 11:42 .
drwxr-sr-x 14 tingshuo staff 4.0K 11月 20 11:42 ..
drwxr-sr-x  2 tingshuo staff 4.0K 11月 20 11:42 dead_letter_queue
-rw-r--r--  1 tingshuo staff    0 11月 20 11:42 .lock
drwxr-sr-x  2 tingshuo staff 4.0K 11月 20 11:42 queue
-rw-r--r--  1 tingshuo staff   36 11月 20 11:42 uuid

#删除并重新启动
rm -rf .lock