CentOS7.9 安装Logstash 8.1.0

目录

环境

一、Logstash

1.安装rpm

2.准备log文件

3.配置

4.测试

5.启动

 6.配置kibana

二、FileBeat

遇到的问题

1.[ERROR] 2022-05-31 10:36:40.687 [Agent thread] sourceloader - No configuration found in the configured sources.

2.[FATAL] 2022-05-31 10:41:55.769 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)

3.WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

4. [2022-05-31T10:57:48,997][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

5.[2022-05-31T11:05:09,544][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://192.168.50.101:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://192.168.50.101:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

---

环境

书接上文，环境还是之前的CentOS7.9，目前已经安装好Elasticsearch+Kibana+MetricBeat都是8.1.0

---

一、Logstash

1.安装rpm

rpm -ivh logstash-8.1.0-x86_64.rpm

效果

设为开机启动

systemctl enable logstash.service

2.准备log文件

vi /userap/log/test.log

#这里是用“ - ”来分隔不同项，也可以用别的，比如“|”
Date: 2022-05-31 10:00:00,751 - LogLevel:INFO  - Method:SendAndReceiveLog - UserName: DESKTOP-JFMPLUM\admin - Message:Send[GetCode]: [UserName]100281[limitFlag]True[ItemID]AAAAA
Date: 2022-05-31 10:00:00,754 - LogLevel:INFO  - Method:SendAndReceiveLog - UserName: DESKTOP-JFMPLUM\admin - Message:Receive[GetCode]: [UserName]100281[limitFlag]True[ItemID]AAAAA[Result]Success
Date: 2022-05-31 10:00:00,754 - LogLevel:DEBUG - Method:SendAndReceiveLog - UserName: DESKTOP-JFMPLUM\admin - Message:Debug[GetCode]: [UserName]100281[limitFlag]True[ItemID]AAAAA[Result]More Info....
Date: 2022-05-31 10:00:01,761 - LogLevel:ERROR - Method:ErrorLog - UserName: DESKTOP-JFMPLUM\admin - Message:Receive[QueryCode]: [UserName]100494[limitFlag]True[ItemID]AABBA[Result] Para Error
Date: 2022-05-31 10:00:02,764 - LogLevel:INFO  - Method:SendAndReceiveLog - UserName: DESKTOP-JFMPLUM\admin - Message:Send[QueryCode]: [UserName]100386[limitFlag]True[ItemID]AAABB

 

3.配置

cd /etc/logstash/conf.d
vi test_log.conf #随便起一个conf

input {
  file {
    path => "/userap/log/*.log" #log路径
    start_position => "beginning"
  }
}

filter {
  grok {
    #patterns_dir => ["./patterns"]
    match => { "message" => "^Date: %{TIMESTAMP_ISO8601:time} - LogLevel:%{DATA:LogLevel} - Method:%{DATA:CodeMethod} - UserName: %{DATA:UserName} - Message:%{DATA:Message}$"}
  }
}

output {
  elasticsearch {
    hosts => ["https://192.168.50.101:9200"]
    index => "testlog" # 如果每天创建一个则可以按照此语法："testlog-%{+YYYY.MM.dd}"
    user => "fbguo"
    password => "xxxxxx"
    cacert => "/etc/elasticsearch/certs/http_ca.crt"
  }
}
~

主要就是三段式结构：

input：输入，可以是指定path，也可以是使用filebeat

filter：筛选器，可以对log进行筛选，我用的是grok，类似于正则吧，可以使用kibana的grok debugger进行调试

还有别的语法可以参考下面这篇文章：

详解 logstash.conf_戴国进的博客-CSDN博客_logstash.conf 

 

output：这个就是输出，一般也就是配置es server，cacert就是http证书，如果换机器使用logstash需要将该证书copy过去，也许可以像kibana一样用那串fingerprinter，但是我目前没看到怎么用，index就是指定的索引名称，后续在es server和kibana中都会看到

4.测试

/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/test_log.conf --path.settings /etc/logstash

最后有出现 Config Validation Result: OK.就代表OK了

5.启动

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test_log.conf --path.settings /etc/logstash

就是把测试的-t 去掉，如果要后台启动的话在最后加上&即可

前台启动效果如下：

 

 6.配置kibana

选择Management->Kibana->数据视图

 创建新的视图，我这里已经创建过了

 

然后到Discover里选择刚刚创建的视图就可以看见

 

这样logstash->es->kibana就完成了

---

二、FileBeat

FileBeat其实就是在logstash前面再加一层，留在下篇吧，鸽了

---

遇到的问题

1.[ERROR] 2022-05-31 10:36:40.687 [Agent thread] sourceloader - No configuration found in the configured sources.

根源：这个应该是调用bin/logstash命令后面跟的conf文件不存在，建议使用命令时使用全路径

解决方法：

#错误命令
/usr/share/logstash/bin/logstash -f test_log.conf
#改成
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test_log.conf

 

2.[FATAL] 2022-05-31 10:41:55.769 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)

根源：这种报错一般就是告诉你哪一行有问题，像我这个就是犯蠢第一行input输错了

解决方案：就按这个提示来找，这个报错是使用-t进行测试时才会遇到的

3.WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

根源：这个也是-t测试的时候遇到的，具体为什么因此我认为就是没找到对应的logstash.yml

解决方法：我就是按照他错误上提示的，加了--path.settings参数

#错误代码
/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/test_log.conf
#改成
/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/test_log.conf --path.settings /etc/logstash

结果这个WARN就没了

 

4. [2022-05-31T10:57:48,997][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

根源：这个问题好像是没指定pipelines.yml，这个我也没解，用VS的老习惯了不看WARN -_-!!

5.[2022-05-31T11:05:09,544][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://192.168.50.101:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://192.168.50.101:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

根源：这个问题就是连不通elasticsearch

解决方案：在test_log.conf中连接elasticsearch的部分加上如下内容，如果是别的机器开logstash需要吧http_ca.crt文件复制到那台机器上

    user => "fbguo"
    password => "xxxxxx"
    cacert => "/etc/elasticsearch/certs/http_ca.crt"

 

---

 

总结

下次再写一篇filebeat吧，这篇鸽了