ELK8.4安装配置错误记录

1. kibana启动报错，kibana_system用户认证

[2022-09-06T19:47:01.491-04:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

解决办法：设置kibana_system用户的密码。
[es@goya1 cert]$ elasticsearch-reset-password -u kibana_system -i
warning: ignoring JAVA_HOME=/app/elasticsearch/jdk; using bundled JDK
This tool will reset the password of the [kibana_system] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y
Enter password for [kibana_system]: changeit123
Re-enter password for [kibana_system]: changeit123
Password for the [kibana_system] user successfully reset.

修改,确认kibana.yml文件的配置部分：
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "changeit123"

2. logstash 启动报错

[2022-09-13T22:41:21,452][ERROR][logstash.configmanagement.bootstrapcheck] There are config files (1) in the '/app/logstash/config/tomcat_log_es.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config
ERROR: There are config files (1) in the '/app/logstash/config/tomcat_log_es.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2022-09-13T22:41:21,463][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
    at app.logstash.lib.bootstrap.environment.

(/app/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

解决办法：原因是logstash.yml配置了xpack相关的参数。开启X-Pack Management功能后，启动logstsh的时候就不用再配置logstash.conf文件了，启动的时候也不用再使用-f指定这个文件进行启动了一旦启动了logstash的集中管理，我们就可以直接启动logstash，而不用跟任何的参数。Logstash集中管理，先启动logstash，然后再设置相关配置。

要么关闭x-Pack，要么不用-f的参数启动，相应的配置在logstash.yml里进行。

3， logstash配置的logstash_admin_user修改密码报错

[es@goya1 head-master]$ elasticsearch-reset-password -u logstash_admin_user
warning: ignoring JAVA_HOME=/app/elasticsearch/jdk; using bundled JDK
This tool will reset the password of the [logstash_admin_user] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

ERROR: Failed to reset password for the [logstash_admin_user] user

解决办法：原因是因为ES里没有这个用户，可以尝试通过kibana控制台创建一个角色和logstash_admin_user用户来解决问题。参考：Logstash8.4在Linux系统上的安装以及配置Tomcat日志(ELK安装part2)(未完待续)_yangkei的博客-CSDN博客

4，Metricbeat启动后报错connection refused

{"log.level":"info","@timestamp":"2022-09-16T03:36:06.216-0400","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-16T03:36:06.217-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://192.168.88.5:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-16T03:36:14.259-0400","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.status: error making http request: Get \"http://localhost:5601/api/status\": dial tcp [::1]:5601: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
解决办法：因为http://localhost:5601这个配置是从metricsbeat目录下的kibana.yml文件读取的，因此需要修改localhost为对应的ip地址。

[es@goya1 modules.d]$ pwd
/app/metricbeat/modules.d
[es@goya1 modules.d]$ cat kibana.yml
# Module: kibana
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/main/metricbeat-module-kibana.html

- module: kibana
  #metricsets:
  #  - status
  period: 10s
  hosts: ["192.168.88.5:5601"]
  #basepath: ""
  #username: "user"
  #password: "secret"

5， Metricbeat启动报错

Exiting: loading configs: 3 errors: invalid config: config file ("/app/metricbeat/modules.d/kibana.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/kibana.yml'); invalid config: config file ("/app/metricbeat/modules.d/logstash.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/logstash.yml'); invalid config: config file ("/app/metricbeat/modules.d/system.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/system.yml')
 问题解决：metricbeat文件权限不能给太大，不然启动检测不能他通过，可以修改到750的权限。

6，Metricbeat监控tomcat日志报错

{"log.level":"info","@timestamp":"2022-09-20T10:58:01.529+0800","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://192.168.88.7:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-20T10:58:10.511+0800","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: error making http request: Get \"http://localhost:9600/\": dial tcp [::1]:9600: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-20T10:58:10.512+0800","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node: error making http request: Get \"http://localhost:9600/\": dial tcp 127.0.0.1:9600: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
问题解决：我使用了logstash提取tomcat日志，因此这里需要在Metricsbeat里再配置一下logstash.yml文件。

[es@goya1 modules.d]$ pwd
/app/metricbeat/modules.d
[es@goya1 modules.d]$ grep 9600 *
logstash-xpack.yml.disabled:  hosts: ["localhost:9600"]
logstash.yml:  hosts: ["localhost:9600"]

7，Filebeat启动报错

[es@goya1 filebeat]$ ./filebeat -e -c filebeat.kafka.yml 

{"log.level":"error","@timestamp":"2022-09-26T16:56:43.325+0800","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: 1 error: setting 'filebeat.prospectors' has been removed","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: 1 error: setting 'filebeat.prospectors' has been removed
解决办法：在6.3版本以后，在配置文件中需要把filebeat.prospectors 修改为filebeat.inputs

[es@goya1 filebeat]$ cat filebeat.kafka.yml
filebeat.prospectors:
  - type: log
    enabled: true
    paths:
        - /app/tomcat/logs/tomcat_access_json.2022-09-23.log

[es@goya1 filebeat]$ cat filebeat.kafka.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
        - /app/tomcat/logs/tomcat_access_json.2022-09-23.log