ELK Stack分布式日志分析

ELK Stack 包括 Elasticsearch、Logstash、Kibana 和 Beats 。能够安全可靠地获取任何来源、任何格式的数据，然后实时地对数据进行搜索、分析和可视化。

Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，能够解决不断涌现出的各种用例。作为 Elastic Stack 的核心，它集中存储您的数据，帮助您发现意料之中以及意料之外的情况。

Kibana 可以对自己的 Elasticsearch 进行可视化，还可以在 Elastic Stack 中进行导航，这样您便可以进行各种操作了，从跟踪查询负载，到理解请求如何流经您的整个应用，都能轻松完成。

Logstash 是免费且开放的服务器端数据处理管道，能够从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中。

Beats 是一个免费且开放的平台，集合了多种单一用途数据采集器。它们从成百上千或成千上万台机器和系统向 Logstash 或 Elasticsearch 发送数据。包括 Filebeat、Metricbeat、Packetbeat、Winlogbeat、Auditbeat、Heartbeat、Functionbeat。

流程

摘自网络

环境及版本

CentOS 7

jdk1.8.0

ELK Stack 7.0

官网地址 https://www.elastic.co/cn/elastic-stack
以下操作，都以非 root 用户操作。默认不能使用root用户。

Elasticsearch

下载elasticsearch-7.7.1-linux-x86_64.tar.gz

#解压 elasticsearch
[vagrant@localhost ~]$ tar -xvf elasticsearch-7.7.1-linux-x86_64.tar.gz 
#目录结构
[vagrant@localhost elasticsearch-7.7.1]$ ls
bin  config  jdk  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.asciidoc
#配置文件目录
[vagrant@localhost elasticsearch-7.7.1]$ cd config/
[vagrant@localhost config]$ ls
elasticsearch.yml  jvm.options  jvm.options.d  log4j2.properties  role_mapping.yml  roles.yml  users  users_roles
#修改配置文件
[vagrant@localhost config]$ vi elasticsearch.yml

配置 jvm.options 中的内存

[vagrant@localhost config]$ vi jvm.options
#仅需修改-Xms 和 -Xmx 
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m

示例 elasticsearch.yml 配置 https://www.jianshu.com/p/8d5dff491e9f（篇幅问题，配置文件分开编写）

启动Elasticsearch

# -d 后台启动 
[vagrant@localhost elasticsearch-7.7.1]$ ./bin/elasticsearch -d
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_211-amd64/jre] does not meet this requirement
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_211-amd64/jre] does not meet this requirement
#查看启动日志 
[vagrant@localhost elasticsearch-7.7.1]$ view logs/my-application.log 
[2020-06-18T17:04:37,992][INFO ][o.e.n.Node               ] [node-1] starting ...
[2020-06-18T17:04:38,308][INFO ][o.e.t.TransportService   ] [node-1] publish_address {192.168.0.140:9300}, bound_addresses {[::]:9300}
[2020-06-18T17:04:39,314][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-06-18T17:04:39,373][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [KPfQ2tYyQeOGtgMLcGnlhA]
[2020-06-18T17:04:39,906][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{_TNGd5GqQ1uEeu5N-h5rWg}{JODG8Ie1TlasxO8SaJZdYA}{192.168.0.140}{192.168.0.140:9300}{dilmrt}{ml.machine_memory=1927471104, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 48, delta: master node changed {previous [], current [{node-1}{_TNGd5GqQ1uEeu5N-h5rWg}{JODG8Ie1TlasxO8SaJZdYA}{192.168.0.140}{192.168.0.140:9300}{dilmrt}{ml.machine_memory=1927471104, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2020-06-18T17:04:40,149][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{_TNGd5GqQ1uEeu5N-h5rWg}{JODG8Ie1TlasxO8SaJZdYA}{192.168.0.140}{192.168.0.140:9300}{dilmrt}{ml.machine_memory=1927471104, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 4, version: 48, reason: Publication{term=4, version=48}
[2020-06-18T17:04:40,460][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {192.168.0.140:9200}, bound_addresses {[::]:9200}
[2020-06-18T17:04:40,461][INFO ][o.e.n.Node               ] [node-1] started
[2020-06-18T17:04:41,535][INFO ][o.e.l.LicenseService     ] [node-1] license [a7d9f2e4-e934-4fa9-b11c-ada9ddd761cb] mode [basic] - valid
[2020-06-18T17:04:41,548][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled
[2020-06-18T17:04:41,576][INFO ][o.e.g.GatewayService     ] [node-1] recovered [2] indices into cluster_state
[2020-06-18T17:04:43,550][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-test-2020.06.18][0]]]).
#检查是否正常访问
[vagrant@localhost elasticsearch-7.7.1]$ curl http://192.168.0.140:9200
{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "KPfQ2tYyQeOGtgMLcGnlhA",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Elasticsearch启动成功后，Kibana和Logstash启动没有先后顺序。

Kibana

下载 kibana-7.7.1-linux-x86_64.tar.gz

#解压 kibana
[vagrant@localhost ~]$ tar -xvf kibana-7.7.1-linux-x86_64.tar.gz 
#目录结构
[vagrant@localhost kibana-7.7.1]$ ls
bin  built_assets  config  data  LICENSE.txt  node  node_modules  nohup.out  NOTICE.txt  optimize  package.json  plugins  README.txt  src  webpackShims  x-pack
#修改配置文件
[vagrant@localhost kibana-7.7.1]$ vi config/kibana.yml

示例kibana.yml配置 https://www.jianshu.com/p/ce5105beeb67

启动 Kibana

#后台启动，记录nohup 日志
[vagrant@localhost kibana-7.7.1]$ nohup ./bin/kibana &
[1] 18721
[vagrant@localhost kibana-7.7.1]$ nohup: ignoring input and appending output to ‘nohup.out’
#查看nohup.out日志
[vagrant@localhost kibana-7.7.1]$ tail -f -n10 nohup.out 
{"type":"log","@timestamp":"2020-06-18T17:35:41Z","tags":["warning","reporting"],"pid":18721,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2020-06-18T17:35:50Z","tags":["warning","reporting"],"pid":18721,"message":"正在为 xpack.reporting.encryptionKey 生成随机密钥。要防止待处理报告在重新启动时失败，请在 kibana.yml 中设置 xpack.reporting.encryptionKey"}
{"type":"log","@timestamp":"2020-06-18T17:35:50Z","tags":["status","plugin:[email protected]","info"],"pid":18721,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-06-18T17:35:53Z","tags":["listening","info"],"pid":18721,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2020-06-18T17:35:55Z","tags":["info","http","server","Kibana"],"pid":18721,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2020-06-18T17:35:56Z","tags":["error","reporting"],"pid":18721,"message":"The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports."}
{"type":"log","@timestamp":"2020-06-18T17:35:56Z","tags":["error","reporting"],"pid":18721,"message":"ErrorEvent {\n  target:\n   WebSocket {\n     _events:\n      [Object: null prototype] { open: [Function], error: [Function] },\n     _eventsCount: 2,\n     _maxListeners: undefined,\n     readyState: 3,\n     protocol: '',\n     _binaryType: 'nodebuffer',\n     _closeFrameReceived: false,\n     _closeFrameSent: false,\n     _closeMessage: '',\n     _closeTimer: null,\n     _closeCode: 1006,\n     _extensions: {},\n     _receiver: null,\n     _sender: null,\n     _socket: null,\n     _isServer: false,\n     _redirects: 0,\n     url:\n      'ws://127.0.0.1:41039/devtools/browser/fefdb355-c461-4f5c-8e1f-cfb58978cf0f',\n     _req: null },\n  type: 'error',\n  message: 'connect ECONNREFUSED 127.0.0.1:41039',\n  error:\n   { Error: connect ECONNREFUSED 127.0.0.1:41039\n       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)\n     errno: 'ECONNREFUSED',\n     code: 'ECONNREFUSED',\n     syscall: 'connect',\n     address: '127.0.0.1',\n     port: 41039 } }"}
{"type":"log","@timestamp":"2020-06-18T17:35:56Z","tags":["warning","reporting"],"pid":18721,"message":"See Chromium's log output at \"/home/vagrant/kibana-7.7.1/data/headless_shell-linux/chrome_debug.log\""}
{"type":"log","@timestamp":"2020-06-18T17:35:56Z","tags":["error","reporting"],"pid":18721,"message":"Error: Could not close browser client handle!\n    at browserFactory.test.then.browser (/home/vagrant/kibana-7.7.1/x-pack/legacy/plugins/reporting/server/lib/validate/validate_browser.js:26:15)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"}
{"type":"log","@timestamp":"2020-06-18T17:35:56Z","tags":["warning","reporting"],"pid":18721,"message":"Reporting 插件自检生成警告：Error: Could not close browser client handle!"}
#查看启动状态
[vagrant@localhost kibana-7.7.1]$ netstat -anp|grep 5601
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      18721/./bin/../node

访问Kibana，http://192.168.0.140:5601

Kibana主页，配置文件：i18n.locale: "zh-CN" 设置语言
Kibana启动成功

Logstash

下载 logstash-7.7.1.tar.gz

#解压 logstash
[vagrant@localhost ~]$ tar -xvf logstash-7.7.1.tar.gz
#目录结构 
[vagrant@localhost logstash-7.7.1]$ ls
bin  config  CONTRIBUTORS  data  Gemfile  Gemfile.lock  lib  LICENSE.txt  logs  logstash-core  logstash-core-plugin-api  modules  nohup.out  NOTICE.TXT  plugin-data  tools  vendor  x-pack
#配置文件目录
[vagrant@localhost config]$ ls
filebeat.conf  jvm.options  log4j2.properties  logstash.conf  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options

配置 jvm.options 中的内存

[vagrant@localhost config]$ vi jvm.options 
#仅需修改-Xms 和 -Xmx 
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m

示例logstash.yml配置 https://www.jianshu.com/p/e7efcc7a870e

配置logstash-sample.conf文件，Logstash的事件处理分为3个阶段: inputs、filters、outputs。
Logstash运行时可以指定conf文件，为了介绍Filebeat的使用，创建filebeat-1.conf。
logstash-sample.conf配置方式很多，需要根据实际场景进行相关配置，这里只为演示，后续会单独写一篇文章介绍配置规则。

#创建 filebeat.conf 并且启动时指定
[vagrant@localhost config]$ vi filebeat.conf
#配置端口
input {
  beats {
       port => 5044
       codec => json{
            charset => "UTF-8"
        }
  }
}
#输出至elasticsearch
output {
  elasticsearch{
    hosts => ["192.168.0.140:9200"]
    #索引格式
    index => "logstash-test-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

启动 Logstash

# logstash本身存在日志，使用后台启动并且，不输出控制台日志
[vagrant@localhost logstash-7.7.1]$ ./bin/logstash -f ./config/filebeat.conf >/dev/null 2>&1 &
[11] 22292
#查看启动日志
[vagrant@localhost logs]$ tail -f -n10 logstash-plain.log 
[2020-06-18T18:39:31,182][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.0.140:9200"]}
[2020-06-18T18:39:31,320][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-06-18T18:39:31,455][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-06-18T18:39:31,470][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/home/vagrant/logstash-7.7.1/config/filebeat.conf"], :thread=>"#"}
[2020-06-18T18:39:31,567][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-06-18T18:39:33,410][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-06-18T18:39:33,457][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-06-18T18:39:33,567][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-18T18:39:33,727][INFO ][org.logstash.beats.Server][main][5dc1e5946dbbc3654c409d7a6be51e5de6ae06dcf07ca7a0a3b518759207b8a6] Starting server on port: 5044
[2020-06-18T18:39:34,507][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Logstash启动成功，

Filebeats

下载filebeat-7.7.1-linux-x86_64.tar.gz

#解压filebeat
[vagrant@localhost ~]$ tar -xvf filebeat-7.7.1-linux-x86_64.tar.gz
#目录结构
[vagrant@localhost filebeat-7.7.1]$ ls
data  fields.yml  filebeat  filebeat.log  filebeat.reference.yml  filebeat.yml  kibana  LICENSE.txt  logs  module  modules.d  nohup.out  NOTICE.txt  README.md

示例filebeat.yml配置 https://www.jianshu.com/p/1ec30324a939

启动 Filebeat

#nohup 后台启动
[vagrant@localhost filebeat-7.7.1]$ nohup ./filebeat -e -c filebeat.yml &
[5] 20426
#查看启动nohup日志
[vagrant@localhost filebeat-7.7.1]$ tail -f -n10 nohup.out 
2020-06-18T19:09:34.256Z    INFO    instance/beat.go:297    Setup Beat: filebeat; Version: 7.7.1
2020-06-18T19:09:34.256Z    INFO    [publisher] pipeline/module.go:110  Beat name: localhost.localdomain
2020-06-18T19:09:34.291Z    WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-18T19:09:34.291Z    INFO    instance/beat.go:438    filebeat start running.
2020-06-18T19:09:34.292Z    WARN    beater/filebeat.go:335  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-18T19:09:34.292Z    INFO    registrar/registrar.go:145  Loading registrar data from /home/vagrant/filebeat-7.7.1/data/registry/filebeat/data.json
2020-06-18T19:09:34.292Z    INFO    registrar/registrar.go:152  States Loaded from registrar: 12
2020-06-18T19:09:34.292Z    INFO    beater/crawler.go:73    Loading Inputs: 1
2020-06-18T19:09:34.292Z    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-06-18T19:09:34.293Z    INFO    log/input.go:152    Configured paths: [/home/vagrant/apache-tomcat-9.0.20/logs/catalina.*.out]
2020-06-18T19:09:34.293Z    INFO    input/input.go:114  Starting input of type: log; ID: 7647873084758492354 
2020-06-18T19:09:34.295Z    INFO    beater/crawler.go:105   Loading and starting Inputs completed. Enabled inputs: 1
2020-06-18T19:09:34.295Z    INFO    cfgfile/reload.go:175   Config reloader started
2020-06-18T19:09:34.295Z    INFO    cfgfile/reload.go:235   Loading of config files completed.

Filebeat启动成功，Tomcat的内容这里不赘述，直接启动然后查看catalina.out日志的同步情况

[vagrant@localhost apache-tomcat-9.0.20]$ ./bin/startup.sh
Using CATALINA_BASE:   /home/vagrant/apache-tomcat-9.0.20
Using CATALINA_HOME:   /home/vagrant/apache-tomcat-9.0.20
Using CATALINA_TMPDIR: /home/vagrant/apache-tomcat-9.0.20/temp
Using JRE_HOME:        /usr/java/jdk1.8.0_211-amd64
Using CLASSPATH:       /home/vagrant/apache-tomcat-9.0.20/bin/bootstrap.jar:/home/vagrant/apache-tomcat-9.0.20/bin/tomcat-juli.jar
Tomcat started.

Filebeat日志打印内容

2020-06-18T19:16:24.325Z    INFO    log/harvester.go:297    Harvester started for file: /home/vagrant/apache-tomcat-9.0.20/logs/catalina.2020-06-18.out
2020-06-18T19:16:32.331Z    INFO    [publisher_pipeline_output] pipeline/output.go:101  Connecting to backoff(async(tcp://192.168.0.140:5044))
2020-06-18T19:16:32.332Z    INFO    [publisher_pipeline_output] pipeline/output.go:111  Connection to backoff(async(tcp://192.168.0.140:5044)) established
2020-06-18T19:16:34.295Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":7}},"total":{"ticks":200,"time":{"ms":9},"value":200},"user":{"ticks":70,"time":{"ms":2}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"06132633-b464-490e-afbc-1a6492cae22f","uptime":{"ms":420136}},"memstats":{"gc_next":9997632,"memory_alloc":7814672,"memory_total":21107344,"rss":774144},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"files":{"72f618d7-3a12-4518-b707-414633178d64":{"last_event_published_time":"2020-06-18T19:16:31.331Z","last_event_timestamp":"2020-06-18T19:16:24.325Z","name":"/home/vagrant/apache-tomcat-9.0.20/logs/catalina.2020-06-18.out","read_offset":7192,"size":5058,"start_time":"2020-06-18T19:16:24.325Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1,"batches":1,"total":1},"read":{"bytes":6},"write":{"bytes":1863}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":1,"retry":1,"total":2},"queue":{"acked":1}}},"registrar":{"states":{"current":10,"update":2},"writes":{"success":2,"total":2}},"system":{"load":{"1":0.07,"15":0.58,"5":0.46,"norm":{"1":0.07,"15":0.58,"5":0.46}}}}}}

此时Logstash的日志中也会有信息输出
打开Kibana页面，Management菜单创建索引模式

image.png
Discover菜单中查看同步的日志信息

image.png
日志菜单中设置添加日志索引

image.png
流式传输中查看日志

image.png

结尾

实际使用中，会有很多详细的配置，会放在后续文章中进行讲解。
还可以将MySQL的慢查询日志、MySQL错误日志、Nginx的日志，整合建立不同索引，方便我们进行查询。

ELK Stack分布式日志分析

流程

环境及版本

Elasticsearch

Kibana

Logstash

Filebeats

结尾

你可能感兴趣的:(ELK Stack分布式日志分析)