ELK分布式日志收集系统的搭建

ELK分布式日志收集系统的搭建

filebeat+logstash+elasticsearch+kibana搭建一个分布式的日志收集系统

1.linux下filebeat下载安装

• deb:安装

 curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.1-amd64.deb

sudo dpkg -i filebeat-7.6.1-amd64.deb

• 启动filebeat

 sudo service filebeat start

• filebeat配置文件在/etc/filebeat/filebeat.yml中

#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/folio/okapi/okapi.log
  tag: ["okapi"]
  fields:
    index: 'filebeat-7.6-okapi'
- type: log
  enabled: true
  paths:
    - /var/lib/okapi/mod-circulation/info.log
  tag: ["circulation"]
  fields:
    index: 'filebeat-7.6-circulation'

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
#============================= Filebeat modules ===============================

filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1

#============================== Kibana =====================================

setup.kibana:

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["172.168.13.208:5044"]

#================================ Processors=====================================

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

3.linux下logstash下载安装

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install logstash

• logstash启动

 sudo systemctl start logstash.service

• logstash配置文件在/etc/logstash目录下conf.d 中以config为结尾的配置文件

input {

  beats {
           port => 5044
        }
}
filter {
  grok {
    match => { "message" => "(?>%{TIMESTAMP_ISO8601})" }
  }
}

output {

  if [fields][index] == "filebeat-7.6-okapi" {
    elasticsearch {
      hosts => ["172.168.13.208:9200"]
      ## index 默认值是"logstash-%{+YYYY.MM.dd}",不允许包含大写字母
      index => "filebeat-okapi"
      }
   }else if [fields][index] == "filebeat-7.6-circulation" {
    elasticsearch {
      hosts => ["172.168.13.208:9200"]
      index => "filebeat-circulation"
      }
    }
}

3.linux下elasticsearch下载安装

 wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install elasticsearch

• 要将Elasticsearch配置为在系统启动时自动启动，请运行以下命令：

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

• Elasticsearch启动停止

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

• elasticsearch配置文件在/etc/elasticsearch/elasticsearch.yml

# ------------------------------------ Node ------------------------------------

node.name: master

# ----------------------------------- Paths ------------------------------------

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

# ---------------------------------- Network -----------------------------------

network.host: 0.0.0.0

# --------------------------------- Discovery ----------------------------------

cluster.initial_master_nodes: ["master"]

# ---------------------------------- Various -----------------------------------

http.cors.enabled: true
http.cors.allow-origin: "*"

3.linux下kibana下载安装

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install kibana

• 要将Kibana配置为在系统启动时自动启动，请运行以下命令：

sudo /bin/systemctl daemon-reload

sudo /bin/systemctl enable kibana.service

• Kibana启动停止

sudo systemctl start kibana.service

sudo systemctl stop kibana.service

• logstash配置文件在/etc/kibana/kibana.yml

server.basePath: "/kibana"
#i18n.locale: "en"
i18n.locale: "zh-CN"
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9200/"]
kibana.index: ".kibana"