ELK日志系统(ElasticSearch+Logstash+Kibana+kafka+Filebeat)

本文章仅作为个人笔记

elastic官网

kafka官网

zookeeper官网

ElasticSearch官方yum安装文档

Logstash官方yum安装文档

Kibana官方yum安装文档

Filebeat官方yum安装文档

kafka2.6.0安装包官方下载地址

zookeeper3.6.2安装包官方下载地址

zookeeper官方运行文档

先贴上各工具安装教程，可先安装不启动，配置后再启动。

------------------安装部分----------------------

• 安装java

  • yum install java-1.8.0-openjdk.x86_64 java-1.8.0-openjdk-devel.x86_64 -y

• ElasticSearch安装

  • rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

  • vim /etc/yum.repos.d/elasticsearch.repo

    [elasticsearch]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=0
    autorefresh=1
    type=rpm-md
    

  • yum install --enablerepo=elasticsearch elasticsearch -y

• Logstash安装

  • rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

  • vim /etc/yum.repos.d/logstash.repo

    [logstash-7.x]
    name=Elastic repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    

  • yum install logstash -y

• Kibana安装

  • rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

  • vim /etc/yum.repos.d/kibana.repo

    [kibana-7.x]
    name=Kibana repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    

  • yum install kibana -y

  • chkconfig --add kibana

  • service kibana start

• Filebeat安装

  • rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

  • vim /etc/yum.repos.d/filebeat.repo

    [kibana-7.x]
    name=Kibana repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    

  • yum install filebeat -y

  • systemctl enable filebeat

  • chkconfig --add filebeat

• kafka安装

  • 因为kafka的服务依赖于java及zookeeper
  • zookeeper安装
    • wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz * tar -zxvf apache-zookeeper-3.6.2-bin.tar.gz
    • cd apache-zookeeper-3.6.2-bin
  • kafka安装
    • wget https://mirror.bit.edu.cn/apache/kafka/2.6.0/kafka_2.13-2.6.0.tgz
    • tar -zxvf kafka_2.13-2.6.0.tgz
    • cd kafka_2.13-2.6.0
      ------------------配置部分----------------------

• 配置ElasticSearch(默认yum安装的不更改配置亦可，看个人需求)

  • vim /etc/elasticsearch/elasticsearch.yml

    找到配置文件中的cluster.name，打开该配置并设置集群名称
    cluster.name: demon
    找到配置文件中的node.name，打开该配置并设置节点名称
    node.name: elk-1
    解决启动报错
    cluster.initial_master_nodes: ["node-1"]
    配置内存使用用交换分区
    bootstrap.memory_lock: true
    监听的网络地址
    network.host: 0.0.0.0
    开启监听的端口
    http.port: 9200
    增加新的参数，这样head插件可以访问es (5.x版本，如果没有可以自己手动加)
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    

  • 集群配置(非集群可略过)

    discovery.zen.ping.unicast.hosts: ["192.168.60.201", "192.168.60.202","192.168.60.203"]  
    # 集群各节点IP地址，也可以使用els、els.demo.com等名称，需要各节点能够解析
    discovery.zen.minimum_master_nodes: 2              # 为了避免脑裂，集群节点数最少为 半数+1
    

  • 配置开启自启

  • chkconfig --add elasticsearch

  • service elasticsearch start

• kibana配置

  • vi /etc/kibana/kibana.yml

    server.port: 5601
    server.host: “0.0.0.0”
    elasticsearch.hosts: [“http://localhost:9200”]
    kibana.index: “.kibana”
    

  • service kibana start

• kafka配置(kafka依赖于zookeeper，因此先配置zookeeper并启动)

  • zookeeper配置
    • cp conf/zoo_sample.cfg conf/zoo.cfg
    • bin/zkServer.sh start
  • kafka配置

    • vim kafka-2.6.0-src/config/zookeeper.properties

      server.1=192.168.1.190:2888:3888 //kafka集群ip:port
      

    • vim kafka-2.6.0-src/config/server.properties

      broker.id=0
      listeners=PLAINTEXT://192.168.1.190:9092
      zookeeper.connect=192.168.1.190:2181,192.168.1.191:2181,192.168.1.192:2181
      

    • 启动服务

    • ./bin/kafka-server-start.sh config/server.properties

    • 创建topic(测试)

    • ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic

    • 查询topic(测试)

    • ./bin/kafka-topics.sh --zookeeper localhost:2181 --list

    • 发送消息(测试)

    • ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic

    • 接收消息(测试)

    • ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testtopic --from-beginning

• Logstash配置(input为监控入地址，output为elasticsearch出地址)

  • 设置输入数据

  • vim /etc/logstash/conf.d/input.conf

    input {
            kafka {
                    type => "nginx_kafka"
                    codec => "json"
                    topics => "nginx"
                    decorate_events => true
                    bootstrap_servers => "localhost:9092"
            }
    }
    

  • 设置输出数据

    output {
            if [type] == "nginx_kafka" {
                    elasticsearch {
                            hosts => ["localhost"]
                            index => 'logstash-nginx-%{+YYYY-MM-dd}'
                    }
            }
    }
    

  • service logstash start

• filebeat配置

  • vim /etc/filebeat/filebeat.yml

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/nginx/access.log
        json.keys_under_root: true
        json.add_error_key: true
        json.message_key: log
    output.kafka:
      hosts: ["localhost:9092"]
      topic: "nginx"
    

  • service filebeat start

• 至此整套elk服务便部署好了

• 可以打开 http://localhost:5601 -> StackManagement -> index Patterns 选择 Create index pattern 设置过滤的key，然后创建规则，创建好后。

• 打开 http://localhost:5601 -> Discover 查看对应的log，使用filter进行过滤。