docker部署logstash的自定义pattern

#### 前言

为了能够更好的根据不同的日志格式，去进行处理展示我们需要日志进行，需要使用到logstash的自定义pattern

 

#### 准备工作

##### 时区文件

保证容器服务的时间与宿主机的时间一致

```

cat > /etc/timezone <<-EOF
Asia/Shanghai

EOF

```

 

##### gunicorn

```

cat > /data/logstash/patterns/gunicorn <<-EOF
# Log formats
STATUS [0-9]*
LOGLEVEL [A-Za-z]*
INFO .*
GUNICORNCOMMONLOG \[%{HTTPDATE:timestamp}\] \[%{STATUS:status}\] \[%{LOGLEVEL:loglevel}\] % {INFO:info}

EOF

```

 

##### logstash.yml文件

```

cat > /data/logstash/config/logstash.yml <<-EOF
http.host: 0.0.0.0

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: url/ip:9200   # 建议: 云主机上部署，url采用内网的url，同理，ip也是采用内网的ip
xpack.monitoring.elasticsearch.username: elasticsearch_username
xpack.monitoring.elasticsearch.password: elasticsearch_password

EOF

```

 

##### logstash.conf文件

```

cat > /data/logstash/pipeline/logstash.conf <<-EOF
input {
  beats {
    port => 5044    # 本机部署的logstash端口，注:是容器暴露在宿主机的端口
    codec => plain { charset => "UTF-8" }    #  由于从filebeat段推送到logstash的日志文件不是json格式的，同时存在中文字符，故采用plain格式，并对数据进行UTF-8编码转换
  }
}

filter {
  grok {
    patterns_dir => "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns"
    match => { "message" => "%{GUNICORNCOMMONLOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]    # 时间格式再次进行转换
  }
  urldecode {
    all_fields => true    # 进行转码
  }
}

output {
  elasticsearch {
    action => "index"

    manage_template => true    # 使用自定义的patterns
    hosts => ["url/ip:9200"]    # 建议: 云主机上部署，url采用内网的url，同理，ip也是采用内网的ip
    index => "logstash-dev-img-%{+YYYYMMdd}"    # logstash-dev-img，此内容，可自定义
    user => "elasticsearch_username"
    password => "elasticsearch_password"
  }
}

EOF

```

注释：用于json格式化文件的input写法

```

input {
  tcp {
    port => 5044
    codec => "json_lines"    # 每行读取json序列化数据
  }
}

```

 

#### 部署方式2:docker-compose形式部署docker服务

```

cat > docker-compose.yml <<-EOF
version: "3.5"
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:7.6.2
    container_name: logstash
    hostname: logstash-
    privileged: true
    user: root
    ports:
      - 5044:5044
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /data/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - /data/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro

      - /data/logstash/patterns/gunicorn:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/gunicorn:ro
    restart: always
    tty: true

EOF

```

> docker-compose up -d

 

#### 补充

此文档存在错误，请不要照抄；此时，记录下来，便于后期排查