Docker安装logstash，配置多管道

docker安装logstash首先需要知道logstash的目录结构如下图：

创建挂载目录

mkdir -p logstash/{config,pipeline,data}

修改 config/logstash.yml

config:
  reload:
    automatic: true
    interval: 3s
xpack:
  management.enabled: false
  monitoring.enabled: false
#path.config: /usr/share/logstash/config/conf.d/*.conf
#path.logs: /usr/share/logstash/logs
#以下配置能在kibana查看logstash状态
xpack.monitoring.enabled: true 
xpack.monitoring.elasticsearch.username: "logstash46"
xpack.monitoring.elasticsearch.password: "123456"
xpack.monitoring.elasticsearch.hosts: ["http://172.16.151.46:9200"]

修改 config/pipelines.yml

每一个pipeline.id对应一个管道，本项目是使用logstash消费kafka，针对不同的topic，建立不同的通道，效果等同配置文件中的if判断匹配tag将不同的数据写入不同的index，如果使用一个通道（默认通道是main）将会导致一个索引能查到所有数据，无法进行分类。且当接入数据类型太多使用if判断会导致配置文件臃肿。

 - pipeline.id: video
   path.config: "/usr/share/logstash/pipeline/kanba-video.conf"
 - pipeline.id: pay
   path.config: "/usr/share/logstash/pipeline/kanba-pay.conf"
 - pipeline.id: ott
   path.config: "/usr/share/logstash/pipeline/kanba-ott.conf"
 - pipeline.id: tls
   path.config: "/usr/share/logstash/pipeline/kanba-tls.conf"

修改pipeline/xxx.conf文件

此处以一个文件为例

input{
     kafka {
        topics => "kanba-pay" 
	    group_id => "kanba-pay"
        type => "kanba-pay47"
        bootstrap_servers => "172.16.151.46:9092,172.16.151.47:9092,172.16.151.48:9092"
        codec => "json" 
     }
}

filter{
	grok{
			match => {
				"message" => "\[bgctvpayservice\]\[%{WORD:interface}\]"
			}
	    }
	    grok{
			match => {
				"message" => "uid=%{NUMBER:uid}\&"
			}
	    }
	    grok{
			match => {
				"message" => "\&ret_code=%{WORD:ret_code}\&"
			}
	    }
	    grok{
			match => {
				"message" => "vid=%{NUMBER:vid}"
			}
	    }

    mutate{
        remove_field => ["beat"]
        remove_field => ["@version"]
        remove_field => ["_score"]
        remove_field => ["prospector"]
        remove_field => ["_type"]
    }
}


output {
    elasticsearch { 
      hosts => ["172.16.151.46:9200","172.16.151.47:9200","172.16.151.48:9200"] 
      index => "kanba-pay"
    }
    stdout { codec => rubydebug }
} 

启动logstash

docker run -d --restart=always --privileged=true --name logstash47 -p 5044:5044 -p 5047:5047 -p 9600:9600 --network host -v /storage/brick/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml -v /storage/brick/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /storage/brick/logstash/pipeline/:/usr/share/logstash/pipeline/ 172.16.151.46:5000/logstash:6.8.6