日志收集流程为：fluentd - kafka - logstash - elasticsearch - kibana

使用镜像如下：

registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1                       version:1.4
registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1                                   version:2.1
docker.io/zookeeper:3.5
docker.elastic.co/elasticsearch/elasticsearch:6.5.4
docker.elastic.co/logstash/logstash:6.5.4
docker.elastic.co/kibana/kibana:6.5.4
docker.io/mobz/elasticsearch-head:5


192.168.2.244 fluentd、zk、kafka、es、kibana
192.168.2.245 logstash、es-head、zk、kafka、es
192.168.2.246 es-balance、zk、kafka、es

创建数据持久化目录，配置文件目录

mkdir -p /efk/{es,es-balance,kafka,zk,fluentd,logstash}/conf && mkdir /efk/{es,kafka,zk}/data


  • 系统环境配置

echo "vm.max_map_count=262144" >> /etc/sysctl.conf && sysctl -p
echo -e "* soft nofile 65536\n* hard nofile 65536" >> /etc/security/limits.conf


  • zk集群安装

配置文件(三台机器使用相同配置)

cat > /efk/zk/conf/zoo.cfg << EOF
tickTime=2000
dataDir=/data/zookeeper/
dataLogDir=/data/zookeeper/logs
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.2.244:2888:3888
server.2=192.168.2.245:2888:3888
server.3=192.168.2.246:2888:3888
EOF

为每台机器设置myid

echo 1 > /efk/zk/data/myid    #244     跟配置文件server.id设置一致
echo 2 > /efk/zk/data/myid    #245
echo 3 > /efk/zk/data/myid    #246

启动容器(三台机器执行相同命令)

docker run -d --restart=always --name zk --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/zk/conf/zoo.cfg:/conf/zoo.cfg -v /efk/zk/data/:/data/zookeeper zookeeper:3.5

zk集群启动后查看各节点身份

docker exec -it zk zkServer.sh status



  • kafka集群安装

配置文件(每台机器的broker.id不能相同)

cat > /efk/kafka/conf/server.properties << EOF
broker.id=1
listeners=PLAINTEXT://192.168.2.246:9092
port=9092
logs.dirs=/tmp/kafka-logs
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
zookeeper.connect=192.168.2.244:2181,192.168.2.245:2181,192.168.2.246:2181
EOF

启动容器(三台机器执行相同命令)

docker run -d --restart=always --name=kafka --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/kafka/conf/server.properties:/usr/local/kafka_2.11-2.1.0/config/server.properties -v /efk/kafka/data:/tmp/kafka-logs registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1



  • es集群安装

配置文件(修改node.name、network.publish_host)

cat > /efk/es/conf/elasticsearch.yml << EOF
cluster.name: es-cluster
node.name: es-node1
network.bind_host: 0.0.0.0
network.publish_host: 192.168.2.244
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true  
discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"]
discovery.zen.minimum_master_nodes: 2
EOF

balance节点配置文件

cat > /efk/es-balance/conf/elasticsearch.yml << EOF
cluster.name: es-cluster
node.name: es-node4
network.bind_host: 0.0.0.0
network.publish_host: 192.168.2.246
http.port: 9201
transport.tcp.port: 9301
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: false
node.data: false
node.ingest: false
discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"]
discovery.zen.minimum_master_nodes: 2
EOF

启动容器(三台机器执行相同命令，数据节点)

docker run -d --restart=always --name=es --net=host -e ES_JAVA_OPTS="-Xms2g -Xmx2g" -v /etc/localtime:/etc/localtime:ro -v /efk/es/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /efk/es/data:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.5.4

balance节点

docker run -d --restart=always --name=es-balance --net=host -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -v /etc/localtime:/etc/localtime:ro -v /efk/es-balance/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.5.4

启动es-head插件

docker run -d --name=es-head -p 9100:9100 mobz/elasticsearch-head:5



  • kibana安装

启动容器

docker run -d --name kibana --restart=always -e ELASTICSEARCH_URL=http://192.168.2.246:9201 -v /etc/localtime:/etc/localtime:ro -p 5601:5601 docker.elastic.co/kibana/kibana:6.5.4



  • logstash安装

配置文件

consumer_threads与kafka设置的分区数保持一致

touch /efk/logstash/conf/logstash.yml
cat > /efk/logstash/conf/kafka-logstash-es.conf << EOF
input {
    kafka {
bootstrap_servers => "192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092"
group_id => "kafka-logstash"
topics => ["docker_log","sys_log"]
consumer_threads => 1
codec => "json"
        }
    }
output {
    elasticsearch {
hosts => ["192.168.2.244:9200","192.168.2.245:9200","192.168.2.246:9200"]
index => "logstash-%{+YYYY.MM.dd}"
codec => "json"
    }
    stdout {
codec => "rubydebug"
        }
    }
EOF

启动logstash容器

docker run -d --name=logstash --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/logstash/conf/logstash.yml:/usr/share/logstash/config/logstash.yml -v /efk/logstash/conf/kafka-logstash-es.conf:/usr/share/logstash/pipeline/kafka-logstash-es.conf docker.elastic.co/logstash/logstash:6.5.4



  • Fluentd安装

配置文件(收集容器日志以及系统日志)

最后一个match需要设置为**，否则会出现warn信息

cat > /efk/fluentd/conf/fluent.conf << EOF

  @type tail
  path /var/log/containers/*/*.log
  pos_file /var/log/containers/containers.log.pos
  tag docker.log 
  
    @type json
  
  read_from_head true


  @type syslog
  port 5140
  bind 0.0.0.0
  
    message_format rfc3164
  
  tag system.log


  @type record_transformer
  
    hostname "192.168.2.244"
  


  @type kafka_buffered
  brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092
  default_topic docker_log
  output_data_type json


  @type kafka_buffered
  brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092
  default_topic sys_log
  output_data_type json

EOF

收集系统日志需要开启rsyslog服务(根据服务器IP进行配置)

echo '*.* @192.168.2.244:5140' >> /etc/rsyslog.conf && systemctl restart rsyslog

启动fluentd容器

docker run -d --name=fluentd --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf -v /var/lib/docker/containers/:/var/log/containers registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1


参考文档：

https://www.elastic.co/guide

https://github.com/mobz/elasticsearch-head

https://docs.fluentd.org

http://kafka.apache.org/21/documentation.html

http://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperAdmin.html