以C-S模式使用fluentd来收集docker容器日志

• 1.安装服务器端
  环境：CentOS Linux release 7.4.1708 (Core)
  安装td-agent:

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

下载服务器端所需插件：

td-agent-gem install fluent-plugin-file-alternative
td-agent-gem install fluent-plugin-forest

• 2.安装客户端及所需插件
  环境：Ubuntu 16.04.3 LTS
  安装td-agent:

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh

# 客户端需要安装的插件
td-agent-gem install fluent-plugin-rewrite-tag-filter
td-agent-gem install fluent-plugin-grep
td-agent-gem install fluent-plugin-record-reformer
 

• 3.赋予td-agent用户root权限：

修改 /etc/passwd 文件

vim /etc/passwd

找到如下行，把td-agent用户ID修改为 0 ，如下所示：

td-agent:x:0:133::/var/lib/td-agent:/bin/false

• 4.修改客户端和服务器端的配置文件：
  客户端/etc/td-agent/td-agent.conf：

root@ubuntu:/etc/td-agent# vim td-agent.conf
# 客户端配置文件

  @type forward
  port 24224
  bind 0.0.0.0


# To get the informations of plugins http://localhost:24220/api/plugins.json

  @type monitor_agent
  bind 0.0.0.0
  port 24220




    @type forward
    
      host 192.16.1.148
      port 24224
    
    
    
      flush_interval 10s
        

服务器端/etc/td-agent/td-agent.conf：

  @type forward
  port 24224
  bind 0.0.0.0



  @type stdout

• 5.关闭客户端和服务器端的防火墙
• 先测试能否通信：（ctrl+] 然后在telnet 命令行输入quit 就可以退出telnet）

#客户机端
root@ubuntu:~# telnet 192.16.1.148 24224
Trying 192.16.1.148...
Connected to 192.16.1.148.
#服务器端
[root@worker1 td-agent]# telnet 192.16.1.157 24224
Trying 192.16.1.157...
Connected to 192.16.1.157.

• 若不能通信，可能是打开了防火墙所致，粗暴的解决方式：关闭防火墙

service  firewalld status
service  firewalld stop

• 更精准的解决方式是添加防火墙规则,例如：

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.16.1.157" port protocol="tcp" port="24224" accept"

//在server端也就是centos（ip=192.16.1.148）上
[root@worker1 td-agent]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.16.1.157" port protocol="tcp" port="24224" accept" 
success
[root@worker1 td-agent]# systemctl restart firewalld.service
[root@worker1 td-agent]# firewall-cmd --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
        rule family="ipv4" source address="192.16.1.157" port port="24224" protocol="tcp" accept
[root@worker1 td-agent]# 

• 6.查看服务器端收集到的来着客户端的日志

[root@worker1 td-agent]# tail -f td-agent.log 
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=13
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=17
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=8
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=93
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=13
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 20:01:50.000000000 +0800 docker.nginx_test: {"source":"stdout","log":"172.17.0.1 - - [14/Sep/2018:12:01:50 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog"}
2018-09-14 20:02:03.000000000 +0800 docker.test_of_nginx01: {"container_id":"2c27be33f81ba79f10c56183b0f7f71c833b021d59333d9876b80e66fda756c9","container_name":"/test_of_nginx01","source":"stdout","log":"172.17.0.1 - - [14/Sep/2018:12:02:03 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\""}

成功效果图