Linux下td-agent(fluentd)的安装和配置

1.Fluentd安装之前的准备工作

参考的官网链接

  • 1.在节点上设置NTP守护程序,以获得准确的当前时间戳。

安装ntp服务

//ubuntu
sudo apt install ntp
//centos
yum install ntp -y

编辑 /etc/ntp.conf

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

#新增:日志目录.
logfile /var/log/ntpd.log

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#新增:时间服务器列表.
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst
  • 注意:ubuntu下服务叫做ntp,centos下叫做ntpd
    修改配置文件之后,你需要重新加载 ntpd:
systemctl enable ntp
systemctl start ntp

或者

/etc/init.d/ntp stop
/etc/init.d/ntp start
  • ubuntu主机下例子:
root@ubuntu:~# systemctl status ntp
● ntp.service - LSB: Start NTP daemon
   Loaded: loaded (/etc/init.d/ntp; bad; vendor preset: enabled)
   Active: active (running) since 四 2018-10-11 14:15:51 CST; 1 months 8 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 2
   Memory: 2.2M
      CPU: 3min 42.282s
   CGroup: /system.slice/ntp.service
           └─1506 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 123:132

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@ubuntu:~# systemctl is-enabled ntp
ntp.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install is-enabled ntp
enabled
  • centos主机下例子:
[root@master ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
[root@master ~]# systemctl enable ntpd
[root@master ~]# systemctl is-enabled ntpd
enabled
[root@master ~]# systemctl start ntpd     
[root@master ~]# systemctl status ntpd    
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-11-19 16:55:01 CST; 5s ago
  Process: 7170 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 7172 (ntpd)
    Tasks: 1
   Memory: 1.4M
   CGroup: /system.slice/ntpd.service
           └─7172 /usr/sbin/ntpd -u ntp:ntp -g

Nov 19 16:55:01 master.novalocal systemd[1]: Starting Network Time Service...
Nov 19 16:55:01 master.novalocal ntpd[7172]: proto: precision = 0.089 usec
Nov 19 16:55:01 master.novalocal ntpd[7172]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 19 16:55:01 master.novalocal systemd[1]: Started Network Time Service.

查看节点同步状态

root@ubuntu:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
+ntp.hkg10.hk.le 130.133.1.10     2 u   38   64  377  121.432  -10.265   0.534
*45.125.1.20 (45 101.231.167.217  2 u   30   64  377  353.921   12.721   0.429
+118.140.184.98  223.255.185.2    2 u   34   64  157    9.668   13.966  40.916
-alphyn.canonica 192.53.103.108   2 u   37   64   73  341.456   41.184  37.807
-chilipepper.can 17.253.34.253    2 u   96   64   76  299.193  -17.247  29.960
-golem.canonical 145.238.203.14   2 u   36   64   37  274.246  -28.804  28.527
root@ubuntu:~# ntpstat
synchronised to NTP server (45.125.1.20) at stratum 3 
   time correct to within 236 ms
   polling server every 64 s
  • 2.增加最大文件描述符数

使用ulimit -n命令检查当前号码。

$ ulimit -n
1024

如果控制台显示1024,是不够的。在/etc/security/limits.conf文件中添加以下行并重新启动计算机。

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536
  • 3.优化网络内核参数。

对于由许多Fluentd实例组成的高负载环境,请将这些参数添加到您的/etc/sysctl.conf文件中。

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

并输入sysctl -p或重新启动节点以使更改生效。

sysctl -p

2. 安装Fluentd的客户端td-agent

  • 参考官网安装教程
  • 对配置文件进行配置
//根据版本下载相应客户端,我的是16
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
//CentOS Linux release 7.4.1708 (Core)
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
//开始
systemctl start td-agent.service
//查看状态
systemctl status td-agent.service
//停止
systemctl stop td-agent.service
//打开配置文件
vim /etc/td-agent/td-agent.conf

配置文件的语法可以参考官网给出的,或者参考博客1和博客2

3.简明使用

Fluentd 安装使用可以参考该文章。
参考该博客,有比较详尽的fluentd工具使用经验, 日志收集工具Fluentd使用总结

  • fluentd配置文件所在目录:/etc/td-agent/td-agent.conf
  • fluentd日志文件所在目录:/var/log/td-agent/td-agent.log
  • fluentd二进制文件所在目录:/opt/td-agent/embedded/bin/
  • 安装插件的方法:td-agent-gem install [插件名称]

4.监听python和HTTP请求的配置实例

  • 配置文件的配置如下:

  @type forward
  port 24224


#匹配项:监听python的配置

  @type stdout


#匹配项:监听curl命令产生的HTTP的get和post请求
# listening for HTTP Requests

  @type http
  port 8888
  bind 0.0.0.0


# print the data arrived on each incoming request to standard output

  @type stdout



  @type stdout

开一个终端用于跟踪Fluentd收集到的应用日志使用以下命令:

tail -f /var/log/td-agent/td-agent.log

打开另外一个终端用于测试和监听

  • curl命令操作
curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

curl -i -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/test.cycle
  • python文件执行操作
    创建两个文件如下图:


    simple.py和test.py
# test.py
from fluent import sender
from fluent import event
sender.setup('fluentd.test', host='localhost', port=24224)
event.Event('follow', {
  'from': 'userA',
  'to':   'userB'
})
# simple.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import itertools
import json
from fluent import sender
from fluent import event

sender.setup('fluentd.test', host='localhost', port=24224)

class cartesian(object):
    def __init__(self):
        self._data_list=[]

    def add_data(self,data=[]): #添加生成笛卡尔积的数据列表
        self._data_list.append(data)

    def build(self): #计算笛卡尔积
        for item in itertools.product(*self._data_list):
            print(item)

if __name__=="__main__":
    car=cartesian()
    car.add_data([1,2,3,4])
    car.add_data([5,6,7,8])
    car.add_data([9,10,11,12])
    car.build()
    event.Event('Cartesian', {
       'Set': '[1,2,3,4],[5,6,7,8],[9,10,11,12]',
       'Cartesian product': 'results are stdout,not here'
    })

分别执行两个文件

python test.py
python simple.py

执行终端显示结果如下


执行终端显示

日志跟踪终端显示如下:

root@ubuntu:/var/log/td-agent# tail -f td-agent.log
2018-09-06 10:35:15 +0800 [info]: #0 starting fluentd worker pid=8670 ppid=8665 worker=0
2018-09-06 10:35:15 +0800 [info]: #0 listening port port=24224 bind="0.0.0.0"
2018-09-06 10:35:15 +0800 [info]: #0 fluentd worker is now running worker=0
……
2018-09-06 15:34:56.333452308 +0800 debug.test: {"json":"message"}
2018-09-06 15:35:08.899919433 +0800 test.cycle: {"action":"login","user":2}
2018-09-06 15:35:23.000000000 +0800 fluentd.test.follow: {"to":"userB","from":"userA"}
2018-09-06 15:35:33.000000000 +0800 fluentd.test.Cartesian: {"Cartesian product":"results are stdout,not here","Set":"[1,2,3,4],[5,6,7,8],[9,10,11,12]"}

5.监听docker容器的配置实例

Docker版本需要在17.05以上才可以使用日志驱动插件(见官网)。
使用参见官网说明:https://docs.docker.com/config/containers/logging/fluentd/

  • 修改 /etc/passwd 文件
vim /etc/passwd

找到如下行,把td-agent用户ID修改为 0 ,如下所示:

td-agent:x:0:133::/var/lib/td-agent:/bin/false
  • 修改配置文件(/etc/td-agent/td-agent.conf )如下:

  @type forward
  port 24224
  bind 0.0.0.0


  @type stdout

td-agent.conf
  • 重启fluentd
systemctl restart td-agent.service 
  • 运行容器
docker run  -d -p 8081:80 --name nginx_bashlog -v /work/DOCKER/ubuntu-fluentd-test01:/root --log-driver=fluentd --log-opt tag=docker.nginx_test --log-opt fluentd-async-connect nginx
查看fluentd运行状态
  • 查看收集到的日志
cd /var/log/td-agent
tail -f td-agent.log

收集到的日志如下:

2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"log":"2018/09/10 08:01:41 [error] 10#10: *1 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"localhost:8081\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog","source":"stderr"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"container_name":"/nginx_bashlog","source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET /favicon.ico HTTP/1.1\" 404 169 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6"}

td-agent.log

你可能感兴趣的:(Linux下td-agent(fluentd)的安装和配置)