部署 fluentd failover集群

High Availability Config - Fluentd

Install by RPM Package (Red Hat Linux) - Fluentd

一、环境准备

Hostname	IP
fluentd1	10.0.31.15
fluentd2	10.0.31.16

1. NTP 服务

yum install -y chrony

修改配置文件 /etc/chrony.conf ，配置时间源

# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
server time1.aliyun.com iburst

启动服务

systemctl restart chronyd

查看同步情况

[root@fluentd2 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 203.107.6.88                  2   6    17     2   -480us[-1320us] +/-   21ms

2. ulimit 设置

将如下内容追加到 /etc/security/limits.conf

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

3. 网络内核参数调优

追加如下内容到 ``

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
# If forward uses port 24224, reserve that port number for use as an ephemeral port.
# If another port, e.g., monitor_agent uses port 24220, add a comma-separated list of port numbers.
# net.ipv4.ip_local_reserved_ports = 24220,24224
net.ipv4.ip_local_reserved_ports = 24224

执行 sysctl -p 使生效

4. Use sticky bit symlink/hardlink protection

检查是否配置如下

fs.protected_hardlinks = 1
fs.protected_symlinks = 1

检查方法

[root@fluentd1 ~]# sysctl -a |grep "fs.protected"
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

如果不对，请检查文件 /etc/sysctl.d/10-link-restrictions.conf or /usr/lib/sysctl.d/50-default.conf

5. 依赖包

yum install -y glibc

准备工作完毕后，重启服务器检查是否生效

二、安装 td-agent

td-agent 是 Treasure Data, Inc 提供的 stable distribution of Fluentd

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh

默认配置文件 /etc/td-agent/td-agent.conf

三、failover配置

failover是在 客户端 配置的，两个配置参数 standby & weight。
forward - Fluentd

standby
Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward plugin until then.
weight

Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward plugin until then.

1. active-standby配置

  @type forward
  port 24224
  bind 0.0.0.0



  @type forward
  
    name fluentd_1
    host 10.0.31.15
    port 24224
    standby
  
  
    name fluentd_2
    host 10.0.31.16
    port 24224
  

该模式下，一台主，一台备。主机宕的时候，有如下提示：

2022-10-13 02:54:11 +0000 [warn]: #0 detached forwarding server 'fluentd_2' host="10.0.31.16" port=24224 phi=16.25190074780215 phi_threshold=16
2022-10-13 02:54:11 +0000 [warn]: #0 using standby node 10.0.31.15:24224 weight=60

四、测试

$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
$ tail -n 1 /var/log/td-agent/td-agent.log
2018-01-01 17:51:47 -0700 debug.test: {"json":"message"}