zabbix-server mysql 高可用
# zabbix server
http://mychenqi.com/chenqi/6.html
内存溢出
添加一个模板后 zabbix-server 宕了，删除模板后 zabbix-server 自动起来了。通过分析 zabbix-log 发现产生了 oom 内存溢出。
日志如下
cannot initialize history value cache: cannot get private shared memory of size 8589934592 for value cache size: [12] Cannot allocate memory 94065:20190903:135513.117 enabling SNMP agent checks on host "172.102.252.1": host became available |
|
94041:20190903:135558.738 __mem_malloc: skipped 0 asked 104 skip_min 4294967295 skip_max 0 |
|
94041:20190903:135558.738 [file:strpool.c,line:51] zbx_mem_malloc(): out of memory (requested 102 bytes) |
|
94041:20190903:135558.738 [file:strpool.c,line:51] zbx_mem_malloc(): please increase CacheSize configuration parameter |
|
94041:20190903:135558.738 === memory statistics for string pool === |
|
94041:20190903:135558.739 free chunks of size 24 bytes: 2 |
|
94041:20190903:135558.739 free chunks of size 72 bytes: 1 |
|
94041:20190903:135558.739 min chunk size: 24 bytes |
|
94041:20190903:135558.739 max chunk size: 72 bytes |
|
94041:20190903:135558.739 memory of total size 1257920 bytes fragmented into 13242 chunks |
|
94041:20190903:135558.739 of those, 120 bytes are in 3 free chunks |
|
94041:20190903:135558.739 of those, 1045944 bytes are in 13239 used chunks |
|
94041:20190903:135558.739 ================================ |
|
94041:20190903:135558.740 === Backtrace: === |
|
94041:20190903:135558.753 14: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](zbx_backtrace+0x35) [0x563a47395889] |
|
94041:20190903:135558.753 13: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](__zbx_mem_malloc+0x163) [0x563a473917a8] |
|
94041:20190903:135558.753 12: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](+0xe5eec) [0x563a47391eec] |
|
94041:20190903:135558.754 11: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](zbx_hashset_insert_ext+0x2d4) [0x563a4739aa46] |
|
94041:20190903:135558.754 10: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](zbx_strpool_intern+0x5d) [0x563a473920ab] |
|
94041:20190903:135558.754 9: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](+0xba9bb) [0x563a473669bb] |
|
94041:20190903:135558.754 8: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](+0xc0788) [0x563a4736c788] |
|
94041:20190903:135558.754 7: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](DCsync_configuration+0xc0f) [0x563a47370452] |
|
94041:20190903:135558.754 6: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](dbconfig_thread+0x190) [0x563a472f3c44] |
|
94041:20190903:135558.754 5: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](zbx_thread_start+0x37) [0x563a473a2e1d] |
|
94041:20190903:135558.754 4: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0x83a) [0x563a472ec7c7] |
|
94041:20190903:135558.754 3: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](daemon_start+0x325) [0x563a473950d6] |
|
94041:20190903:135558.755 2: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](main+0x312) [0x563a472ebf8b] |
|
94041:20190903:135558.755 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f04bc19c495] |
|
94041:20190903:135558.755 0: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1567490158.213288 sec, syncing configuration](+0x346a9) [0x563a472e06a9] |
|
94036:20190903:135558.778 One child process died (PID:94041,exitcode/signal:1). Exiting ... |
|
94036:20190903:135600.783 syncing history data... |
|
94036:20190903:135600.784 syncing history data done |
|
94036:20190903:135600.784 syncing trend data... |
|
94036:20190903:135601.740 syncing trend data done |
|
94036:20190903:135601.741 Zabbix Server stopped. Zabbix 3.4.15 (revision 86739). |
|
110408:20190903:135611.861 Starting Zabbix Server. Zabbix 3.4.15 (revision 86739). |
|
110408:20190903:135611.862 ****** Enabled features ****** |
|
110408:20190903:135611.862 SNMP monitoring: YES |
|
110408:20190903:135611.862 IPMI monitoring: YES |
|
110408:20190903:135611.862 Web monitoring: YES |
|
110408:20190903:135611.862 VMware monitoring: YES |
|
110408:20190903:135611.862 SMTP authentication: YES |
|
110408:20190903:135611.862 Jabber notifications: YES |
|
110408:20190903:135611.863 Ez Texting notifications: YES |
|
110408:20190903:135611.863 ODBC: YES |
|
110408:20190903:135611.863 SSH2 support: YES |
|
110408:20190903:135611.863 IPv6 support: YES |
|
110408:20190903:135611.863 TLS support: YES |
|
110408:20190903:135611.863 ****************************** |
|
110408:20190903:135611.863 using configuration file: /etc/zabbix/zabbix_server.conf |
|
110408:20190903:135611.879 current database version (mandatory/optional): 03040000/03040007 |
|
110408:20190903:135611.879 required mandatory version: 03040000 |
|
110408:20190903:135612.819 __mem_malloc: skipped 0 asked 168 skip_min 4294967295 skip_max 0 |
|
110408:20190903:135612.819 [file:strpool.c,line:51] zbx_mem_malloc(): out of memory (requested 161 bytes) |
|
110408:20190903:135612.819 [file:strpool.c,line:51] zbx_mem_malloc(): please increase CacheSize configuration parameter |
|
110408:20190903:135612.819 === memory statistics for string pool === |
|
110408:20190903:135612.819 free chunks of size 48 bytes: 1 |
|
110408:20190903:135612.820 min chunk size: 48 bytes |
|
110408:20190903:135612.820 max chunk size: 48 bytes |
|
110408:20190903:135612.820 memory of total size 1257920 bytes fragmented into 13247 chunks |
|
110408:20190903:135612.820 of those, 48 bytes are in 1 free chunks |
|
110408:20190903:135612.820 of those, 1045936 bytes are in 13246 used chunks |
|
110408:20190903:135612.820 ================================ |
|
110408:20190903:135612.820 === Backtrace: === |
|
110408:20190903:135612.821 12: /usr/sbin/zabbix_server(zbx_backtrace+0x35) [0x55af9031b889] |
|
110408:20190903:135612.821 11: /usr/sbin/zabbix_server(__zbx_mem_malloc+0x163) [0x55af903177a8] |
|
110408:20190903:135612.821 10: /usr/sbin/zabbix_server(+0xe5eec) [0x55af90317eec] |
|
110408:20190903:135612.821 9: /usr/sbin/zabbix_server(zbx_hashset_insert_ext+0x2d4) [0x55af90320a46] |
|
110408:20190903:135612.821 8: /usr/sbin/zabbix_server(zbx_strpool_intern+0x5d) [0x55af903180ab] |
|
110408:20190903:135612.821 7: /usr/sbin/zabbix_server(+0xba9bb) [0x55af902ec9bb] |
|
110408:20190903:135612.822 6: /usr/sbin/zabbix_server(+0xc07a8) [0x55af902f27a8] |
|
110408:20190903:135612.822 5: /usr/sbin/zabbix_server(DCsync_configuration+0xc0f) [0x55af902f6452] |
|
110408:20190903:135612.822 4: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x5e8) [0x55af90272575] |
|
110408:20190903:135612.822 3: /usr/sbin/zabbix_server(daemon_start+0x325) [0x55af9031b0d6] |
|
110408:20190903:135612.822 2: /usr/sbin/zabbix_server(main+0x312) [0x55af90271f8b] |
|
110408:20190903:135612.822 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f123f79b495] |
|
110408:20190903:135612.822 0: /usr/sbin/zabbix_server(+0x346a9) [0x55af902666a9] |
|
110432:20190903:135623.090 Starting Zabbix Server. Zabbix 3.4.15 (revision 86739). |
|
110432:20190903:135623.090 ****** Enabled features ****** |
|
110432:20190903:135623.090 SNMP monitoring: YES |
|
110432:20190903:135623.090 IPMI monitoring: YES |
|
110432:20190903:135623.090 Web monitoring: YES |
|
110432:20190903:135623.090 VMware monitoring: YES |
|
110432:20190903:135623.090 SMTP authentication: YES |
|
110432:20190903:135623.090 Jabber notifications: YES |
|
110432:20190903:135623.091 Ez Texting notifications: YES |
|
110432:20190903:135623.092 ODBC: YES |
|
110432:20190903:135623.093 SSH2 support: YES |
|
110432:20190903:135623.093 IPv6 support: YES |
|
110432:20190903:135623.093 TLS support: YES |
|
110432:20190903:135623.094 ****************************** |
通过日志发现 zabbix-server 一直在反复重启。判断原因为 cachesize 缓存容量不够大。修改 zabbix_server.conf 配置文件，将 cachesize 设置成 2G
### Option: CacheSize |
|
# Size of configuration cache, in bytes. |
|
# Shared memory size for storing host, item and trigger data. |
|
# |
|
# Mandatory: no |
|
# Range: 128K-8G |
|
# Default: |
|
CacheSize=2G |
|
配置后发现问题解决。在使用一些较大模板（包含很多自动发现规则）时，建议调高 zabbix-server 缓存
zabbix 无法正常启动，错误 z3001, 日志如下
13080:20191118:104522.195 database is down: reconnecting in 10 seconds |
|
13080:20191118:104532.196 [Z3001] connection to database 'zabbix' failed: [0] could not connect to server: Connection refused |
|
Is the server running on host "localhost" (::1) and accepting |
|
TCP/IP connections on port 3306? |
|
received invalid response to SSL negotiation: R |
由于安装了 zabbix-server-pgsql，所以要求 ssl 连接，卸载了即可
[root@localhost zabbix-server-mysql-3.4.15]# rpm -qa |grep zabbix |
|
zabbix-release-3.4-2.el7.noarch |
|
zabbix-web-3.4.15-1.el7.noarch |
|
zabbix-server-pgsql-3.4.15-1.el7.x86_64 |
|
zabbix-server-mysql-3.4.15-1.el7.x86_64 |
|
zabbix-web-mysql-3.4.15-1.el7.noarch |
|
[root@localhost zabbix-server-mysql-3.4.15]# rpm -e zabbix-server-pgsql-3.4.15-1.el7.x86_64 |
zabbix4.4 中文乱码问题，4.4 版本的字体路径有改变
将字体上传到 /usr/share/zabbix/assets/fonts，修改 /usr/share/zabbix/assets/fonts
define('ZBX_GRAPH_FONT_NAME', 'simkai'); // font file name |
|
#修改67行，修改为simkai |
mysql limit 5000 ：mysql报错Changed limits: max_open_files: 5000 等 需要修改配置文件
# 修改 fs.file-max https://blog.csdn.net/qq_26614295/article/details/81502338
# echo 6553560 > /proc/sys/fs/file-max //sysctl -w "fs.file-max=34166"，前面2种重启机器后会恢复为默认值
vim /etc/sysctl.conf, 加入以下内容，重启生效
fs.file-max = 6553560原文：http://blog.itpub.net/20893244/viewspace-2152685/
echo ulimit -n 65535 >>/etc/profile
source /etc/profile #加载修改后的profile
ulimit -n #显示65535，修改完毕！cat /usr/lib/systemd/system/mysqld.service
# Copyright (c) 2015, 2016, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License, version 2.0,
# as published by the Free Software Foundation.
#
# This program is also distributed with certain software (including
# but not limited to OpenSSL) that is licensed under separate terms,
# as designated in a particular file or component or in included license
# documentation. The authors of MySQL hereby grant you an additional
# permission to link the program and your derivative works with the
# separately licensed software that they have included with MySQL.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License, version 2.0, for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
#
# systemd service file for MySQL forking server
#
[Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
Type=forking
PIDFile=/var/run/mysqld/mysqld.pid
# Disable service start and stop timeout logic of systemd for mysqld service.
TimeoutSec=0
# Execute pre and post scripts as root
PermissionsStartOnly=true
# Needed to create system tables
ExecStartPre=/usr/bin/mysqld_pre_systemd
# Start main service
ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS
# Use this to switch malloc implementation
EnvironmentFile=-/etc/sysconfig/mysql
# Sets open_files_limit
LimitNOFILE = 65535
Restart=on-failure
RestartPreventExitStatus=1
PrivateTmp=false
cat /etc/my.cnf |grep -v ^$ |grep -v ^#
[mysqld]
read_rnd_buffer_size = 2M
sort_buffer_size = 2M
join_buffer_size = 128M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
max_connections = 1000
max_connect_errors = 20000
connect_timeout = 60 #connect_timeout：默认为10S
interactive_timeout = 120
wait_timeout = 120
log_warnings=1
symbolic-links = yes
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
[mysqladmin]
user=root
password=PASSWD
[mysql]
user=zabbix
password=PASSWD
#
show variables like '%table_open_cache%';
set global table_open_cache=8192;