腾讯云主机每天早上内存暴涨引发CPU满载问题

腾讯云主机每天早上内存暴涨引发CPU满载问题

系统：Debian9

问题，每天固定时间段，总会导致CPU占用100%，通过top发现kswapd0占用内存，本来内存1G就很小。

Linux设置大内存页解决kswapd0进程过渡消耗CPU的问题

通过系统日志发现是无人值守自动更新导致占用内存和CPU

系统日志

# 查看系统日志

Svr:~# tail -n 500 /var/log/syslog
Mar  1 06:35:59 StarMeow-Svr systemd[1]: Starting Daily apt upgrade and clean activities...
Mar  1 06:36:01 StarMeow-Svr CRON[15637]: (root) CMD (/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &)
# 省略多条日志
Mar  1 07:04:46 StarMeow-Svr systemd[1]: Started Daily apt upgrade and clean activities.
Mar  1 07:04:46 StarMeow-Svr systemd[1]: apt-daily-upgrade.timer: Adding 12min 54.290649s random time.
Mar  1 07:04:46 StarMeow-Svr systemd[1]: apt-daily-upgrade.timer: Adding 15min 4.419940s random time.

参考文档：https://superuser.com/questions/1327884/how-to-disable-daily-upgrade-and-clean-on-ubuntu-16-04

找到服务，参考 https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1615482 可以找到有关apt-daily ***说明，然后搜索到这个链接： https://wiki.debian.org/UnattendedUpgrades 讲解了Debian无人值守升级

查看apt-daily计划

Svr:/etc/systemd/system/timers.target.wants# pwd
/etc/systemd/system/timers.target.wants
Svr:/etc/systemd/system/timers.target.wants# ls
apt-daily.timer  apt-daily-upgrade.timer

# 查看文件，这些文件都是/lib/systemd/system/对应文件软连接过来的！
# 用于下载
Svr:/etc/systemd/system/timers.target.wants# cat apt-daily.timer 

[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

# 用于升级
Svr:/etc/systemd/system/timers.target.wants# cat apt-daily-upgrade.timer 

[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

解决方法

Svr:/etc/systemd/system/timers.target.wants
# 移除安装软件，没尝试
apt-get remove unattended-upgrades

# 停止
Svr:/etc/systemd/system/timers.target.wants# systemctl stop apt-daily-upgrade.timer

# 禁止，会在本地删除该文件
Svr:/etc/systemd/system/timers.target.wants# systemctl disable apt-daily-upgrade.timer
Removed /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer.

# 重启
Svr:/etc/systemd/system/timers.target.wants# systemctl daemon-reload

Svr:/lib/systemd/system# ls -l | grep 'apt-daily'
-rw-r--r-- 1 root root  225 Jan 18 18:42 apt-daily.service
-rw-r--r-- 1 root root  156 Jan 18 18:42 apt-daily.timer
-rw-r--r-- 1 root root  238 Jan 18 18:42 apt-daily-upgrade.service
-rw-r--r-- 1 root root  184 Mar  1 11:02 apt-daily-upgrade.timer
-rw-r--r-- 1 root root  184 Jan 18 18:42 apt-daily-upgrade.timer.bak2019.03.01

恢复每日更新启动

用法：ln -s 源文件 目标文件

Svr:/etc/systemd/system/timers.target.wants# ln -s /lib/systemd/system/apt-daily-upgrade.timer apt-daily-upgrade.timer 
Svr:/etc/systemd/system/timers.target.wants# ls -li
total 0
262474 lrwxrwxrwx 1 root root 35 Jul 15  2017 apt-daily.timer -> /lib/systemd/system/apt-daily.timer
262252 lrwxrwxrwx 1 root root 43 Mar  1 11:04 apt-daily-upgrade.timer -> /lib/systemd/system/apt-daily-upgrade.timer

# 恢复回来后，需要使用下面命令重新加载
Svr:/etc/systemd/system/timers.target.wants# systemctl daemon-reload