Linux 问题解决 :/lib/systemd/systemd-journald 占用内存过高

早上在用htop查看Linux系统状况时，发现/lib/systemd/systemd-journald占用了过高的内存。

然后我感觉是因为我的Java程序遇到问题阻塞了的原因，我先把JVM的内存dump出来：

jmap -dump:format=b,file=dump.hprof  pid

然后还没等我看详细情况，这个任务的占用就下去了

下次再出现这种情况时再查吧。

---

第二天又出现了这个问题，先使用ps查看/lib/systemd/systemd-journald的pid,我这里为14990，然后查看具体内容：

-> # ps -aux | grep systemd-journald
root     14990  0.0  8.6 242076 76732 ?        Ss   May08   2:39 /lib/systemd/systemd-journald
-> # systemctl status 14990
● systemd-journald.service - Journal Service
   Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
   Active: active (running) since Wed 2019-05-08 11:30:56 CST; 3 weeks 0 days ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 14990 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 55.4M
      CPU: 2min 35.456s
   CGroup: /system.slice/systemd-journald.service
           └─14990 /lib/systemd/systemd-journald

May 08 11:30:55 VM-73-203-debian systemd-journald[14990]: Journal started
May 08 11:30:55 VM-73-203-debian systemd-journald[14990]: System journal (/var/log/journal/db823f05086649f099c1146d5e128eeb) is 3.9G, max 4.0G, 35.
May 08 11:30:13 VM-73-203-debian systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
May 08 11:30:14 VM-73-203-debian systemd[1]: systemd-journald.service: Killing process 27382 (systemd-journal) with signal SIGABRT.

上面提到systemd-journald.service: Watchdog timeout，然后我进行搜索，发现大部分都是bug的汇报，从这里找不到解决方案。

---

然后我尝试查看内核信息：

-> # dmesg  
[14583522.160882] [ 5130]     0  5130     1069       22       8       3        0             0 sh
[14583522.172462] [ 5141]     0  5141    17488      193      36       3        0             0 sshd
[14583522.173601] Out of memory: Kill process 31761 (redis-server) score 330 or sacrifice child
[14583522.174813] Killed process 31761 (redis-server) total-vm:343960kB, anon-rss:302144kB, file-rss:0kB, shmem-rss:0kB
[15009615.723223] Adding 1048572k swap on /swapfile.  Priority:-1 extents:84 across:7541440k FS
[15015827.304120] TCP: request_sock_TCP: Possible SYN flooding on port 46835. Sending cookies.  Check SNMP counters.
[15015828.520053] TCP: request_sock_TCP: Possible SYN flooding on port 57441. Sending cookies.  Check SNMP counters.
[15026625.563121] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module.

这里可以看出问题出现在Redis上，重启Redis后解决了问题。

---

更多参考资料：

• The systemd journalctl Process Hogs A lot of Memory - Bootloader Blog
  GitHub上有2个相关的未关闭的issue，所以对Linux进行升级也许可以解决问题:
• Memory “leak” of systemd-journalctl to the point total memory usage · Issue #9141 · systemd/systemd
• systemd-journald: very large virt memory leak (heap) · Issue #11502 · systemd/systemd