502 BAD GATEWAY-k8s的cgroup限制了apache的可用内存

1.release的组件逻辑图

 

 

 

 

 2.表象：按F12,总是报502 BAD GATEWAY

3.nginx日志

[error] 8564#0: *348734 upstream prematurely closed connection while reading response header from upstream, client: 83.234.185.35, server: tqy.hangzhouali.net.cn

4.又nginx日志假设是后端apache的问题，由于以前服务是正常的，所以假设nginx配置也没问题，apache的规则也没问题

5.去看apache日志

Resource temporarily unavailable: AH03104: apr_thread_create: unable to create worker thread
httpd apache centos Cannot allocate memory: AH00481: fork: Unable to fork new process

看样子apache不能fork新的线程

6.去k8s的主机看系统日志

[  +0.000001] kmem: usage 111176kB, limit 9007199254740988kB, failcnt 0
[  +0.000001] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active
[  +0.000024] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4/b229faca56cfc16d36fcd6bc7d12ac01508830c3ecc01f1ba1e01d15d6db5946: cache:0KB rss:296KB rss_huge:0KB mapped_file:0KB 
[  +0.000009] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4/d9a5f7981ad793e525179a2b7ed1858a29e3a8e1852fbb6f02ac130774db56f8: cache:328264KB rss:75332KB rss_huge:4096KB mapped
[  +0.000007] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  +0.000061] [ 3105]     0  3105    66332       69       7        0          -998 pause
[  +0.000002] [ 3433]     0  3433    85369    83094     170        0           984 httpd
[  +0.000002] [ 3528]     0  3528      756      178       4        0           984 rotatelogs
[  +0.000002] [ 3529]     0  3529      758      170       4        0           984 rotatelogs
[  +0.000007] [16673]     0 16673      760      181       5        0           984 rotatelogs

[  +0.000001] Memory cgroup out of memory: Kill process 3433 (httpd) score 1618 or sacrifice child
[  +0.001943] Killed process 5346 (httpd) total-vm:1603408kB, anon-rss:6256kB, file-rss:980kB, shmem-rss:12kB
[  +1.721253] cbr0: port 9(vethbeabb887) entered disabled state
[  +0.003202] device vethbeabb887 left promiscuous mode

7.得出结论：apache的容器的内存溢出，被cgroup限制了，应该是在腾讯云哪里设置内存的地方设置错了

 

 一看，原来才给了512M.改为2048M后问题不再出现