记一次 go 服务内存泄漏
pprof inuse_space 正常
一般查看内存是否泄漏，看下 inuse_space 下的函数占用是否符合预期即可定位
pprof goroutine 数量正常
其次，常见的协程泄漏也是一大内存泄漏常见问题
debug.FreeOSMemory() 释放不掉
因此初步定位，是否是 golang GC 持有了内存尚未归还系统
结果无效
使用 pmap 查看 go 进程内存占用情况
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 20568 12460 0 r-x-- game
0000000001816000 22524 14816 0 r---- game
0000000002e15000 1272 448 200 rw--- game
0000000002f53000 412 280 280 rw--- [ anon ]
000000c000000000 1146880 722488 722488 rw--- [ anon ]
000000c046000000 32768 0 0 ----- [ anon ]
00007f51ed034000 8227428 8223952 8223952 rw--- [ anon ]
00007f53e32d4000 11148 11092 11092 rw--- [ anon ]
00007f53e3dc1000 7432 7396 7396 rw--- [ anon ]
00007f53e4504000 5380 5360 5360 rw--- [ anon ]
00007f53e4a4b000 7948 7936 7936 rw--- [ anon ]
00007f53e5212000 1088 1084 1084 rw--- [ anon ]
00007f53e532a000 47176 15380 15380 rw--- [ anon ]
00007f53e813c000 263680 0 0 ----- [ anon ]
00007f53f82bc000 4 4 4 rw--- [ anon ]
00007f53f82bd000 293564 0 0 ----- [ anon ]
00007f540a16c000 4 4 4 rw--- [ anon ]
00007f540a16d000 36692 0 0 ----- [ anon ]
00007f540c542000 4 4 4 rw--- [ anon ]
00007f540c543000 4580 0 0 ----- [ anon ]
00007f540c9bc000 4 4 4 rw--- [ anon ]
00007f540c9bd000 508 0 0 ----- [ anon ]
00007f540ca3c000 384 72 72 rw--- [ anon ]
00007ffe19062000 132 16 16 rw--- [ stack ]
00007ffe191bc000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------- ------- -------
total kB 10131592 9022800 8995272
- 000000c000000000 地址段，被 anon 分配
- 00007f51ed034000 地址段，被 anon 分配
pprof 查看 ReadMemStats 信息
# runtime.MemStats
# Alloc = 511215184
# TotalAlloc = 2387040868864
# Sys = 9002755864
# Lookups = 0
# Mallocs = 62979597032
# Frees = 62974616503
# HeapAlloc = 511215184
# HeapSys = 1170604032
# HeapIdle = 566894592
# HeapInuse = 603709440
# HeapReleased = 407527424
# HeapObjects = 4980529
# Stack = 3801088 / 3801088
# MSpan = 7122184 / 14925824
# MCache = 19200 / 32768
# BuckHashSys = 17712073
# GCSys = 52772768
# OtherSys = 7742907311
# NextGC = 666518528
# LastGC = 1650076291400621539
# PauseNs = [84573 100742 95422 86156 133630 119480 157597 95889 102826 130018 142217 121599 101298 110370 82708 104168 138280 117345 107443 86141 127083 90278 170451 116930 107839 170532 140186 111303 117594 125248 99441 106593 148126 107836 107129 127699 163856 92907 125739 119035 166194 127290 102103 140246 136436 87921 145587 126129 111681 156016 118715 91237 99338 108100 102268 106736 113862 105178 107456 88706 132915 128570 117247 106522 283094 91266 131885 105878 184512 95340 130203 94400 87651 110036 88427 109461 147071 144883 100193 94076 103126 123305 96265 119695 100927 138258 110019 104490 117543 128856 149506 127457 112336 157489 164125 112896 116319 95642 99765 119875 113373 99349 152705 118192 105129 92584 103465 137815 110037 138332 124392 125516 95171 138386 83369 110704 105518 113375 99595 94450 105359 120784 110603 120853 104693 109312 94535 104162 148201 112623 96646 110248 95551 92161 139134 113652 100086 89162 148071 106154 116013 97700 176868 82888 166987 82671 124975 93754 155597 109755 111886 90802 113215 89200 126413 88021 211183 111504 112437 155757 170727 89700 95937 109558 93163 138129 142533 84487 144022 133514 109569 109482 123691 85565 104291 122424 126503 92026 98294 124989 118067 86417 121080 116200 140696 104853 127884 88646 168958 132182 96966 84681 117802 104542 170031 163697 139453 104788 105676 122178 110008 148136 109332 104812 111864 124278 92567 130310 128673 116356 84261 98960 119525 94409 107832 90657 112062 102394 138133 90932 89751 134503 113936 282115 123879 114994 173142 117095 115490 148489 122029 116195 99539 127257 109065 124970 112145 87292 108048 109239 103802 110349 125632 88913 86671 111746 123310 103824 120306 116051 113493 134173 128964 129422 110302 150610]
# PauseEnd = [1650074907793309572 1650074913396918506 1650074920335162142 1650074925144468319 1650074932859012721 1650074942313341929 1650074949267112279 1650074958158122715 1650074968497424179 1650074976977695429 1650074983047976731 1650074992326277418 1650075000466416580 1650075009571331404 1650075015416775747 1650075020816785866 1650075029648228278 1650075035962783305 1650075041239596075 1650075047802456347 1650075051025438953 1650075057464909094 1650075062132772014 1650075069174660988 1650075073698554039 1650075077854827956 1650075081855064317 1650075089265349478 1650075095582167801 1650075103384351383 1650075112140630038 1650075120441048580 1650075126897575991 1650075132972721862 1650075139291965124 1650075145717409431 1650075150425996576 1650075158108693105 1650075163891519762 1650075170550129583 1650075173783531053 1650075179734785897 1650075185427019298 1650075191943856019 1650075197782666090 1650075205099547536 1650075211716083540 1650075218788580446 1650075230296093984 1650075242747655486 1650075252531606796 1650075260405515358 1650075268030584405 1650075275457451399 1650075282168453222 1650075288464604185 1650075294152377708 1650075303357424440 1650075308571992189 1650075315954899507 1650075323323421111 1650075327930647133 1650075335041528713 1650075342058140673 1650075347821698784 1650075354060050093 1650075360998888077 1650075367358475977 1650075376938175863 1650075385196604867 1650075389608187714 1650075395723426955 1650075399353796334 1650075404764257394 1650075412206459135 1650075416012550893 1650075424032614155 1650075430061064275 1650075436561721678 1650075442260280854 1650075448407312455 1650075455134500522 1650075461441708660 1650075466379846550 1650075472338926587 1650075476417986209 1650075482286980295 1650075485949928480 1650075491693563081 1650075496035885877 1650075502561442678 1650075509265879193 1650075516312382817 1650075522612904799 1650075529082181383 1650075535078814859 1650075539311155086 1650075543498765158 1650075550163090467 1650075554840800614 1650075558349293260 1650075565270964589 1650075570082731330 1650075575398421068 1650075579107139958 1650075583432754351 1650075589349150220 1650075593284968929 1650075598143653024 1650075604986251736 1650075610123010468 1650075616462004668 1650075623162178750 1650075628762453094 1650075635541321714 1650075639978824427 1650075645526848159 1650075649717315184 1650075654904989019 1650075661091680021 1650075667701965578 1650075672952135485 1650075678929796029 1650075684317004206 1650075689615440376 1650075695166510805 1650075700342853595 1650075708143313840 1650075714937621628 1650075722138984261 1650075726100218942 1650075731212669343 1650075737813614408 1650075744184247666 1650075750870552271 1650075756855846420 1650075762922551898 1650075767502054049 1650075773693072132 1650075782776573349 1650075789738459884 1650075795048810044 1650075802652759193 1650075813435872500 1650075820547061359 1650075826630972381 1650075835884919477 1650075842289328105 1650075849780925956 1650075858911889919 1650075868361710796 1650075876961367497 1650075885116735428 1650075894711754177 1650075902706689569 1650075911818750167 1650075917709117862 1650075925412125148 1650075933269604301 1650075942535736379 1650075950906683931 1650075959629794746 1650075967933305065 1650075976420339513 1650075985003712420 1650075990927711203 1650075999499439566 1650076008267453502 1650076013126423888 1650076017573219249 1650076024722877957 1650076029030522129 1650076035078471553 1650076041062368331 1650076047220887524 1650076054901562261 1650076061438885212 1650076066932238323 1650076072621984478 1650076078227546006 1650076083811487083 1650076088210648290 1650076094023054178 1650076099236088290 1650076102643577821 1650076108559700671 1650076113498345196 1650076118079665695 1650076124658325703 1650076130404544760 1650076135625048270 1650076141557619643 1650076147381981230 1650076152353995521 1650076158320702886 1650076164726445404 1650076171474477917 1650076180962701605 1650076189310411730 1650076198227179126 1650076203707684313 1650076211074630193 1650076219552644247 1650076228433773048 1650076240557865349 1650076252071306308 1650076260685924321 1650076265939017102 1650076272904789470 1650076278353756276 1650076285884915105 1650076291400621539 1650074613831432503 1650074621127669860 1650074625924765096 1650074632414145229 1650074637432545721 1650074642532722198 1650074647104951876 1650074651310030412 1650074657406936541 1650074664651456814 1650074670131129335 1650074676358206000 1650074682131143254 1650074687943923189 1650074695262720395 1650074699137761675 1650074704196859062 1650074711392103979 1650074715514289092 1650074720236524869 1650074724304349366 1650074731029428411 1650074738791414632 1650074748059746769 1650074753621874361 1650074762848227590 1650074767931946167 1650074776701506995 1650074784450433035 1650074793371644164 1650074799343160699 1650074806146194070 1650074812018014008 1650074823385075918 1650074831899319080 1650074842018092210 1650074851549454755 1650074859345790034 1650074868042891953 1650074875485060840 1650074882161588109 1650074886686383040 1650074893767341442 1650074900079577500]
# NumGC = 7380
# NumForcedGC = 26
# GCCPUFraction = 0.0012668645429657324
# DebugGC = false
# MaxRSS = 8586375168
仔细看，确定 OtherSys = 7742907311 OtherSys 分配持有了大量内存
看下 OtherSys 的解释：
// OtherSys is bytes of memory in miscellaneous off-heap
// runtime allocations.
OtherSys 是 golang 不在 heap 内的一些杂项功能的运行时分配
有点懵，golang 自己出问题了？
runtime.StartTrace
最后经排查，代码不规范，在某些情况下，调用了runtime.StartTrace而未调用runtime.StopTrace导致 golang 自身一直在分配内存
而这些内存，不会在 pprof inuse_space 中可以看到
看下 runtime.StartTrace 的注释：
// StartTrace enables tracing for the current process.
// While tracing, the data will be buffered and available via ReadTrace.
// StartTrace returns an error if tracing is already enabled.
// Most clients should use the runtime/trace package or the testing package’s
// -test.trace flag instead of calling StartTrace directly.
While tracing, the data will be buffered tracing 期间，会一直 buffered ，而这些分配的内存会到 off-heap ，不算在 heap 上。导致 pprof 失灵
其他
最后，默默的 XX OO 下这写样代码的同学，随手写下，后面是大量排查成本
敲下陌生 API 时，应当本能 2 个反应：
- 如何处理销毁逻辑（类似 c++ 程序，声明一个指针，脑子的冒出的应该是，这个指针在哪里处理释放）
- 这个 API 执行期间对程序有何影响
以上