2018-11-09 tcpdump辅助排查gt系统api接口反应延迟问题

  1. 快下班了,gt系统收到商务和运营反馈说批量创建某些广告时页面始终在转圈圈。
  2. 前后端一起定位了半天问题,并不能找到原因。测试环境测试没有问题的。
    3.我这边看slb流量带宽使用达到50M了,由于成本更合适,我们采用了带宽付费模式。我担心是否是因为出口流量达到带宽最大值就被丢弃了呢?
  3. 为了验证上述假设,直接拿到api容器对应的宿主机的ip和端口。然后在前端打开f12,按照原来方式触发api请求,将network中的请求复制为curl.
  4. 终端下使用api容器对应的宿主机ip和端口使用curl触发,同时在宿主机启动tcpdump:
[mextest@gdswarm2 ~]$ sudo tcpdump -n -A -X src host 119.57.68.20 and dst port 32953 and tcp[13]=24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:45:59.948730 IP 119.57.68.20.65000 > 172.17.0.193.32953: Flags [P.], seq 1669645032:1669645949, ack 390157184, win 259, length 917
        0x0000:  4514 03bd 387e 4000 3406 a289 7739 4414  [email protected].
        0x0010:  ac11 00c1 fde8 80b9 6384 c2e8 1741 5380  ........c....AS.
        0x0020:  5018 0103 ba54 0000 504f 5354 202f 6170  P....T..POST./ap
        0x0030:  692f 6361 6d70 6169 676e 7320 4854 5450  i/campaigns.HTTP
        0x0040:  2f31 2e31 0d0a 486f 7374 3a20 3437 2e39  /1.1..Host:.47.9
        0x0050:  352e 3232 322e 3734 3a33 3239 3533 0d0a  5.222.74:32953..
        0x0060:  4163 6365 7074 2d45 6e63 6f64 696e 673a  Accept-Encoding:
        0x0070:  2064 6566 6c61 7465 2c20 677a 6970 0d0a  .deflate,.gzip..
        0x0080:  4163 6365 7074 3a20 6170 706c 6963 6174  Accept:.applicat
        0x0090:  696f 6e2f 6a73 6f6e 2c20 7465 7874 2f70  ion/json,.text/p
        0x00a0:  6c61 696e 2c20 2a2f 2a0d 0a52 6566 6572  lain,.*/*..Refer
        0x00b0:  6572 3a20 6874 7470 733a 2f2f 6765 656b  er:.https://geek
        0x00c0:  2e67 6474 7261 6666 6963 6164 2e63 6f6d  .gdtrafficad.com
        0x00d0:  2f63 7265 6174 4144 4361 6d70 6169 676e  /creatADCampaign
        0x00e0:  2f63 7265 6174 4361 6d70 6169 676e 0d0a  /creatCampaign..
        0x00f0:  4f72 6967 696e 3a20 6874 7470 733a 2f2f  Origin:.https://
        0x0100:  6765 656b 2e67 6474 7261 6666 6963 6164  geek.gdtrafficad
        0x0110:  2e63 6f6d 0d0a 5573 6572 2d41 6765 6e74  .com..User-Agent
        0x0120:  3a20 4d6f 7a69 6c6c 612f 352e 3020 284d  :.Mozilla/5.0.(M
        0x0130:  6163 696e 746f 7368 3b20 496e 7465 6c20  acintosh;.Intel.
        0x0140:  4d61 6320 4f53 2058 2031 305f 3134 5f31  Mac.OS.X.10_14_1
        0x0150:  2920 4170 706c 6557 6562 4b69 742f 3533  ).AppleWebKit/53
        0x0160:  372e 3336 2028 4b48 544d 4c2c 206c 696b  7.36.(KHTML,.lik
        0x0170:  6520 4765 636b 6f29 2043 6872 6f6d 652f  e.Gecko).Chrome/
        0x0180:  3730 2e30 2e33 3533 382e 3737 2053 6166  70.0.3538.77.Saf
        0x0190:  6172 692f 3533 372e 3336 0d0a 4175 7468  ari/537.36..Auth
        0x01a0:  6f72 697a 6174 696f 6e3a 2042 6561 7265  orization:.Beare
        0x01b0:  7220 6579 4a68 6247 6369 4f69 4a49 557a  r.eyJhbGciOiJIUz
        0x01c0:  5578 4d69 4a39 2e65 794a 7a64 5749 694f  UxMiJ9.eyJzdWIiO
        0x01d0:  694a 6d59 5735 6e62 576c 7565 5856 6c51  iJmYW5nbWlueXVlQ
        0x01e0:  4746 6b62 576c 754c 6d4e 7662 5349 7349  GFkbWluLmNvbSIsI
        0x01f0:  6d46 3164 4767 694f 694a 5354 3078 4658  mF1dGgiOiJST0xFX
        0x0200:  3046 4852 5535 5558 3046 4554 556c 4f4c  0FHRU5UX0FETUlOL
        0x0210:  464a 5054 4556 6656 564e 4655 6949 7349  FJPTEVfVVNFUiIsI
        0x0220:  6d56 3463 4349 364d 5455 304e 4445 344e  mV4cCI6MTU0NDE4N
        0x0230:  444d 354e 6e30 2e55 4753 6242 7462 5458  DM5Nn0.UGSbBtbTX
        0x0240:  6b34 2d4a 3265 7074 6f69 4772 4966 7074  k4-J2eptoiGrIfpt
        0x0250:  324b 5a4b 7a64 5a4c 7a5a 394d 767a 4f54  2KZKzdZLzZ9MvzOT
        0x0260:  3647 4c66 376e 4a78 7a44 4441 3942 3942  6GLf7nJxzDDA9B9B
        0x0270:  7a4b 6475 4e79 4a30 6a30 434d 6767 7549  zKduNyJ0j0CMgguI
        0x0280:  7955 3067 5837 564d 5f37 4f44 410d 0a43  yU0gX7VM_7ODA..C
        0x0290:  6f6e 7465 6e74 2d54 7970 653a 2061 7070  ontent-Type:.app
        0x02a0:  6c69 6361 7469 6f6e 2f6a 736f 6e3b 6368  lication/json;ch
        0x02b0:  6172 7365 743d 5554 462d 380d 0a43 6f6e  arset=UTF-8..Con
        0x02c0:  7465 6e74 2d4c 656e 6774 683a 2032 3333  tent-Length:.233
        0x02d0:  0d0a 0d0a 7b22 6164 7665 7274 6973 6572  ....{"advertiser
        0x02e0:  4964 223a 3334 2c22 7072 6f64 7563 7449  Id":34,"productI
        0x02f0:  6422 3a32 382c 226e 616d 6522 3a22 e6b3  d":28,"name":"..
        0x0300:  a2e5 a587 e58a a8e7 89a9 5f41 6e64 726f  .........._Andro
        0x0310:  6964 222c 2261 6374 6976 6550 7269 6365  id","activePrice
        0x0320:  223a 2231 3522 2c22 746f 7461 6c42 7564  ":"15","totalBud
        0x0330:  6765 7422 3a22 3130 3030 3030 222c 2264  get":"100000","d
        0x0340:  6179 4275 6467 6574 223a 2231 3030 3030  ayBudget":"10000
        0x0350:  222c 2264 6174 6522 3a5b 2232 3031 382d  ","date":["2018-
        0x0360:  3131 2d30 3122 2c22 3230 3138 2d31 312d  11-01","2018-11-
        0x0370:  3330 225d 2c22 6265 6769 6e44 6174 6522  30"],"beginDate"
        0x0380:  3a22 3230 3138 2d31 312d 3031 222c 2265  :"2018-11-01","e
        0x0390:  6e64 4461 7465 223a 2232 3031 382d 3131  ndDate":"2018-11
        0x03a0:  2d33 3022 2c22 6d65 6469 6154 7970 6522  -30","mediaType"
        0x03b0:  3a22 4d45 4449 415f 4255 5922 7d         :"MEDIA_BUY"}

  1. 看到上述对应的ip和端口的信息后,使用指定端口,在宿主机启动新的tcpdump来监控何时输出响应的:
[mextest@gdswarm2 ~]$ sudo tcpdump -n -A -X dst host 119.57.68.20 and dst port 65000 and tcp[13]=24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:47:38.317085 IP 172.17.0.193.32953 > 119.57.68.20.65000: Flags [P.], seq 390157184:390158314, ack 1669645949, win 243, length 1130
        0x0000:  4500 0492 3fb2 4000 3f06 8f94 ac11 00c1  E...?.@.?.......
        0x0010:  7739 4414 80b9 fde8 1741 5380 6384 c67d  w9D......AS.c..}
        0x0020:  5018 00f3 6ca4 0000 4854 5450 2f31 2e31  P...l...HTTP/1.1
        0x0030:  2032 3031 2043 7265 6174 6564 0d0a 4578  .201.Created..Ex
        0x0040:  7069 7265 733a 2030 0d0a 4361 6368 652d  pires:.0..Cache-
        0x0050:  436f 6e74 726f 6c3a 206e 6f2d 6361 6368  Control:.no-cach
        0x0060:  652c 206e 6f2d 7374 6f72 652c 206d 6178  e,.no-store,.max
        0x0070:  2d61 6765 3d30 2c20 6d75 7374 2d72 6576  -age=0,.must-rev
        0x0080:  616c 6964 6174 650d 0a58 2d58 5353 2d50  alidate..X-XSS-P
        0x0090:  726f 7465 6374 696f 6e3a 2031 3b20 6d6f  rotection:.1;.mo
        0x00a0:  6465 3d62 6c6f 636b 0d0a 5072 6167 6d61  de=block..Pragma
        0x00b0:  3a20 6e6f 2d63 6163 6865 0d0a 4c6f 6361  :.no-cache..Loca
        0x00c0:  7469 6f6e 3a20 2f61 7069 2f63 616d 7061  tion:./api/campa
        0x00d0:  6967 6e73 2f37 380d 0a44 6174 653a 2054  igns/78..Date:.T
        0x00e0:  6875 2c20 3038 204e 6f76 2032 3031 3820  hu,.08.Nov.2018.
        0x00f0:  3130 3a34 373a 3338 2047 4d54 0d0a 436f  10:47:38.GMT..Co
        0x0100:  6e6e 6563 7469 6f6e 3a20 6b65 6570 2d61  nnection:.keep-a
        0x0110:  6c69 7665 0d0a 582d 6774 4d67 7241 7070  live..X-gtMgrApp
        0x0120:  2d70 6172 616d 733a 2037 380d 0a41 6363  -params:.78..Acc
        0x0130:  6573 732d 436f 6e74 726f 6c2d 4578 706f  ess-Control-Expo
        0x0140:  7365 2d48 6561 6465 7273 3a20 4175 7468  se-Headers:.Auth
        0x0150:  6f72 697a 6174 696f 6e0d 0a58 2d67 744d  orization..X-gtM
        0x0160:  6772 4170 702d 616c 6572 743a 2067 744d  grApp-alert:.gtM
        0x0170:  6772 4170 702e 6361 6d70 6169 676e 2e63  grApp.campaign.c
        0x0180:  7265 6174 6564 0d0a 4163 6365 7373 2d43  reated..Access-C
        0x0190:  6f6e 7472 6f6c 2d41 6c6c 6f77 2d4f 7269  ontrol-Allow-Ori
        0x01a0:  6769 6e3a 2068 7474 7073 3a2f 2f67 6565  gin:.https://gee
        0x01b0:  6b2e 6764 7472 6166 6669 6361 642e 636f  k.gdtrafficad.co
        0x01c0:  6d0d 0a56 6172 793a 204f 7269 6769 6e0d  m..Vary:.Origin.
        0x01d0:  0a41 6363 6573 732d 436f 6e74 726f 6c2d  .Access-Control-
        0x01e0:  416c 6c6f 772d 4372 6564 656e 7469 616c  Allow-Credential
        0x01f0:  733a 2074 7275 650d 0a58 2d43 6f6e 7465  s:.true..X-Conte
        0x0200:  6e74 2d54 7970 652d 4f70 7469 6f6e 733a  nt-Type-Options:
        0x0210:  206e 6f73 6e69 6666 0d0a 436f 6e74 656e  .nosniff..Conten
        0x0220:  742d 5479 7065 3a20 6170 706c 6963 6174  t-Type:.applicat
        0x0230:  696f 6e2f 6a73 6f6e 3b63 6861 7273 6574  ion/json;charset
        0x0240:  3d55 5446 2d38 0d0a 436f 6e74 656e 742d  =UTF-8..Content-
        0x0250:  4c65 6e67 7468 3a20 3530 310d 0a58 2d41  Length:.501..X-A
        0x0260:  7070 6c69 6361 7469 6f6e 2d43 6f6e 7465  pplication-Conte
        0x0270:  7874 3a20 6774 2d6d 6772 3a70 726f 642c  xt:.gt-mgr:prod,
        0x0280:  7377 6167 6765 722c 6e6f 2d6c 6971 7569  swagger,no-liqui
        0x0290:  6261 7365 3a39 3839 380d 0a0d 0a7b 2261  base:9898....{"a
        0x02a0:  6374 6976 6550 7269 6365 223a 3135 2c22  ctivePrice":15,"
        0x02b0:  6164 7665 7274 6973 6572 223a 7b22 6368  advertiser":{"ch
        0x02c0:  6563 6b53 7461 7475 7322 3a5b 7b22 6368  eckStatus":[{"ch
        0x02d0:  6563 6b53 7461 7475 7322 3a22 696e 6974  eckStatus":"init
        0x02e0:  222c 226d 6564 6961 223a 2257 4158 222c  ","media":"WAX",
        0x02f0:  226d 7367 223a 22e6 9caa e980 81e5 aea1  "msg":".........
        0x0300:  222c 226f 7468 6572 7322 3a7b 7d7d 5d2c  ","others":{}}],
        0x0310:  2269 6422 3a33 347d 2c22 6164 7665 7274  "id":34},"advert
        0x0320:  6973 6572 4964 223a 3334 2c22 6265 6769  iserId":34,"begi
        0x0330:  6e44 6174 6522 3a22 3230 3138 2d31 312d  nDate":"2018-11-
        0x0340:  3031 222c 2263 7265 6174 6542 7922 3a31  01","createBy":1
        0x0350:  372c 2263 7265 6174 6544 6174 6522 3a22  7,"createDate":"
        0x0360:  3230 3138 2d31 312d 3038 5431 303a 3435  2018-11-08T10:45
        0x0370:  3a35 392e 3936 315a 222c 2264 6179 4275  :59.961Z","dayBu
        0x0380:  6467 6574 223a 3130 3030 302c 2264 656c  dget":10000,"del
        0x0390:  466c 6167 223a 224f 4e22 2c22 656e 6444  Flag":"ON","endD
        0x03a0:  6174 6522 3a22 3230 3138 2d31 312d 3330  ate":"2018-11-30
        0x03b0:  222c 2268 6973 746f 7279 4c69 7374 223a  ","historyList":
        0x03c0:  5b5d 2c22 6964 223a 3738 2c22 6d65 6469  [],"id":78,"medi
        0x03d0:  614c 6973 7422 3a5b 5d2c 226d 6564 6961  aList":[],"media
        0x03e0:  5479 7065 223a 224d 4544 4941 5f42 5559  Type":"MEDIA_BUY
        0x03f0:  222c 226e 616d 6522 3a22 e6b3 a2e5 a587  ","name":"......
        0x0400:  e58a a8e7 89a9 5f41 6e64 726f 6964 222c  ......_Android",
        0x0410:  2270 726f 6475 6374 223a 7b22 6964 223a  "product":{"id":
        0x0420:  3238 7d2c 2270 726f 6475 6374 4964 223a  28},"productId":
        0x0430:  3238 2c22 7374 6174 7573 223a 224e 4f5f  28,"status":"NO_
        0x0440:  5245 4144 5922 2c22 746f 7461 6c42 7564  READY","totalBud
        0x0450:  6765 7422 3a31 3030 3030 302c 2275 7064  get":100000,"upd
        0x0460:  6174 6542 7922 3a31 372c 2275 7064 6174  ateBy":17,"updat
        0x0470:  6544 6174 6522 3a22 3230 3138 2d31 312d  eDate":"2018-11-
        0x0480:  3038 5431 303a 3435 3a35 392e 3936 315a  08T10:45:59.961Z
        0x0490:  227d      
  1. 可以看到请求时间是18:45:59.948730 IP 119.57.68.20.65000 > 172.17.0.193.32953,响应时间是18:47:38.317085 IP 172.17.0.193.32953 > 119.57.68.20.65000

  2. 由此证明了延迟并不是slb或者公用网络原因导致的。因为tcpdump在宿主机上执行的。问题处在api代码层。再仔细看响应中的updateDate时间和发出请求的时间基本一致。说明写入数据库很快就完成了。但是为何返回结果却延迟了一分钟呢?

  3. 至此,我们体会了一次tcpdump为我们解忧:

 sudo tcpdump -n -A -X src host 119.57.68.20 and dst port 32953 and tcp[13]=24
sudo tcpdump -n -A -X dst host 119.57.68.20 and dst port 65000 and tcp[13]=24

你可能感兴趣的:(2018-11-09 tcpdump辅助排查gt系统api接口反应延迟问题)