页面性能监控
很多公司都会做页面性能的检测，做的方法也非常多。其实比较简单的是利用js去做，js可以很方便的调取浏览器的api，获取network里的相关信息，这个资料还是比较多的。唯一的难点在于如何注入js脚本，业内的方法的普遍有两种
- 利用selenium的execute_script，注入js脚本
- 在nginx层加入subfilter配置，注入js脚本
这个具体的实现，可以去调研，暂时没有实践过。
不注入js脚本的前提下，如何获取页面数据
组内对于以上两种方法，认为注入js对代码的侵入性太大，不是很客观，所以一直在搜索其他的实现方法，目前能想到的也就只有通过selenium和chromedriver，看能否从里面提取一些页面请求信息。本文只是记录一下调研的过程。
在stackoverflow上搜到文章 Browser performance tests through selenium，里面给了一个例子，是可以获取到相关日志，日志的量非常庞大
import json
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.CHROME
caps['loggingPrefs'] = {'performance': 'ALL'}
driver = webdriver.Chrome(desired_capabilities=caps)
driver.get('https://www.baidu.com')
logs = [json.loads(log['message'])['message'] for log in driver.get_log('performance')]
with open('devtools.json', 'wb') as f:
json.dump(logs, f)
driver.close()
试了一下，可以拉起pc端的chrome，但我需要在安卓机上跑，所以需要更改下chrome的配置，代码如下
https://sites.google.com/a/chromium.org/chromedriver/capabilities
https://github.com/webdriverio/webdriverio/issues/476
from selenium import webdriver
from pprint import pprint
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.CHROME
caps['loggingPrefs'] = {'performance': 'ALL'}
options = webdriver.ChromeOptions()
options.add_experimental_option('androidPackage', 'com.android.chrome')
options.add_experimental_option('perfLoggingPrefs',{'enableNetwork': True})
driver = webdriver.Chrome(chrome_options=options,desired_capabilities=caps)
driver.get('https://www.baidu.com')
log=driver.get_log('performance')
pprint(log)
拿到log后，接下来就可以分析这些日志了，日志非常多，格式化后有三千多行。
首先最外层是个大数组，包含若干字典格式的数据，如下：
[
{'level': 'INFO',
'message': '{"message":...}',
'timestamp': 1515665118083},
{'level': 'INFO',
'message': '{"message":...}',
'timestamp': 1515665118083}
]
在这一层没提取到什么信息，最主要的信息都是在message字段里，是一个非常长的json字符串，message字段里又包含message和webview字段，格式化后如下：
{'message': {'method': 'Network.requestWillBeSent',
'params': {'documentURL': 'https://www.baidu.com/',
'frameId': '9616.1',
'initiator': {'type': 'other'},
'loaderId': '9616.4',
'request': {'headers': {'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 '
'(Linux; Android '
'4.4.4; SM-A7000 '
'Build/KTU84P) '
'AppleWebKit/537.36 '
'(KHTML, like '
'Gecko) '
'Chrome/58.0.3029.83 '
'Mobile '
'Safari/537.36'},
'initialPriority': 'VeryHigh',
'method': 'GET',
'mixedContentType': 'none',
'referrerPolicy': 'no-referrer-when-downgrade',
'url': 'https://www.baidu.com/'},
'requestId': '9616.88',
'timestamp': 6117.672547,
'type': 'Document',
'wallTime': 1515665221.86887}},
'webview': '0'}
我们再拆解message['message']字段，里面一般会包含method和params两项。methods主要有以下这些：
Network.requestWillBeSent
Network.responseReceived
Network.dataReceived # 触发多次
Network.loadingFinished
各个阶段的返回信息如下：
Page.frameStartedLoading
Network.requestWillBeSent
Network.responseReceived
Network.dataReceived
Page.frameNavigated
Network.requestServedFromCache
Network.loadingFinished
Network.resourceChangedPriority
Page.domContentEventFired
Network.loadingFailed
Page.loadEventFired
Page.frameStoppedLoading
其中一个正常的请求大致需要经历以下过程：
{'method': 'Network.requestWillBeSent',
'params': {'documentURL': 'https://www.baidu.com/',
'frameId': '9616.1',
'initiator': {'type': 'other'},
'loaderId': '9616.4',
'request': {'headers': {'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Linux; Android '
'4.4.4; SM-A7000 '
'Build/KTU84P) '
'AppleWebKit/537.36 (KHTML, '
'like Gecko) '
'Chrome/58.0.3029.83 Mobile '
'Safari/537.36'},
'initialPriority': 'VeryHigh',
'method': 'GET',
'mixedContentType': 'none',
'referrerPolicy': 'no-referrer-when-downgrade',
'url': 'https://www.baidu.com/'},
'requestId': '9616.88',
'timestamp': 6117.672547,
'type': 'Document',
'wallTime': 1515665221.86887}}
{'method': 'Network.responseReceived',
'params': {'frameId': '9616.1',
'loaderId': '9616.4',
'requestId': '9616.88',
'response': {'connectionId': 707,
'connectionReused': False,
'encodedDataLength': 875,
'fromDiskCache': False,
'fromServiceWorker': False,
'headers': {'Cache-Control': 'no-cache',
'Connection': 'Keep-Alive',
'Content-Encoding': 'gzip',
'Content-Type': 'text/html;charset=utf-8',
'Coremonitorno': '0',
'Date': 'Thu, 11 Jan 2018 10:07:03 GMT',
'Server': 'apache',
'Set-Cookie': 'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758; '
'path=/; domain=.baidu.com\n'
'bd_traffictrace=111752_111807\n'
'BDSVRTM=278; path=/\n'
'eqid=deleted; path=/; '
'domain=.baidu.com; '
'expires=Thu, 01 Jan 1970 '
'00:00:00 GMT',
'Strict-Transport-Security': 'max-age=172800',
'Tracecode': '04233185700634468874011118\n'
'04230518060407801866011118',
'Traceid': '151566522306294144108392574322183443433',
'Transfer-Encoding': 'chunked',
'Vary': 'Accept-Encoding'},
'headersText': 'HTTP/1.1 200 OK\r\n'
'Cache-Control: no-cache\r\n'
'Connection: Keep-Alive\r\n'
'Content-Encoding: gzip\r\n'
'Content-Type: '
'text/html;charset=utf-8\r\n'
'Coremonitorno: 0\r\n'
'Date: Thu, 11 Jan 2018 10:07:03 '
'GMT\r\n'
'Server: apache\r\n'
'Set-Cookie: '
'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758; '
'path=/; domain=.baidu.com\r\n'
'Set-Cookie: '
'bd_traffictrace=111752_111807\r\n'
'Set-Cookie: BDSVRTM=278; path=/\r\n'
'Set-Cookie: eqid=deleted; path=/; '
'domain=.baidu.com; expires=Thu, 01 '
'Jan 1970 00:00:00 GMT\r\n'
'Strict-Transport-Security: '
'max-age=172800\r\n'
'Tracecode: '
'04233185700634468874011118\r\n'
'Tracecode: '
'04230518060407801866011118\r\n'
'Traceid: '
'151566522306294144108392574322183443433\r\n'
'Vary: Accept-Encoding\r\n'
'Transfer-Encoding: chunked\r\n'
'\r\n',
'mimeType': 'text/html',
'protocol': 'http/1.1',
'remoteIPAddress': '61.135.169.125',
'remotePort': 443,
'requestHeaders': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, '
'sdch, br',
'Accept-Language': 'zh-CN,zh;q=0.8',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'Cookie': 'BAIDUID=21B2249D2761E4A9CE64A356CFA0A21D:FG=1; '
'bd_traffictrace=111752; '
'BDSVRTM=22; '
'plus_lsv=bddd969db4a4e207; '
'BDORZ=AE84CDB3A529C0F8A2B9DCDD1D18B695; '
'plus_cv=1::m:21732389; '
'Hm_lvt_12423ecbc0e2ca965d84259063d35238=1515664352; '
'Hm_lpvt_12423ecbc0e2ca965d84259063d35238=1515664352; '
'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758',
'Host': 'www.baidu.com',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Linux; '
'Android 4.4.4; '
'SM-A7000 '
'Build/KTU84P) '
'AppleWebKit/537.36 '
'(KHTML, like Gecko) '
'Chrome/58.0.3029.83 '
'Mobile '
'Safari/537.36'},
'securityState': 'secure',
'status': 200,
'statusText': 'OK',
'timing': {'connectEnd': 352.51100000005,
'connectStart': 0,
'dnsEnd': 0,
'dnsStart': 0,
'proxyEnd': -1,
'proxyStart': -1,
'pushEnd': 0,
'pushStart': 0,
'receiveHeadersEnd': 1468.39399999953,
'requestTime': 6117.724403,
'sendEnd': 1062.45399999989,
'sendStart': 1061.98099999983,
'sslEnd': 352.458999999726,
'sslStart': 107.189999999719,
'workerReady': -1,
'workerStart': -1},
'url': 'https://www.baidu.com/'},
'timestamp': 6119.205715,
'type': 'Document'}}
{'method': 'Network.dataReceived',
'params': {'dataLength': 7150,
'encodedDataLength': 0,
'requestId': '9616.88',
'timestamp': 6119.209894}}
{'method': 'Network.dataReceived',
'params': {'dataLength': 32768,
'encodedDataLength': 3221,
'requestId': '9616.88',
'timestamp': 6119.256851}}
{'method': 'Network.loadingFinished',
'params': {'encodedDataLength': 42585,
'requestId': '9616.88',
'timestamp': 6119.430418}}
其中我们所需要的信息都在Network.responseReceived里面，有url、timing、remoteIPAddress、status状态码。其中这个encodedDataLength在response里面仅有875，感觉不太对，这个如何获取请求的大小比较复杂，不同的请求，获取方法不太一样。这个还没研究出如何获取。
如何获取网页使用的内存信息
参考文献：
https://sites.google.com/a/chromium.org/chromedriver/logging/performance-log
https://chromedevtools.github.io/devtools-protocol/
https://stackoverflow.com/questions/27596229/browser-performance-tests-through-selenium/
https://github.com/webdriverio/webdriverio/issues/476
https://stackoverflow.com/questions/27657655/unable-to-get-performance-logs
https://groups.google.com/forum/#!topic/google-chrome-developer-tools/FCCV2J7BaIY
https://chromedevtools.github.io/devtools-protocol/tot/Network/