locust压测工具：http测试过程与定时停止
locust压测环境描述
本文环境python3.5.2
locust版本0.9.0
locust示例的执行过程
上文大概描述了locust的启动了流程，本文主要是接着上文继续分析，示例代码中的http的测试的执行过程，是如何去访问远端的http接口的流程。接着就分析如何通过传入的运行时间参数来停止locust的运行。
http测试用例的执行
示例代码中访问url的代码如下：
@task(2)
def index(self):
self.client.get("/")
@task(1)
def profile(self):
self.client.get("/profile")
其中的client就是继承自HttpLocust初始化过程中创建的，由于示例代码里面的WebsiteUser继承自HttpLocust，在初始化的过程的时候；
class HttpLocust(Locust):
"""
Represents an HTTP "user" which is to be hatched and attack the system that is to be load tested.
The behaviour of this user is defined by the task_set attribute, which should point to a
:py:class:`TaskSet ` class.
This class creates a *client* attribute on instantiation which is an HTTP client with support
for keeping a user session between requests.
"""
client = None
"""
Instance of HttpSession that is created upon instantiation of Locust.
The client support cookies, and therefore keeps the session between HTTP requests.
"""
def __init__(self):
super(HttpLocust, self).__init__()
if self.host is None:
raise LocustError("You must specify the base host. Either in the host attribute in the Locust class, or on the command line using the --host option.")
self.client = HttpSession(base_url=self.host) # 初始化一个client实例
此时的client就是一个HttpSession的实例，分析该类：
class HttpSession(requests.Session): # 继承自requests.Session
"""
Class for performing web requests and holding (session-) cookies between requests (in order
to be able to log in and out of websites). Each request is logged so that locust can display
statistics.
This is a slightly extended version of `python-request `_'s
:py:class:`requests.Session` class and mostly this class works exactly the same. However
the methods for making requests (get, post, delete, put, head, options, patch, request)
can now take a *url* argument that's only the path part of the URL, in which case the host
part of the URL will be prepended with the HttpSession.base_url which is normally inherited
from a Locust class' host property.
Each of the methods for making requests also takes two additional optional arguments which
are Locust specific and doesn't exist in python-requests. These are:
:param name: (optional) An argument that can be specified to use as label in Locust's statistics instead of the URL path.
This can be used to group different URL's that are requested into a single entry in Locust's statistics.
:param catch_response: (optional) Boolean argument that, if set, can be used to make a request return a context manager
to work as argument to a with statement. This will allow the request to be marked as a fail based on the content of the
response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a request
and then mark it as successful even if the response code was not (i.e 500 or 404).
"""
def __init__(self, base_url, *args, **kwargs):
super(HttpSession, self).__init__(*args, **kwargs) # 调用父类的初始化方法
self.base_url = base_url # 设置Host
# Check for basic authentication
parsed_url = urlparse(self.base_url) # 解析url
if parsed_url.username and parsed_url.password: # 检查是否配置了用户名与密码
netloc = parsed_url.hostname
if parsed_url.port:
netloc += ":%d" % parsed_url.port
# remove username and password from the base_url
self.base_url = urlunparse((parsed_url.scheme, netloc, parsed_url.path, parsed_url.params, parsed_url.query, parsed_url.fragment))
# configure requests to use basic auth
self.auth = HTTPBasicAuth(parsed_url.username, parsed_url.password)
def _build_url(self, path):
""" prepend url with hostname unless it's already an absolute URL """
if absolute_http_url_regexp.match(path): # 检查是否是绝对路径
return path # 如果是绝对路径则直接返回
else:
return "%s%s" % (self.base_url, path) # 返回完整的url
def request(self, method, url, name=None, catch_response=False, **kwargs):
"""
Constructs and sends a :py:class:`requests.Request`.
Returns :py:class:`requests.Response` object.
:param method: method for the new :class:`Request` object.
:param url: URL for the new :class:`Request` object.
:param name: (optional) An argument that can be specified to use as label in Locust's statistics instead of the URL path.
This can be used to group different URL's that are requested into a single entry in Locust's statistics.
:param catch_response: (optional) Boolean argument that, if set, can be used to make a request return a context manager
to work as argument to a with statement. This will allow the request to be marked as a fail based on the content of the
response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a request
and then mark it as successful even if the response code was not (i.e 500 or 404).
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
:param files: (optional) Dictionary of ``'filename': file-like-objects`` for multipart encoding upload.
:param auth: (optional) Auth tuple or callable to enable Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) How long in seconds to wait for the server to send data before giving up, as a float,
or a (`connect timeout, read timeout `_) tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Set to True by default.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param stream: (optional) whether to immediately download the response content. Defaults to ``False``.
:param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
"""
# prepend url with hostname unless it's already an absolute URL
url = self._build_url(url) # 获取访问路径
# store meta data that is used when reporting the request to locust's statistics
request_meta = {} # 请求头部信息
# set up pre_request hook for attaching meta data to the request object
request_meta["method"] = method # 请求的方法
request_meta["start_time"] = time.time() # 请求开始的时间
response = self._send_request_safe_mode(method, url, **kwargs) # 访问请求并获取返回值
# record the consumed time
request_meta["response_time"] = (time.time() - request_meta["start_time"]) * 1000 # 记录响应处理的时间
request_meta["name"] = name or (response.history and response.history[0] or response).request.path_url
# get the length of the content, but if the argument stream is set to True, we take
# the size from the content-length header, in order to not trigger fetching of the body
if kwargs.get("stream", False):
request_meta["content_size"] = int(response.headers.get("content-length") or 0) # 获取返回的文件长度
else:
request_meta["content_size"] = len(response.content or b"") # 否则获取响应的整体大小
if catch_response: # 如果要包含请求信息
response.locust_request_meta = request_meta
return ResponseContextManager(response) # 用ResponseContextManager包裹response
else:
try:
response.raise_for_status() # 检查返回状态
except RequestException as e:
events.request_failure.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
exception=e,
) # 如果失败则通知所有的失败请求事件执行
else:
events.request_success.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
response_length=request_meta["content_size"],
) # 如果成功则通知所有的成功事件执行
return response
def _send_request_safe_mode(self, method, url, **kwargs):
"""
Send an HTTP request, and catch any exception that might occur due to connection problems.
Safe mode has been removed from requests 1.x.
"""
try:
return requests.Session.request(self, method, url, **kwargs) # 调用requests的Session去请求接口
except (MissingSchema, InvalidSchema, InvalidURL):
raise
except RequestException as e:
r = LocustResponse()
r.error = e
r.status_code = 0 # with this status_code, content returns None
r.request = Request(method, url).prepare()
return r
由该代码可知，处理的请求都是通过requests库中的Session来进行请求的，示例代码中的client都是通过requests的代码进行请求，并且还可以使用session来保持会话，从而使接口请求的时候能够带上权限检查等额外信息。
locust定时退出
由于可以在启动locust可以指定执行的时间，可以到时间退出，我们分析一下该退出函数的执行，
def spawn_run_time_limit_greenlet():
logger.info("Run time limit set to %s seconds" % options.run_time)
def timelimit_stop():
logger.info("Time limit reached. Stopping Locust.")
runners.locust_runner.quit()
gevent.spawn_later(options.run_time, timelimit_stop)
等到了run_time之后，就会执行timelimit_stop函数，而该函数就是调用了实例化的locust类实例的quit方法；
def stop(self):
# if we are currently hatching locusts we need to kill the hatching greenlet first
if self.hatching_greenlet and not self.hatching_greenlet.ready(): # 检查是否有hatch_greenlet并且没有准备好 就杀掉该协程
self.hatching_greenlet.kill(block=True)
self.locusts.kill(block=True) # 杀死所有的协程组
self.state = STATE_STOPPED # 更改状态为停止态
events.locust_stop_hatching.fire() # 通知所有locust_stop_hatching的函数执行
def quit(self):
self.stop() # 停止执行所有的greenlet
self.greenlet.kill(block=True) # 主greenlet杀死
此时就是停止所有的协程执行，终止该测试用例的执行。其中events.locust_stop_hatching使用了典型的观察者设计模式：
locust_stop_hatching = EventHook()
class EventHook(object):
"""
Simple event class used to provide hooks for different types of events in Locust.
Here's how to use the EventHook class::
my_event = EventHook()
def on_my_event(a, b, **kw):
print "Event was fired with arguments: %s, %s" % (a, b)
my_event += on_my_event
my_event.fire(a="foo", b="bar")
If reverse is True, then the handlers will run in the reverse order
that they were inserted
"""
def __init__(self):
self._handlers = [] # 所有待处理的handlers
def __iadd__(self, handler):
self._handlers.append(handler) # 添加到处理的Handler列表中
return self
def __isub__(self, handler):
self._handlers.remove(handler) # 移除handler
return self
def fire(self, reverse=False, **kwargs):
if reverse:
self._handlers.reverse() # 是否排序
for handler in self._handlers: # 依次遍历handler并执行
handler(**kwargs)
总结
本文主要是继续分析了locust启动之后，http的请求的处理与定时退出的功能，其中http的请求都是基于requests.Session来实现的，定时退出的功能主要还是依赖于gevent中的Group来控制所有已经运行的协程，通过停止所有运行的协程来达到关闭停止运行的目的，其中还有些许细节并没有详细说明，大家有兴趣可自行查阅相关源码。鉴于本人才疏学浅，如有疏漏请批评指正