Grinder 性能测试

Performance testing is important. In this blog post we will explain why, introduce you to a tool for testing performance called Grinder, and take you through five examples which will get you started using Grinder for testing your web application.

The examples in this post is based on workshops held at ROOTS and FreeTest, and the materials – slides, tasks and code – are of course available on GitHub. To get the most out of the examples, you may want to follow along with the materials trying each task on your own, before reading the solutions we present here.

This blog post consists of three parts:

1. A small motivational speech to get you pumped and ready to learn the nuts and bolts of performance testing.
2. An introduction to the tool Grinder, with a small example to show you how it works.
3. And last, the big part: 5 real-world examples that will teach you everything you need to feel comfortable testing your own stuff with Grinder.

Without further ado, let’s get started!

Well, the fact that you are already reading this post is a good sign – you obvoiously have at least a gut feeling that performance testing is important. And we totally agree with you.

There are a few reasons why we think it’s important:

#1: Performance is a feature!
It really is! Your users will love you, you will earn more money, and unicorns will dance in joy if you focus on performance. Don’t belive us? Well, this guy says the same. Don’t you belive him either? He created StackOverflow and stuff, so you should! We won’t repeat all the examples here, but suffice to say: this stuff matters! Performance really is a feature, and a damn important one too!

#2: Bad performance can kill you!
Literary! Or figuratively, whatever word kids use these days. The daily press really loves a good “important site down, users take to the streets”-story. You definitely don’t want your site’s logo on those demonstation signs! And when you then have to go into rush-mode and try to fix things, bad things can (and according to mr. Murphy, often does) happen. That’s when Kenneth36 and all those nasty things happens! So you definitely want to test performance yourself, and don’t leave that task to your users on release-day.

#3: It’s good for your health!
Do you sleep well at night? Or do you constantly worry which server or system is going to go down next? Do you take pride in what you do for a living? Some guy (with lots of gray hairs, that’s how you know he’s a smart one) once uttered this brilliant quote:

Measurement is the first step that leads to control
and eventually to improvement.
– If you can’t measure something, you can’t understand it.
– If you can’t understand it, you can’t control it.
– If you can’t control it, you can’t improve it.

In the end, performance testing really boils down to the basic task of caring about what you do. You unit test your code, sweat over small details in your user interface, and even care about *how* you work (Scrum, Kanban and all that stuff). Why shouldn’t you take the same pride in your product’s ability to actually serve your users, and don’t fall down when everyone and their grandmas come knocking.

Well, we think you should!

Enough about that. Now that you are properly motivated, we’ll put on our “serious-glasses” and start looking at how you can approach this performance test thing!

Okay, you get it already, performance testing is important! But why should you use Grinder in particular? And what is Grinder anyway? Lets have a closer look.

Grinder is a load testing framework written in Java. It is free, open source under a BSD-style licence, and supports large scale testing using distributed load injector machines. The main selling point of Grinder, however, is that it is lightweight and easy to use. With Grinder there are no licences to buy or large environments to set up. You just write a simple test script and fire up the grinder.jar file.

The reason you, as a developer, will prefer Grinder is that you create your tests in code, not some quirky GUI. Although Grinder itself is written in Java, the test scripts are written in Jython or Closure, which means you can utilize the expressiveness of a dynamic or functional language while still tapping into the resources of Java and the JVM. (Of course, should you be so unfortunate not to be a developer, then Grinder might not be the tool for you. But you still need to understand why your developers want and should use Grinder!)

So, lets have a closer look on the nuts and bolts of Grinder.

Grinder 101

The Grinder framework is comprised of three types of processes (or programs):

1. Worker processes: Interprets test scripts and performs the tests using a number of parallel worker threads.
2. Agent processes: Long running processes that start and stop worker processes as required.
3. The Console: Coordinates agent processes, and collates and displays statistics.

In this tutorial we’ll keep things simple, and focus on the worker and agent precesses. We will start an agent process manually, providing it with a test script and configuration, leaving the console and distributed testing out for now.

The test script is simply a Python (or Clojure) source file. Grinder expects to find a class called TestRunner which should implement two methods: __init__ and __call__. The init-method is called by Grinder initially to set up the test, and the call-method is then called repeatedly for each iteration of the testing.

A Simple Example

Consider the code below. This is the Grinder-equivalent of a “Hello World” program. It defines a test script which will print the thread number of the worker thread and the string “hello world” each time it is triggered.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

from net.grinder.script.Grinder import grinder from net.grinder.script import Test   # defining a simple "hello world" function, in order to have something to test def hello_world(): thread = grinder.getThreadNumber() print '> worker thread %d: hello world!' % thread   class TestRunner:   def __init__(self): # creating a Grinder test object test = Test(1, "saying hello") # creating a proxy, by wrapping the hello world function with our Test-object self.wrapped_hello = test.wrap(hello_world)   def __call__(self): # calling "hello world" through the wrapped function self.wrapped_hello()

The important part to note in the above code is the Test object. This object represents a basic operation of testing with Grinder. We wrap our hello_world function with the test object, which tells Grinder to start timing and recording it. Notice also that we are calling Grinder’s wrapped version, not the function itself.

Next we need a test configuration.

1 2 3 4 5 6 7 8

grinder.script = hello.py   grinder.processes = 1 grinder.threads = 4 grinder.runs = 5   grinder.useConsole = false grinder.logDirectory = log

Here we first tells Grinder which test script to run. Then we specify that we only want one worker process started, and that the worker process should initiate four threads, each making five runs, i.e. calls to our __call__ method. The final two lines tell Grinder not to expect us to use the console (thus avoiding some output warnings) and where to put the log files describing the test results.

Now we are ready to run the test. Start Grinder using the following command:

java -cp lib/grinder.jar net.grinder.Grinder hello.properties

Or, more conveniently by using the startAgent script:

./startAgent.sh hello.properties

This should result in the printing of “hello world” a number of times and produce some files in the log directory. The data_xyz.log file contains a detailed summary of the test events, while the out_xyz.log file provides a summary of the results.

         Tests    Errors   Mean Test    Test Time    TPS
                           Time (ms)    Standard
                                        Deviation
                                        (ms)
 
Test 1   20       0        0.90         3.05         625.00    "saying hello"
 
Totals   20       0        0.90         3.05         625.00

Let’s get started learning Grinder by looking at five real-world examples (or, at least as real-world as simple explanatory examples can be). To lean as much as possible, you should really try to do the examples yourself as you move along. Or, at least you should check out the example code and try to run the finished solutions.

So lets check out the example code before we move on:

Bootstrapping the Framework

TL;DR: run this command

git clone git://github.com/kvalle/grinder-workshop.git; cd grinder-workshop; ./startAgent.sh example/scenario.properties

A bit more detailed:

What you need to do is:

• Make sure you have git installed
• Make sure you have Java installed
• Check out the repository containing the examples from the workshop:

git clone  git://github.com/kvalle/grinder-workshop.git

When you are finished with these simple steps, you can check that everything works by running the sample test (which we described in the last section):

cd grinder-workshop
./startAgent.sh example/scenario.properties

When you run this example, Grinder will output some information. First there will be some information about what happens during the start-up of Grinder, and then some lines of ‘hello world’ while running the test script. This should look something like the following:

> worker thread 2: hello world!
> worker thread 0: hello world!
> worker thread 1: hello world!
> worker thread 3: hello world!
...

When the test has finished running, you can also check that everything is okay by inspecting the results that have been stored in the newly created directory grinder-workshop/log. There should be two files with names like out_xyz-0.log and data_xyz-0.log where xyz is the name of your computer. The out-file contains a summary of the test results, and the data file contains all the details in a comma-separated format. If everything ran smoothly there should not be any files with names like error_xyz-0.log!

If you for some weird reason can’t install git it is also possible to download the code as zip file.

Now that you have all of the example code, let’s start with the examples!

In the first example, we will start easy by writing a test which makes a HTTP GET request for a single URL, and measures the response time.

First, we need some setup. Like in the “hello world” example, we start with a simple configuration file in 1.properties.

1 2 3 4 5 6 7 8

grinder.script = scripts/task1.py   grinder.processes = 1 grinder.threads = 1 grinder.runs = 10   grinder.useConsole = false grinder.logDirectory = log

There’s not much new here. Again, the first property informs Grinder which test script to run. The second group of properties specifies how much load the test will utilize. The defaults for all three properties are 1. Our setup will run the test 10 times sequentially in a single thread. The last properties are identical to the previous example.

Next we’ll need to write the test script we just referenced. Our scripts/task1.py should look something like this:

1 2 3 4 5 6 7 8 9 10 11 12

from net.grinder.script.Grinder import grinder from net.grinder.script import Test from net.grinder.plugin.http import HTTPRequest   class TestRunner:   def __init__(self): test = Test(1, "GETing some webpage") self.request = test.wrap(HTTPRequest())   def __call__(self): self.request.GET("http://foobar.example.com/page42")

In the init-method we start by crating a Test object and then use its wrap method to create a proxy object wrapping an instance of HTTPRequest. This proxy is then stored as an instance variable self.request, ready for use in the test runs.

The reason we need to use the Test to create a proxy is in order for Grinder to be able to track what is happening. Every method invoked on the proxy is recorded and timed by Grinder, before passing the call on to the wrapped object or function.

In this example, the call-method simply invoks the GET method on the proxy, which is then passed through to the HTTPRequest instance performing the actual request.

If you are following along with the workshop materials, you can now run the test by using the startAgent script.

./startAgent.sh tasks/1.properties

(Or, if you checked out the code but didn’t bother to do the tasks, substitute solutions for tasks and do the same.)

This should generate a few files in the log directory specified in the test configuration, where the results of the test are recorded.

Testing the response time of a single URL isn’t very useful, so the next step is naturally to time the requests of a bunch of different URLs. We also don’t want to hard code the URLs into the test script, but list them in a separate file.

Lets say we have a file with URLs that look something like this:

http://example.com/
http://example.com/foo.php
http://example.com/bar.php
http://example.com/baz.php
http://example.com/unreliable-link.php
http://example.com/bad-link.php
http://example.com/404.php

In this example we’ll read this file, create a test for each URL, and “GET” it each time the script runs.

The configuration in 2.properties is very similar to the previous example.

1 2 3 4 5 6 7

grinder.script = scripts/task2.py   grinder.runs = 10 grinder.useConsole = false grinder.logDirectory = log   task2.urls = solutions/scripts/urls.txt

We reference, of course, a new script file and also add one additional parameter. The new parameter task2.urls is a custom parameter which holds the path to our text file with the URLs, since this is something we really don’t want to hard code into the script.

Next we implement the script as follows.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

from net.grinder.script.Grinder import grinder from net.grinder.script import Test from net.grinder.plugin.http import HTTPRequest   url_file_path = grinder.getProperties().getProperty('task2.urls')   class TestRunner:   def __init__(self): url_file = open(url_file_path) self.tests = [] for num, url in enumerate(url_file): url = url.strip() test = Test(num, url) request = test.wrap(HTTPRequest()) self.tests.append((request, url)) url_file.close()   def __call__(self): for request, url in self.tests: request.GET(url)

Notice first how we extract custom property value as the file path in the start of the script. The rest of the script is familiar, but with some additional logic in the __init__ and __call__ methods. The code should hopefully be pretty self explanatory, but lets walk through it.

The first thing we do in the __init__ method is to open the file with the URLs. We then iterate through the file, using enumerate to get the line numbers. For each line we extract the URL, create a Test object, wrap a HTTPRequest object using the test object, and append the wrapped request together with the URL to our list of tests.

In __call__ we simply walk through the list of tests, making a GET request for the URL using the the corresponding wrapped HTTPRequest.

The important thing to notice in this example is that we don’t simply GET all the URLs using a single HTTPRequest. We want each URL to be timed separately, and must therefore create a grinder-proxy for each one.

Now we are ready to try it out.

Run:

./startAgent.sh tasks/2.properties

Which should give you something like the following included at the end of the log file:

             Tests        Errors       Mean Test    Test Time    TPS          Mean         Response     Response     Mean time to Mean time to Mean time to 
                                       Time (ms)    Standard                  response     bytes per    errors       resolve host establish    first byte   
                                                    Deviation                 length       second                                 connection                
                                                    (ms)                                                                                                    
 
Test 0       10           0            453.70       105.15       1.09         483.00       528.16       0            15.20        51.40        451.00        "http://example.com/"
Test 1       10           0            59.80        19.55        1.09         163.00       178.24       0            15.20        51.40        58.90         "http://example.com/foo.php"
Test 2       10           0            131.20       21.48        1.09         18433.00     20156.37     0            15.20        51.40        66.80         "http://example.com/bar.php"
Test 3       10           0            59.10        16.08        1.09         616.00       673.59       0            15.20        51.40        57.90         "http://example.com/baz.php"
Test 4       10           0            57.60        25.24        1.09         71.00        77.64        4            15.20        51.40        56.60         "http://example.com/unreliable-link.php"
Test 5       10           0            75.00        26.10        1.09         0.00         0.00         0            15.20        53.70        73.30         "http://example.com/bad-link.php"
Test 6       10           0            75.00        69.34        1.09         169.00       184.80       10           15.20        53.70        74.10         "http://example.com/404.php"
 
Totals       70           0            130.20       143.58       1.09         2847.86      3114.11      14           15.20        52.06        119.80

If you looked carefully at the results from the previous example, you might have discovered a problem with the test scripts we have written so far. The errors column reports that everything went OK although, as we can see, there where several errors in the response errors column!

In this example, we’ll look at how to inspect the HTTP responses, and tell Grinder to fail a test if we find anything fishy.

Again we start with the test configuration:

1 2 3 4 5 6 7

grinder.script = scripts/task3.py   grinder.runs = 10 grinder.useConsole = false grinder.logDirectory = log   task3.urls = solutions/scripts/urls.txt

Not much is changed here, so lets move on to the test script.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

from net.grinder.script.Grinder import grinder from net.grinder.script import Test from net.grinder.plugin.http import HTTPRequest   url_file_path = grinder.getProperties().getProperty('task3.urls')   class TestRunner:   def __init__(self): url_file = open(url_file_path) self.tests = [] for num, url in enumerate(url_file): url = url.strip() test = Test(num, url) request = test.wrap(HTTPRequest()) self.tests.append((request, url)) url_file.close() grinder.statistics.setDelayReports(True)   def __call__(self): for request, url in self.tests: response = request.GET(url) if not self.is_valid(response): self.fail() grinder.statistics.report()   def fail(self): grinder.statistics.getForLastTest().setSuccess(False)   def is_valid(self, response): if len(response.getData()) < 10: return False if response.getStatusCode() != 200: return False if "epic fail" in response.getText(): return False else: return True

There are a few differences here, comprared to the last example. Most obviously there are two new methods, but we also do some new things in the old ones.

First off, we need to tell Grinder not to automatically assume everything is okay as long as the code executes and the GET method returns. We do this with the call to grinder.statistics.setDelayReports(True) at the end of the __init__ method.

Next, the __call__ method has been extended to capture the HTTP response object. We validate the response using our new is_valid method, and tell Grinder to fail the test if validation fails. Lastly, grinder.statistics.report() makes Grinder record the result of the test and move on.

The new is_valid method is where most of the magic happens. Here we can look at any aspect of the response we might like, and simply return False if we are not satisfied.

(It would of course also be possible to make different validation checks for different URLs. We won’t go into how to do that here, but you might have a look at this example to see how it could be done.)

By running the script we get these new results:

             Tests        Errors       Mean Test    Test Time    TPS          Mean         Response     Response     Mean time to Mean time to Mean time to 
                                       Time (ms)    Standard                  response     bytes per    errors       resolve host establish    first byte   
                                                    Deviation                 length       second                                 connection                
                                                    (ms)                                                                                                    
 
Test 0       10           0            434.00       63.22        1.14         483.00       551.24       0            13.50        43.40        430.90        "http://example.com/"
Test 1       10           0            48.00        5.59         1.14         163.00       186.03       0            13.50        43.40        46.80         "http://example.com/foo.php"
Test 2       10           0            148.60       89.52        1.14         18433.00     21037.43     0            13.50        43.40        80.60         "http://example.com/bar.php"
Test 3       10           0            63.50        23.10        1.14         616.00       703.04       0            13.50        43.40        62.30         "http://example.com/baz.php"
Test 4       6            4            39.33        7.99         0.68         103.00       70.53        0            22.50        50.50        38.67         "http://example.com/unreliable-link.php"
Test 5       0            10           �            0.00         0.00         �            0.00         0            �            �            �             "http://example.com/bad-link.php"
Test 6       0            10           �            0.00         0.00         �            0.00         0            �            �            �             "http://example.com/404.php"
 
Totals       46           24           156.02       160.39       0.75         4294.96      3221.18      0            14.67        44.33        139.96

And indeed, the bad responses actually show up under errors this time.

Up until now, we have tested some pretty static pages. Now, it’s time to do some more fancy parsing.

We’ll be testing against an API which returns JSON-formatted data. This JSON-object will contain links to more stuff you can test against. These links will theoretically change for each request, which means that we’ll have to parse the JSON to fetch the links – we can’t hard-code all the links in the script beforehand.

When we do a manual call against the webpage: http://grinder.espenhh.com/json.php, we get some nicely formatted JSON back:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

{ "fetched":"19.04.2012", "tweets":[ { "user":"Espenhh", "tweet":"Omg, this grinder workshop is amazing", "profile_image":"http://grinder.espenhh.com/pics/1337.jpg" }, { "user":"kvalle", "tweet":"Have you seen my new ROFL-copter? It`s kinda awesome!", "profile_image":"http://grinder.espenhh.com/pics/rofl.gif" }, { "user":"Hurtigruta", "tweet":"I`m on a boat! Yeah!", "profile_image":"http://grinder.espenhh.com/pics/123.jpg" } ] }

The property file is basically the same now as in the last three examples, the only change is that it points to the correct script.

The script is as usual where all the magic happens:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

from net.grinder.script.Grinder import grinder from net.grinder.script import Test from net.grinder.plugin.http import HTTPRequest from org.json import *   class TestRunner:   def __init__(self): test1 = Test(1, "GET some JSON") self.request1 = test1.wrap(HTTPRequest())   test2 = Test(2, "GET profilepicture") self.request2 = test2.wrap(HTTPRequest())   def __call__(self): # Fetches the initial JSON response = self.request1.GET("http://grinder.espenhh.com/json.php") print "JSON: " + response.getText()   # Parses the JSON (Using a Java library). Then prints the field "fetched" jsonObject = JSONObject(response.text) fetched = jsonObject.getString("fetched") print "FETCHED: " + fetched   # Gets the single tweets, and loops through them tweetsJsonObject = jsonObject.getJSONArray("tweets") for i in range(0,tweetsJsonObject.length()): singleTweet = tweetsJsonObject.getJSONObject(i)   # Print out a single tweet userName = singleTweet.getString("user") tweet = singleTweet.getString("tweet") print "TWEET: " + userName + ": " + tweet   # Fetch the URL to the profile picture, and GET it. profilePictureUri = singleTweet.getString("profile_image") print "GET against profile picture uri: " + profilePictureUri self.request2.GET(profilePictureUri)

First, note that we now import the package org.json, which is a normal Java library. Nothing fancy is going on here, we are just taking advantage of the fact that you are freely able to call Java code from your Grinder scripts. We could just as easily have used a Python module to do the parsing, but this way you’ll get some experience working across the language barrier.

Knowing that, the first part of the script is easy to understand. We just fetch the content of the previously mentioned link, and stores it in a variable. Then we start using the Java-library. jsonObject = JSONObject(response.text) parses the raw JSON from the response into a simple object we can use. We then demonstrate a few methods: jsonObject.getString("fetched") and jsonObject.getJSONArray("tweets"). It’s really simple to navigate the JSON-structure doing this.

To get down to action and actually perform some more HTTP requests, we loop throuh the array of tweets, and get the links of each tweet-authors profile picture. We then do a new GET-request to get this photo. Please note that we here uses the same Test-object for all these requests. You might wonder why we don’t use a new Test-object for each and every request, because the links are different, and they therefore are different objects. That’s something you just have to decide for yourself, but in our opinion, each Test-object should really be about a concept, and not about a single link. Each tweet-profile-photo is different, but they are all tweet-profile-photos, therefore they share the same Test-object. But, you’ll need to decide what exactly you want to test on a case-to-case-basis.

Running this script, we get the following output:

                 Tests        Errors       Mean Test    Test Time    TPS          Mean         Response     Response     Mean time to Mean time to Mean time to 
                                           Time (ms)    Standard                  response     bytes per    errors       resolve host establish    first byte   
                                                        Deviation                 length       second                                 connection                
                                                        (ms)                                                                                                    
 
    Test 1       1            0            500.00       0.00         1.31         413.00       541.28       0            381.00       421.00       489.00        "GET some JSON"
    Test 2       3            0            82.00        37.21        3.93         3806.33      14965.92     0            381.00       421.00       69.67         "GET profilepicture"
 
    Totals       4            0            186.50       183.85       2.62         2958.00      7753.60      0            381.00       421.00       174.50

We here see that we are doing one GET-request to get the initial JSON, and then the loop performs three GET-requests to get the profile-pictures. By coding your tests this way (and having a good and RESTful API to test against, which contains links), you’ll be able to write small and simple tests that test large amounts of functionality with simple code. They’ll also be quite robust and won’t break if you change your backend in small ways.

Sometimes, you don’t want to write all your tests by hand, you just want to simulate a user clicking through some pages in a browser. Grinder has support for this; by using the Grinder TCPProxy you can record a web-browsing-session and replay it using Grinder afterwards. This technique will also generate a script which you can later modify (this is something you almost certainly would want to do!).

Here, we’ll let you do the hard lifting yourself. Using the scripts provided in the example Git project, do the following:

1. Start the proxy server by running the script ./startProxy.sh. This will start a simple console that lets you input comments, and stop the proxy cleanly.
2. Configure your browser to send traffic through the proxy. This most likely means localhost, port 8001 – the output from the first step will tell you if this is correct.
3. Browse to a simple web page (we recommend starting with http://grinder.espenhh.com/simple/ ). If you browse to a complex page, the generated script will be crazy long!
4. After the page has loaded in the browser, click “stop” in the simple console window.
5. Inspect the generated script: it’s located at proxy/proxygeneratedscript.py.
6. Try running the script: ./startAgent.sh proxy/proxygeneratedscript.properties.
7. Check the log, try modifying the script, experiment. You can start by removing all the sleep statements in the script. Then try it on a more complicated page.

As you’ll see, the scripts generated using this method will be ugly as hell, and overly complex. Therefore, we really don’t recommend using the Grinder TCPProxy to generate scripts which you plan to be using and maintaining in the future. But for one-time scripts, it can be quite handy.

That’s it, really! We hope you have learned something, and feel inspired to start testing your applications.

The gist of it all is that we belive performance testing is important. We find Grinder to be one of the best tools for developers, mainly because it is lightweight and easy to use. All you need to get started is a couple of small files and the grinder.jar. This makes it easy to do some testing now and again, without making performance testing into a big fuzz. A little bit of testing regularly while a application is under development beats big bang testing after the fact any day.

One thing we did not focus much on in this blogpost, but that should be noted nonetheless, is the fact that Grinder, and Jython, runs on Java. This means that you don’t nessicarily need to wrap HTTPRequests, you could wrap calls to your application directly!

After taking you though these five gradually more complex examples, we have hopefully given you some insight into how Grinder works and what it can be used for. Now you should be ready to write some scripts of your own.

Good luck, and have fun!