比较Jmeter、Grinder和JAVA多线程本身压力测试所带来的性能开销

1. 测试环境

jmeter版本 ：jmeter 2.4

grinder的版本 ： Grinder 3

JAVA的版本：JDK 1.6

2. 测试代码

Jmeter测试代码

public class Sampler {
    public void test() {
        return;
    }
}

public class JmeterTest extends AbstractJavaSamplerClient {
    Sampler sampler;

    @Override
    public SampleResult runTest(JavaSamplerContext context) {

        SampleResult results = new SampleResult();

        results.sampleStart();

        sampler.test();

        results.sampleEnd();    

        results.setSuccessful(true);

        return results;

    }

    @Override
    public void setupTest(JavaSamplerContext arg0) {
        sampler = new Sampler();
    }
}

grinder测试代码

public class Sampler {

        public void test() {
                return;
        }

}

# Test.py
#
# A minimal script that tests The Grinder logging facility.
#
# This script shows the recommended style for scripts, with a
# TestRunner class. The script is executed just once by each worker
# process and defines the TestRunner class. The Grinder creates an
# instance of TestRunner for each worker thread, and repeatedly calls
# the instance for each run of that thread.

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from sampler import Sampler
test = Test(1, "Sample")
class TestRunner:

    # This method is called for every run.
    def __call__(self):
        mySampler = test.wrap(Sampler())
        mySampler.test()

Java本身多线程

public static void test(int numOfThreads, int times) throws InterruptedException, ExecutionException {
		ExecutorService executor = Executors.newFixedThreadPool(numOfThreads);
		final Sampler sampler = new Sampler();
		List<Future<Long>> results = new ArrayList<Future<Long>>(); 
		for (int i = 0; i < times; i++) {
			results.add(executor.submit(new Callable<Long>() {
				@Override
				public Long call() throws Exception {
						long begin = System.currentTimeMillis();
						sampler.test();
						long end = System.currentTimeMillis();
					return end - begin;
				}
			}));
		}
		executor.shutdown();
		while(!executor.awaitTermination(1, TimeUnit.SECONDS));
		
		long sum = 0;
		for (Future<Long> result : results) {
			sum += result.get();
		}
		
		
		System.out.println("---------------------------------");
		System.out.println("number of threads :" + numOfThreads + " times:" + times);
		System.out.println("running time: " + sum + "ms");
		System.out.println("TPS: " + (double)(100000 * 1000) / (double)(sum));
		System.out.println();
	}

3. 测试结果

10个线程 100000次运行

--	TPS
Jmeter	50426.10
Grinder	290275.76
Java threads	2.5E7

20个线程 100000次运行

--	TPS
Jmeter	49215.02
Grinder	225402.91
Java threads	2.5E7

50个线程 100000次运行

--	TPS
Jmeter	29312.61
Grinder	212242.13
Java threads	2.5E7

100个线程 100000次运行

--	TPS
Jmeter	29031.03
Grinder	245507.22
Java threads	2.5E7

200个线程 10000次运行(这里减少了一个0)

--	TPS
Jmeter	28039.87
Grinder	232801.77
Java threads	2.5E7

300个线程 10000次运行

--	TPS
Jmeter	27208.16
Grinder	236537.10
Java threads	1818181.81

1000个线程 10000次运行

--	TPS
Jmeter	27208.16
Grinder	236537.10
Java threads	2.5E7

4. 结论

1. 可以看出Jmeter的本身性能开销是很大的，只适合一般应用的性能测试
2. Grinder在测试的时候发现上下文切换比较严重，而可能是因为内部机制导致的开销较大的，当然如果测试memcache肯定是不适合的，但一般的应用测试基本上没有问题
3. JAVA多线程本身并发框架性能开销也是有的，但是比较低，适合要求较高的性能测试，如对redis和memcache构建的应用进行压测