测试分为好多种类型

测试用例的组织
单元测试
集成测试
端到端测试
消费者测试
契约测试

这是著名的测试金字塔, 也叫测试冰淇淋, 意谓我们应该从下到上地组织编写我们的测试, 大量地单元测试覆盖80%的代码行数, 有了一个坚实的基础, 我们再添加组织测试, 集成测试, API 测试, 端到端和用户界面测试, 越往上维护成本越高, 测试代码越脆弱, 越有可能由于环境问题导致并非代码错误引起的失败

测试用例的组织

首先要胸有成竹, 哪些测试应该做, 在[微服务实战测试之理论篇] 中已提过一些指导原则, 我们要根据这些原则, 结合要测试的特性, 把所有有可能出错的地方覆盖到, 防患于未然.

借用 Cucumber 中的定义的 Gherkin 语法, 一个特性 Feature 有若干个场景 Scenario
而每个场景都必须有独立的意义, 并且不依赖任何其他场景而独立运行.

以表格的形式组织测试用例是比较常见的做法

特性Feature	场景Scenario	给定Given	事件 When	结果 Then
作为系统用户, 我想在任务即将截止设置三次提醒来通知我, 从而我能即时采取措施而不至于超时	今天周一, 我要在周日前交稿	截止日期前一天要有邮件提醒	周六到了	收到提醒邮件

也可以用 wiki 或其他文件格式来存储用例, 推荐用格式化, 易解析的格式化文本文件, 比如 json.

结构层次为 1) Test Suite -- 2) Test Case -- 3) Test Steps(Given, When, Then)

例如:

{
  "testsuites": [
    {
      "name": "login_module_test",
      "testcases": [
        {
          "name": "login_by_phone_step1",
          "feature": "login",
          "scenario": "login by mobile phone",
          "given": "input mobile phone number",
          "when": "submit",
          "then": "send a sms for authenticate code"
        },
        {
          "name": "login_by_phone_step2",
          "feature": "login",
          "scenario": "login by mobile phone",
          "given": "input mobile phone number and authenticate code",
          "when": "submit",
          "then": "redirect the user's home paeg"
        },
        {
          "name": "login_by_error_password",
          "feature": "login",
          "scenario": "login by username and password",
          "given": "input username, password, and captcha",
          "when": "submit",
          "then": "dispatch login failure message: you inputed improper username or password"
        }
      ]
    }
  ]
}

也可以自己写一个注解来自己生成测试用例, 我们在文末给出一个例子

单元测试

在微服务实战测试之理论篇中我们提到测试的分类和测试金字塔, 单元测试是基石, 做好单元是首要的测试工作, 以我熟悉的几种语言为例

测试的写法就四步 SEVT (TVes 许多电视倒过来)

准备 setup
执行 exercise
验证 verify
清理 teardown

简单测试可以忽略1) 和 4) 步

Java 单元测试

哪些库我们可以用呢

如果你使用 spring-boot-starter-test ‘Starter’ (test scope), 你会发现它所提供的下列库:

JUnit — The de-facto standard for unit testing Java applications.
Spring Test & Spring Boot Test — Utilities and integration test support for Spring Boot applications.
AssertJ — A fluent assertion library.
Hamcrest — A library of matcher objects (also known as constraints or predicates).
Mockito — A Java mocking framework.
JSONassert — An assertion library for JSON.
JsonPath — XPath for JSON.

单元测试框架的鼻祖是 junit, 为什么不用 junit 呢? TestNG 有什么独到之处可以后来居上呢? 原因就在于 testng 更为强大的功能, 如 @Test 注解, 可以指定 testcase 的依赖关系, 调用次数, 调用顺序, 超时时间, 并发线程数以及期望的异常, 考虑得非常周到.

当然, 这只是个人喜好, Junit 新版本也多了很多改进.

举个例子, Fibonacci 数列大家很熟悉, 用 Java8 的 stream, lambda 的新写法比老的写法酷很多, 代码行数少了许多.

老写法

public List fibonacci1(int size) {
        List list = new ArrayList<>(size);
        int n0 = 1, n1 = 1, n2;
        list.add(n0);
        list.add(n1);

        for(int i=0;i < size - 2; i++) {
            n2 = n1 + n0;
            n0 = n1;
            n1 = n2;
            list.add(n2);
        }
        return list;
    }

新写法

public List fibonacci2(int size) {
        return Stream.iterate(new int[]{1, 1}, x -> new int[]{x[1], x[0] + x[1]})
                .limit(size).map(x -> x[0])
                .collect(Collectors.toList());
    }

然而性能如何呢? 写个单元测试吧

package com.github.walterfan.example.java8;

import com.google.common.base.Stopwatch;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import java.util.stream.Stream;

/**
 * Created by walter on 24/03/2017.
 * @see http://testng.org/doc/documentation-main.html
 */
public class LambdaPerfTest {
    private static final Logger logger = LoggerFactory.getLogger(LambdaPerfTest.class);

    private Map oldFibonacciResults;
    private Map newFibonacciResults;


    @BeforeClass
    public void init() {
        oldFibonacciResults = new ConcurrentSkipListMap<>();
        newFibonacciResults = new ConcurrentSkipListMap<>();
    }

    @AfterClass
    public void summarize() {
        int rounds = oldFibonacciResults.size();

        System.out.println("--- old vs. new ---");
        oldFibonacciResults.forEach((key, value) -> {
            System.out.println(key + ": " + value + " vs. " + newFibonacciResults.get(key));
           //TODO: add assert for performance compare
        });

    }

    public List fibonacci1(int size) {
        List list = new ArrayList<>(size);
        int n0 = 1, n1 = 1, n2;
        list.add(n0);
        list.add(n1);

        for(int i=0;i < size - 2; i++) {
            n2 = n1 + n0;
            n0 = n1;
            n1 = n2;
            list.add(n2);
        }
        return list;
    }

    public List fibonacci2(int size) {
        return Stream.iterate(new int[]{1, 1}, x -> new int[]{x[1], x[0] + x[1]})
                .limit(size).map(x -> x[0])
                .collect(Collectors.toList());
    }

    @DataProvider
    public Object[][] getFibonacciSize() {
        return new Object[][]{
                {10},
                {50},
                {100},
                {1000},
                {10000}
        };
    }

    @Test(dataProvider = "getFibonacciSize", description = "old fibonacci", timeOut = 1000)
    public void testOldFibonacci(int size) {
        long duration = testFibonacci("testFibonacci1", size, x->fibonacci1(x));
        oldFibonacciResults.put(size, duration);
    }

    @Test(dataProvider = "getFibonacciSize", description = "lambda fibonacci", timeOut = 1000)
    public void testNewFibonacci(int size) {
        long duration = testFibonacci("testFibonacci2", size, x->fibonacci2(x));
        newFibonacciResults.put(size, duration);
    }

    public long testFibonacci(String name, int size, Function > func) {

        Stopwatch stopwatch = Stopwatch.createStarted();
        List list = func.apply(size);
        stopwatch.stop();
        long duration = stopwatch.elapsed(TimeUnit.MICROSECONDS);

        list.stream().forEach(x -> System.out.print(x +", "));
        System.out.println(String.format("\n--> %s (%d): %d\n" , name, size, duration));
        return duration;
    }

}

做了5组数列长度从10到10000 的测试, 输出结果如下

--- old vs. new ---
10: 34 vs. 28965
50: 9 vs. 154
100: 13 vs. 669
1000: 112 vs. 2600
10000: 1019 vs. 13548

不测不知道, 一测吓一跳, 新的写法看起来不错, 但是性能完败, 关键在于多做了两次转换(map , collect), 这里的测试代码用到了 @BeforeClass, @AfterClass, @Test, @DataProvider, TestNG 还有一些不错的功能, 比如 @threadPoolSize, @expectedExceptions, 详情参见 http://testng.org/doc/documentation-main.html

不知道你发现没有, 这里有个大问题, 这段测试代码缺少 Assert, 多数情况下对于功能测试必需要有 assert , 这些 assert 就是检查点, 没有检查点的测试起不到真正的作用. 你不可能去看每个测试的输出, 当然这里说的是单元测试,而对于性能测试, 一般要出一个性能测试的报告, Assert 检查点也不是必需的

所以我们应该这样写, 尽量多地加断言, 例如我们对 google 的 libphonenumber 作一个简单的测试

package com.github.walterfan.devaid.util;


import com.google.i18n.phonenumbers.NumberParseException;
import com.google.i18n.phonenumbers.PhoneNumberUtil;
import com.google.i18n.phonenumbers.Phonenumber;
import lombok.extern.slf4j.Slf4j;
import org.testng.annotations.Test;

import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.fail;

@Slf4j
public class PhoneNumberUtilTest {

    @Test
    public void testIsNumberNsnMatch() {
        String phoneNumberOne = "+86055112345678";
        String phoneNumberTwo = "86055112345678";
        PhoneNumberUtil.MatchType matchType = PhoneNumberUtil.getInstance().isNumberMatch(phoneNumberOne, phoneNumberTwo);
        log.info("matchType is {}", matchType);
        assertFalse(matchType == PhoneNumberUtil.MatchType.NO_MATCH);
        assertFalse(matchType == PhoneNumberUtil.MatchType.NOT_A_NUMBER);
        assertEquals(matchType , PhoneNumberUtil.MatchType.NSN_MATCH);



    }


    @Test
    public void testIsNumberShortMatch() {
        String phoneNumberOne = "+86055112345678";
        String phoneNumberTwo = "086(0551)1234-5678";
        PhoneNumberUtil.MatchType matchType = PhoneNumberUtil.getInstance().isNumberMatch(phoneNumberOne, phoneNumberTwo);
        assertFalse(matchType == PhoneNumberUtil.MatchType.NO_MATCH);
        assertFalse(matchType == PhoneNumberUtil.MatchType.NOT_A_NUMBER);
        assertEquals(matchType , PhoneNumberUtil.MatchType.SHORT_NSN_MATCH);
    }

    @Test
    public void testGetCountryCode() {

        String strPhoneNumber = "+86-0551-12345678";
        try {
            Phonenumber.PhoneNumber phoneNumber = PhoneNumberUtil.getInstance().parse(strPhoneNumber, "US");
            log.info("phoneNumber.getCountryCode() is {}", phoneNumber.getCountryCode());
            assertTrue(phoneNumber.getCountryCode() == 86);
        } catch (NumberParseException e) {
            fail(e.getMessage());
        }


    }
}

数据驱动测试

举例如下, 被测试类为 HttpUtil

public class HttpUtil
{
    public static boolean hasFieldValue(String httpHeader, String fieldKey, String fieldVal) {
        if(null == httpHeader || null == fieldKey || null == fieldVal) {
            return false;
        }

        String[] toggles = httpHeader.split(";");
        for(String toggle: toggles) {
            String[] toggleKeyVal = toggle.split("=");
            if(toggleKeyVal.length > 1) {
                String key = StringUtils.trim(toggleKeyVal[0]);
                String val = StringUtils.trim(toggleKeyVal[1]);

                if(fieldKey.equals(key) && fieldVal.equalsIgnoreCase(val))  {
                    return true;
                }
            }
        }
        return false;
    }

}

我们会用多个不同的 HTTP 头域来测试这个待测方法是否可正确地把相应头域的值判断出来, 用到的测试数据不必手工构造, 可以放在一个在 Object[][]为返回结果的方法中返回, 这些数据会逐个喂给测试方法, 决窍在于这个注解: @Test(dataProvider= "makeHttpHeadFields")

所以我们的一个测试方法最终会生成 8 个测试用例

具体代码如下

public class HttpUtilTest {

    @DataProvider
    public Object[][] makeHttpHeadFields() {

        return new Object[][] {
                { "acl_enabled= true", true },
                { "acl_enabled=true; auth_type=oauth", true },
                { "acl_enabled =TRue; auth_type=basic", true  },
                { "acl_enabled = false; auth_type=basic", false  },
                { " acl_enabled = ; auth_type=basic", false  },
                { "auth_type=basic", false  },
                { "", false  }

        };


    }

    @Test(dataProvider= "makeHttpHeadFields")
    public void testHasFieldValue(String toggleHeader, boolean ret) {
        Assert.assertEquals(HttpUtil.hasFieldValue(toggleHeader, "acl_enabled", "true") ,ret);

    }
}

运行结果如下

Test Results

对于单元测试的测试用例组织主要是要逻辑分支覆盖, 符合微服务实战测试之理论篇中所提到的三大原则

FIRST 原则
Right-BICEP
CORRECT 检查原则

还有很多线程测试, 性能测试, 压力测试, 异常测试, API 测试, 以及消费者契约测试,
这些测试我们后面慢慢道来, Mock 和 API 测试可参见微服务实战之Mock

下面我们就之前提到的测试用例的组织编写一个 TestCase 注解和它的注解处理器, 可在很方便地生成测试用例文档

编写注解来自动生成测试用例

package com.github.walterfan.hello.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;


@Target(ElementType.METHOD)
@Retention(RetentionPolicy.SOURCE)
public @interface TestCase {
    String value();
    String feature() default "";
    String scenario() default "";
    String given() default "";
    String when() default "";
    String then() default "";
    //String[] checkpoints();
}

在编译阶段处理注解并生成测试用例文档

package com.github.walterfan.hello.annotation;


import com.google.auto.service.AutoService;

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.Processor;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;


@SupportedSourceVersion(SourceVersion.RELEASE_8)
@SupportedAnnotationTypes("com.github.walterfan.hello.annotation.TestCase")
@AutoService(Processor.class)
public class TestCaseProcessor extends AbstractProcessor {
    public final static String TABLE_TITLE1 = "| # | feature | case | scenario | given | when | then |\n";
    public final static String TABLE_TITLE2 = "|---|---|---|---|---|---|---|\n";
    public final static String TABLE_ROW = "| %d | %s | %s | %s | %s | %s | %s |\n";

    private File testcaseFile = new File("./TestCases.md");
    private StringBuilder testcaseBuilder = new StringBuilder();
    private AtomicInteger testCaseNum = new AtomicInteger(0);

    @SuppressWarnings("unchecked")
    @Override
    public synchronized void init(ProcessingEnvironment processingEnv) {
        super.init(processingEnv);

        testcaseBuilder.append("# Testcases");
        testcaseBuilder.append("\n");
        testcaseBuilder.append(TABLE_TITLE1);
        testcaseBuilder.append(TABLE_TITLE2);
        try (BufferedWriter bw = new BufferedWriter(new FileWriter(testcaseFile))) {
            bw.write(testcaseBuilder.toString());
            bw.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @SuppressWarnings("unchecked")
    @Override
    public boolean process(Set annotations, RoundEnvironment roundEnvironment) {
        StringBuilder sb = new StringBuilder();
        for (TypeElement annotation : annotations) {

            for (Element element : roundEnvironment.getElementsAnnotatedWith(annotation)) {

                TestCase testCase = element.getAnnotation(TestCase.class);

                if (testCase != null) {
                    String line = String.format(TABLE_ROW, testCaseNum.incrementAndGet(), testCase.feature(), testCase.value(), testCase.scenario(), testCase.given(), testCase.when(), testCase.then());
                    sb.append(line);

                }
            }
        }

        try (BufferedWriter bw = new BufferedWriter(new FileWriter(testcaseFile, true))) {
            bw.write(sb.toString());
            System.out.println("testcases:\n" + sb.toString());
            bw.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return true;
    }
}

假设我们有一个简单的类 User

package com.github.walterfan.hello.annotation;


import lombok.Data;

import java.util.Calendar;
import java.util.Date;


@Data
public class User {
    private String name;
    private String email;
    private Date birthDay;


    public int age() {
        Calendar now = Calendar.getInstance();
        now.setTime(new Date());

        Calendar birth = Calendar.getInstance();
        birth.setTime(birthDay);

        return Math.abs(now.get(Calendar.YEAR) - birth.get(Calendar.YEAR));
    }

}

我们写一个测试类

package com.github.walterfan.hello.annotation;

import com.github.walterfan.hello.annotation.User;
import lombok.extern.slf4j.Slf4j;
import org.testng.annotations.Test;


import java.text.ParseException;
import java.text.SimpleDateFormat;

import java.util.Calendar;
import java.util.Date;

import static org.testng.Assert.assertEquals;


public class UserTest {

    @Test
    @TestCase(value = "testAge", feature = "UserManage", scenario = "CreateUser" ,given = "setBirthday", when="retrieveAge", then = "Age is current time minus birthday")
    public void testAge() throws ParseException {
        User user = new User();
        SimpleDateFormat formatter = new SimpleDateFormat("yyyy/MM/dd");
        Date birthDay = formatter.parse("1980/02/10");
        user.setBirthDay(birthDay);

        Calendar birthCal = Calendar.getInstance();
        birthCal.setTime(birthDay);

        int diffYear = Calendar.getInstance().get(Calendar.YEAR) - birthCal.get(Calendar.YEAR);
        System.out.println("diffYear: "+ diffYear);
        assertEquals(user.age(), diffYear);
    }

    @Test
    @TestCase(value = "testName", feature = "UserManage", scenario = "UpdateUser" ,given = "setName", when="retrieveName", then = "name is same")
    public void testName() throws ParseException {
        String name = "Walter";
        User user = new User();
        user.setName(name);
        user.getName().equals(name);
    }
}

编译这个类会自动生成一个 TestCase.md, 内容如下

 Testcases
| # | feature | case | scenario | given | when | then |
|---|---|---|---|---|---|---|
| 1 | UserManage | testAge | CreateUser | setBirthday | retrieveAge | Age is current time minus birthday |
| 2 | UserManage | testName | UpdateUser | setName | retrieveName | name is same |

也就是

#	feature	case	scenario	given	when	then
1	UserManage	testAge	CreateUser	setBirthday	retrieveAge	Age is current time minus birthday
2	UserManage	testName	UpdateUser	setName	retrieveName	name is same

完整代码可参见 https://github.com/walterfan/helloworld/tree/master/helloannotation

参考资料

TestNG 官方文档

微服务实战测试之Java实例篇