[变异测试] Major使用教程

[变异测试] Major使用教程

Major官网
Major文档
Major论文
Major下载

从官网下载Major的压缩包，1.3.0版本的Major支持JDK8。

使用Major做变异测试需要使用Major自带的javac，可以在bash文件中配置

export MAJOR_DIR=XXX
export PATH=$PATH:$MAJOR_DIR

也可以通过如下方式调用：

# 输入
Major路径/bin/javac -version
# 结果
javac 1.7.0-Major-v1.3.5

Major项目的结构：

1. MML文件

要想做变异测试，首先需要定义变异算子。在Major中，.mml文件负责定义使用的变异算子，bin/mmlc负责变异变异算子文件。

可以直接使用Major默认提供的.mml文件

拿tutorial.mml举例子，该文件中开启了STD、COR与ROR三个变异算子。允许变异的对象是
triangle.Triangle类中的classify(int,int,int)方法（一个项目可能有多个类与多个方法，因此我们可以指定需要变异的对象）。

targetOp{
    // Define the replacements for ROR
    BIN(>)->{>=,!=,FALSE};
    BIN(<)->{<=,!=,FALSE};
    BIN(>=)->{>,==,TRUE};
    BIN(<=)->{<,==,TRUE};
    BIN(==)->{<=,>=,FALSE,LHS,RHS};
    BIN(!=)->{<,>,TRUE,LHS,RHS};
    // Define the replacements for COR
    BIN(&&)->{==,LHS,RHS,FALSE};
    BIN(||)->{!=,LHS,RHS,TRUE};
    // Define the type of statement that STD should delete
    DEL(RETURN);

    // Enable the STD, COR, and ROR mutation operators
    STD;
    COR;
    ROR;
}
// Call the defined operator group for the target method
targetOp<"triangle.Triangle::classify(int,int,int)">;

解读一下，STD变异算子允许对RETURN语句进行删除。

DEL(RETURN);
// Enable the STD, COR, and ROR mutation operators
STD;

想要对tutorial.mml进行编译，生成后续变异分析要用的.mml.bin文件，只需要执行

major路径/bin/mmlc tutorial.mml tutorial.mml.bin

其中tutorial.mml.bin可以自己取名

2. 生成变异体

生成了.mml.bin文件后，就可以做变异分析了。
在major目录下执行执行

./bin/javac -XMutator=./mml/tutorial.mml.bin -d ./bin ./example/ant/src/triangle/Triangle.java

结果

#Generated Mutants: 86 (134 ms)

在major目录下生成mutants.log，可以查看变异体

1:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> a < 0
2:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> a == 0
3:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> true
4:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):11:b <= 0 |==> b < 0
5:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):11:b <= 0 |==> b == 0
6:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):11:b <= 0 |==> true
7:COR:||(boolean,boolean):!=(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 |==> a <= 0 != b <= 0
8:COR:||(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 |==> a <= 0
9:COR:||(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 |==> b <= 0
10:COR:||(boolean,boolean):TRUE(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 |==> true
11:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):11:c <= 0 |==> c < 0
12:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):11:c <= 0 |==> c == 0
13:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):11:c <= 0 |==> true
14:COR:||(boolean,boolean):!=(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 || c <= 0 |==> (a <= 0 || b <= 0) != c <= 0
15:COR:||(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 || c <= 0 |==> a <= 0 || b <= 0
16:COR:||(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 || c <= 0 |==> c <= 0
17:COR:||(boolean,boolean):TRUE(boolean,boolean):triangle.Triangle@classify(int,int,int):11:a <= 0 || b <= 0 || c <= 0 |==> true
18:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):12:return Type.INVALID; |==> <NO-OP>
19:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):14:a == b |==> a <= b
20:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):14:a == b |==> a >= b
21:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):14:a == b |==> false
22:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):16:a == c |==> a <= c
23:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):16:a == c |==> a >= c
24:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):16:a == c |==> false
25:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):18:b == c |==> b <= c
26:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):18:b == c |==> b >= c
27:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):18:b == c |==> false
28:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):20:trian == 0 |==> trian <= 0
29:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):20:trian == 0 |==> trian >= 0
30:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):20:trian == 0 |==> false
31:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):21:a + b <= c |==> a + b < c
32:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):21:a + b <= c |==> a + b == c
33:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):21:a + b <= c |==> true
34:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):21:a + c <= b |==> a + c < b
35:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):21:a + c <= b |==> a + c == b
36:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):21:a + c <= b |==> true
37:COR:||(boolean,boolean):!=(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b |==> a + b <= c != a + c <= b
38:COR:||(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b |==> a + b <= c
39:COR:||(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b |==> a + c <= b
40:COR:||(boolean,boolean):TRUE(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b |==> true
41:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):21:b + c <= a |==> b + c < a
42:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):21:b + c <= a |==> b + c == a
43:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):21:b + c <= a |==> true
44:COR:||(boolean,boolean):!=(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b || b + c <= a |==> (a + b <= c || a + c <= b) != b + c <= a
45:COR:||(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b || b + c <= a |==> a + b <= c || a + c <= b
46:COR:||(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b || b + c <= a |==> b + c <= a
47:COR:||(boolean,boolean):TRUE(boolean,boolean):triangle.Triangle@classify(int,int,int):21:a + b <= c || a + c <= b || b + c <= a |==> true
48:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):22:return Type.INVALID; |==> <NO-OP>
49:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):24:return Type.SCALENE; |==> <NO-OP>
50:ROR:>(int,int):!=(int,int):triangle.Triangle@classify(int,int,int):25:trian > 3 |==> trian != 3
51:ROR:>(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):25:trian > 3 |==> trian >= 3
52:ROR:>(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):25:trian > 3 |==> false
53:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):26:return Type.EQUILATERAL; |==> <NO-OP>
54:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):27:trian == 1 |==> trian <= 1
55:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):27:trian == 1 |==> trian >= 1
56:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):27:trian == 1 |==> false
57:ROR:>(int,int):!=(int,int):triangle.Triangle@classify(int,int,int):27:a + b > c |==> a + b != c
58:ROR:>(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):27:a + b > c |==> a + b >= c
59:ROR:>(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):27:a + b > c |==> false
60:COR:&&(boolean,boolean):==(boolean,boolean):triangle.Triangle@classify(int,int,int):27:trian == 1 && a + b > c |==> trian == 1 == a + b > c
61:COR:&&(boolean,boolean):FALSE(boolean,boolean):triangle.Triangle@classify(int,int,int):27:trian == 1 && a + b > c |==> false
62:COR:&&(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):27:trian == 1 && a + b > c |==> trian == 1
63:COR:&&(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):27:trian == 1 && a + b > c |==> a + b > c
64:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):28:return Type.ISOSCELES; |==> <NO-OP>
65:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):29:trian == 2 |==> trian <= 2
66:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):29:trian == 2 |==> trian >= 2
67:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):29:trian == 2 |==> false
68:ROR:>(int,int):!=(int,int):triangle.Triangle@classify(int,int,int):29:a + c > b |==> a + c != b
69:ROR:>(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):29:a + c > b |==> a + c >= b
70:ROR:>(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):29:a + c > b |==> false
71:COR:&&(boolean,boolean):==(boolean,boolean):triangle.Triangle@classify(int,int,int):29:trian == 2 && a + c > b |==> trian == 2 == a + c > b
72:COR:&&(boolean,boolean):FALSE(boolean,boolean):triangle.Triangle@classify(int,int,int):29:trian == 2 && a + c > b |==> false
73:COR:&&(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):29:trian == 2 && a + c > b |==> trian == 2
74:COR:&&(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):29:trian == 2 && a + c > b |==> a + c > b
75:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):30:return Type.ISOSCELES; |==> <NO-OP>
76:ROR:==(int,int):<=(int,int):triangle.Triangle@classify(int,int,int):31:trian == 3 |==> trian <= 3
77:ROR:==(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):31:trian == 3 |==> trian >= 3
78:ROR:==(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):31:trian == 3 |==> false
79:ROR:>(int,int):!=(int,int):triangle.Triangle@classify(int,int,int):31:b + c > a |==> b + c != a
80:ROR:>(int,int):>=(int,int):triangle.Triangle@classify(int,int,int):31:b + c > a |==> b + c >= a
81:ROR:>(int,int):FALSE(int,int):triangle.Triangle@classify(int,int,int):31:b + c > a |==> false
82:COR:&&(boolean,boolean):==(boolean,boolean):triangle.Triangle@classify(int,int,int):31:trian == 3 && b + c > a |==> trian == 3 == b + c > a
83:COR:&&(boolean,boolean):FALSE(boolean,boolean):triangle.Triangle@classify(int,int,int):31:trian == 3 && b + c > a |==> false
84:COR:&&(boolean,boolean):LHS(boolean,boolean):triangle.Triangle@classify(int,int,int):31:trian == 3 && b + c > a |==> trian == 3
85:COR:&&(boolean,boolean):RHS(boolean,boolean):triangle.Triangle@classify(int,int,int):31:trian == 3 && b + c > a |==> b + c > a
86:STD:<RETURN>:<NO-OP>:triangle.Triangle@classify(int,int,int):32:return Type.ISOSCELES; |==> <NO-OP>

也可以使用ant来做变异，进入到example/ant目录，把mml.bin文件拷贝进来

然后执行即可得到和上面一样的mutants.log

ant -DmutOp="=tutorial.mml.bin" compile

解释一下如何配置ant，进入到build.xml
需要配置三个property

    <property name="mutOp" value=":NONE"/>
    <property name="mutator" value="-XMutator${mutOp}"/>
    <property name="major" value="../../bin/javac"/>

然后在compile这一个target中配置 executable=${major} 以及 标签。开启export=True即可导出变异体。

    <target name="compile" depends="init" description="Compile">
        <javac includeantruntime="true" 
               srcdir="src"
              destdir="bin"
                debug="yes"
	             fork="yes"
	       executable="${major}">
	       <!-- 导出mutants文件
 			<compilerarg value="-J-Dmajor.export.mutants=true"/>
 			-->
            <compilerarg value="${mutator}"/>
        </javac>
    </target>

mutants目录下即可得到全部的mutants的.java文件

执行compile后执行

ant mutation.test

得到

会额外生成文件，保存变异分析结果

具体可以参考example/ant/run.sh脚本。

PLUS，如果想生成更多mutation analysis的细节，可以修改buil.xml中的junit，给一个示例。

        <junit
            printsummary="false"
            showoutput="false"
            fork="no"
            exportKillMap="true"
            resultFile="results.csv"
            mutationAnalysis="true"
            exportCovMap="true"
            timeoutFactor="16"
            sort="sort_methods"
            haltonfailure="true"
            killDetailsFile="${major.kill.log}"
            excludeFile="${major.exclude}"
            >

3. 再谈Mutator

Major支持如下变异算子

all.mml脚本虽然enable了所有变异算子，但是LVR默认是不起作用的，需要添加LIT()才能生成LVR类型的变异体。

LIT(BOOLEAN);
LIT(NUMBER);
LIT(STRING);

想要生成全部的mutants：

# build.xml
<property name="mutOp" value=":NONE" />
<property name="mutator" value="-XMutator${mutOp}" />
<property name="major" value="路径/bin/javac" />

<javac fork="yes" executable="${major}">
	<compilerarg value="${mutator}"/>
	<compilerarg value="-J-Dmajor.export.mutants=true"/>
</javac>

# shell

#!/bin/sh
$MAJOR_HOME/bin/ant \
-f 路径/build.xml \
-DmutOp=":ALL" \
clean compile 

参考 参考链接