在Scala上写单测
下面是之前写的英文版简述，直接粘贴过来了。 你还可以参考：Spark-Scala单元测试实践 - 码农教程 (manongjc.com)
Unit Tests
Unit testing is a powerful tool for software quality -- and has been for decades. Unit tests provide a fundamental check that an application meets its software design specifications and behaves as intended.
What tools are used?
UnitTest Tool: ScalaTest
Mock Tool: Mockito Scala
HTTP Mock Tool: WireMock
Code Coverage Tool: Scoverage
How to write tests?
First, you need to know that writing a single test is easy. More information on ScalaTest
Tips:
- You can mock functions by Mockito Scala
- You can mock http service by WireMock
- When there are public functions, make full use of
before,afterandwithFixture.
重点来了，如何做增量覆盖率测试？
我们用 Scoverage 做覆盖率测试，但是它不支持增量覆盖率测试，所以我们需要手动改造一下。 如果你用 jacoco做的覆盖率测试，可以试试 jacoco: jacoco二开，支持增量代码覆盖率 (gitee.com) 这篇文章。
大概的思路是：
- Calculate the number of files and lines with changes through git diff
- Using the feature of the scoverage, comments are added before and after the number of lines that have changed.
see: https://github.com/scoverage/scalac-scoverage-plugin
贴上python脚本： HandleIncrementalCoverage.py
#!/usr/bin/env python
"""
Handle code for Incremental Coverage
Tips: only active with scoverage
Principle:
1. Calculate the number of files and lines with changes through git diff
2. Using the feature of the scoverage, comments are added before and after the number of lines that have changed.
see: https://github.com/scoverage/scalac-scoverage-plugin
"""
import os
import re
import subprocess
import sys
def getChangedLineInfoFromDiffLines(lines):
"""
args : Lines, the description of a file output by the git-diff command
returns : List[(changedLineStart, changedLineEnd)], Front-closed and back-open interval
"""
changedLineInfo = []
# Get line change information according to "@"
# Matching [0]: "," + the number of rows deleted from here;
# [1]: the number of rows added here;
# [2]: "," + the number of rows added from here
reg = re.compile("^@@ -[0-9]+(,[0-9]+)? \+([0-9]+)(,[0-9]+)? @@")
for line in lines:
r = reg.findall(line)
if len(r) > 0:
changedLineStart = int(r[0][1])
caughtLineCountStr = r[0][2]
if len(caughtLineCountStr) > 0:
changedLineCount = int(caughtLineCountStr[1:])
else:
changedLineCount = 1
changedLineInfo.append((changedLineStart, changedLineStart + changedLineCount))
return changedLineInfo
def getDiffLines(baseBranch='HEAD~1', newBranch='HEAD', dir="./"):
"""get diff lines from two branches."""
gitCmd = f"git diff --unified=0 --diff-filter=d {baseBranch} {newBranch} {dir}"
print("Git Cmd: ", gitCmd)
gitDiffOutputRaw = subprocess.check_output(gitCmd.split(" "))
outputStr = gitDiffOutputRaw.decode('utf-8')
diffOutputLines = outputStr.splitlines()
map = {}
separateLineReg = re.compile("^diff --git a/\S+ b/(\S+)")
currentCheckFileName = ""
diffLinesForCurrentCheckFile = []
for i in range(len(diffOutputLines)):
l = diffOutputLines[i]
separateLineMatchResult = separateLineReg.findall(l)
if len(separateLineMatchResult) > 0:
if len(diffLinesForCurrentCheckFile) > 0:
a = getChangedLineInfoFromDiffLines(diffLinesForCurrentCheckFile)
map[currentCheckFileName] = a
diffLinesForCurrentCheckFile.clear()
# filter submodule name
currentCheckFileName = '/'.join((separateLineMatchResult[0]).split('/')[1:])
else:
diffLinesForCurrentCheckFile.append(l)
if i == len(diffOutputLines) - 1:
a = getChangedLineInfoFromDiffLines(diffLinesForCurrentCheckFile)
map[currentCheckFileName] = a
print("Git Diff Output: ", map)
return map
def findAllFile(base):
for root, ds, fs in os.walk(base):
for f in fs:
if f.endswith('.scala'):
fullname = os.path.join(root, f).replace('\\', '/')
yield fullname
def preHandleIncrementalCoverage(diffDict):
"""PreHandle Incremental Coverage"""
for i in findAllFile("src/main/scala"):
diffInfo = diffDict[i] if i in diffDict else []
print("Add annotation for:", i, diffInfo)
fileData, diffLen = [], len(diffInfo)
with open(i, "r", encoding="utf-8") as f:
lineNum, diffIndex, diffOffset = 0, 0, 0
for line in f:
if "auto add for Incremental Coverage" in line:
continue
lineNum += 1
if lineNum == 2:
fileData.append("// $COVERAGE-OFF$ auto add for Incremental Coverage\n")
if diffIndex < diffLen and diffInfo[diffIndex][diffOffset] == lineNum:
if diffOffset == 0:
fileData.append("// $COVERAGE-ON$ auto add for Incremental Coverage\n")
diffOffset = 1
else:
fileData.append("// $COVERAGE-OFF$ auto add for Incremental Coverage\n")
diffOffset = 0
diffIndex += 1
fileData.append(line)
with open(i, "w", encoding="utf-8") as f:
f.write("".join(fileData))
def cleanIncrementalCoverage():
"""clean Incremental Coverage"""
for i in findAllFile("src/main/scala"):
fileData = []
with open(i, "r", encoding="utf-8") as f:
for line in f:
if "auto add for Incremental Coverage" in line:
continue
fileData.append(line)
with open(i, "w", encoding="utf-8") as f:
f.write("".join(fileData))
if __name__ == '__main__':
# os.getcwd() is the root directory of the submodule
if len(sys.argv) == 2 and sys.argv[1] == "clean":
cleanIncrementalCoverage()
exit(0)
if len(sys.argv) != 3:
raise Exception("Argv not enough. Usage: python3 HandleIncrementalCoverage.py baseBranch newBranch")
baseBranch = sys.argv[1]
newBranch = sys.argv[2]
diff = getDiffLines(baseBranch, newBranch, "./src/main/scala")
preHandleIncrementalCoverage(diff)
对应的再改一下pom.xml
注意：这里的依赖只是ut用到的一些依赖和插件，不包括项目用的，请根据实际情况进行改造
3.2.3
3.1.2
2.12
2.12.10
3.2.0
2.5.0
3.6.6
4.5.6
false
true
HEAD~1
HEAD
true
true
org.scalatest
scalatest_${scala.binary.version}
${scalatest.version}
test
org.scalactic
scalactic_${scala.binary.version}
3.2.12
test
org.pegdown
pegdown
1.4.2
test
com.vladsch.flexmark
flexmark-all
0.35.10
test
org.mockito
mockito-scala_${scala.binary.version}
1.16.37
test
org.apache.hadoop
hadoop-azure
${hadoop.version}
test
com.github.tomakehurst
wiremock-jre8-standalone
2.33.2
test
${basedir}/src/main/scala
org.apache.maven.plugins
maven-compiler-plugin
net.alchim31.maven
scala-maven-plugin
org.apache.maven.plugins
maven-jar-plugin
${project.parent.basedir}/target/
org.scalatest
scalatest-maven-plugin
1.0
${project.build.directory}/surefire-reports
.
WDF TestResult.txt
${project.build.directory}/site/scalatest
false
${skipUT}
test
test
test
org.codehaus.mojo
exec-maven-plugin
add annotation for incremental coverage
compile
exec
${notIncrementalCoverage}
python
HandleIncrementalCoverage.py ${baseBranch} ${newBranch}
remove annotation for incremental coverage
clean
exec
python
HandleIncrementalCoverage.py clean
org.scoverage
scoverage-maven-plugin
${scoverage.plugin.version}
test
test
report
${skipUT}
${scala.version}
true
true
${project.build.sourceEncoding}
计算增量覆盖率：
echo "calculate incremental coverage between master and HEAD"
mvn -P spark-3.1 test -DargLine="-DnotIncrementalCoverage=false" "-DbaseBranch=origin/master" "-DnewBranch=HEAD" "-DnotIncrementalCoverage=false"
计算全量覆盖率：
mvn -P spark-3.1 test