Robot Framework 失败用例重执行方案

背景

rerun

在基于robot framework框架进行自动化测试的时候，有时候测试用例可能由于一些外在原因导致失败（如网络中断，返回超时）等，并不是用例本身验证有问题；而且这些场景有可能是偶发的，因此我们需要一种失败用例重执行的机制。
robot framework 官方并没有提供类似retry等参数来配置失败用例重执行。仅仅提供了--rerunfailed参数对基于结果文件output.xml来选择重新执行失败的用例。

实现

我们先写一个简单的，有50%概率运行失败的测试脚本。

# 存放在tests目录下
*** Settings ***
Library  String

*** Test Cases ***
stable_test
    should be true  ${True}
unstable_test
    ${bool} =  random_boolean
    should be true  ${bool}
    
*** Keywords ***
random_boolean
    ${nb_string} =  generate random string  1  [NUMBERS]
    ${nb_int} =  convert to integer  ${nb_string}
    Run keyword and return  evaluate  (${nb_int} % 2) == 0

重试机制如下：

# first execute all tests
pybot --output original.xml tests 
# then re-execute failing
pybot --rerunfailed original.xml --output rerun.xml tests 
# finally merge results
rebot --merge original.xml rerun.xml

我们查看日志输出：

report

重试机制脚本化

我们编写一个脚本launch_test_and_rerun.sh

#!/bin/bash

# clean previous output files
rm -f output/output.xml
rm -f output/rerun.xml
rm -f output/first_run_log.html
rm -f output/second_run_log.html

echo
echo "#######################################"
echo "# Running portfolio a first time      #"
echo "#######################################"
echo
pybot --outputdir output $@

# we stop the script here if all the tests were OK
if [ $? -eq 0 ]; then
        echo "we don't run the tests again as everything was OK on first try"
        exit 0
fi
# otherwise we go for another round with the failing tests

# we keep a copy of the first log file
cp output/log.html  output/first_run_log.html

# we launch the tests that failed
echo
echo "#######################################"
echo "# Running again the tests that failed #"
echo "#######################################"
echo
pybot --outputdir output --nostatusrc --rerunfailed output/output.xml --output rerun.xml $@
# Robot Framework generates file rerun.xml

# we keep a copy of the second log file
cp output/log.html  output/second_run_log.html

# Merging output files
echo
echo "########################"
echo "# Merging output files #"
echo "########################"
echo
rebot --nostatusrc --outputdir output --output output.xml --merge output/output.xml  output/rerun.xml
# Robot Framework generates a new output.xml

运行脚本查看执行结果

 $ ./launch_test_and_rerun.sh test                                                                                                                     ✔

#######################################
# Running portfolio a first time      #
#######################################

==============================================================================
Test
==============================================================================
Test.Demo
==============================================================================
stable_test                                                           | PASS |
------------------------------------------------------------------------------
unstable_test                                                         | FAIL |
'False' should be true.
------------------------------------------------------------------------------
Test.Demo                                                             | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Test                                                                  | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Output:  /Users/wangyang/opg/demo/output/output.xml
Log:     /Users/wangyang/opg/demo/output/log.html
Report:  /Users/wangyang/opg/demo/output/report.html

#######################################
# Running again the tests that failed #
#######################################

==============================================================================
Test
==============================================================================
Test.Demo
==============================================================================
unstable_test                                                         | PASS |
------------------------------------------------------------------------------
Test.Demo                                                             | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Test                                                                  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /Users/wangyang/opg/demo/output/rerun.xml
Log:     /Users/wangyang/opg/demo/output/log.html
Report:  /Users/wangyang/opg/demo/output/report.html

########################
# Merging output files #
########################

Output:  /Users/wangyang/opg/demo/output/output.xml
Log:     /Users/wangyang/opg/demo/output/log.html
Report:  /Users/wangyang/opg/demo/output/report.html

总结

通过修改robot framework框架代码添加重试机制，会使框架变得复杂和不稳定，因此官方也不推荐修改源码方式增加失败重试机制，所以我们需要在外部逻辑对失败重试进行设计。