5种优化持续集成测试的方法

目录

原链接

翻译内容

1. Create tiny but valuable test suites 创建小而有价值的测试套件

2. Refactor the test setup 重构测试设置

3. Be smart with your wait times 等待时间要聪明设置

4. Trigger tests automatically 自动化触发测试

5. Run tests in parallel 并发执行测试

关于作者

John Ruberto

---

原链接

https://www.stickyminds.com/article/5-ways-optimize-tests-continuous-integration

翻译内容

“Test early, test often.” If you’ve worked with me, you are probably sick of hearing me say that. But it’s common sense that if your tests find a problem shortly after that problem was created, it will be easier to fix. This is one of the principles that makes continuous integration a very powerful strategy.

“早点测试，经常测试。”如果你和我一起工作过，你可能会因为听到我这么说而感到厌倦。 但常识是，如果您的测试在问题出现后不久，便能发现问题，那么修复起来会更容易。 这是使持续集成成为一种非常强大的策略的原则之一。

Many times I’ve seen teams that have many automated tests but are not using those tests as part of a continuous integration program. There are often several reasons why the team thinks it cannot use these tests with continuous integration. Maybe the tests take too long to execute, or they are not reliable enough to give accurate results themselves and it will take humans to interpret the results.

很多时候，我见过团队有很多自动化测试，但没有将这些测试作为持续集成计划的一部分。 团队认为不能在持续集成中使用这些测试通常有几个原因。 也许测试执行时间太长，或者它们不够可靠来给出准确的结果，并且需要人类来解释测试结果。

When assessing these suites, I start with a simple exercise. First, I draw two axes on a white board. The vertical axis shows the value of the tests and the horizontal axis shows the time it takes the suite to execute.

在评估这些测试套件时，我先从一个简单的练习开始。 首先，我在白板上画两个轴。 纵轴表示测试的值，横轴表示套件执行的时间。

The team and I then write down the name of each test suite on a sticky note and place them on the appropriate place on the board. The chart below shows an example of a grid, showing how each test suite measures.

然后我和团队在便利贴上写下每个测试套件的名称，并将它们放在板上的适当位置。 下图显示了一个网格示例，显示了每个测试套件的测量方式。

When we talk about the value of the tests, it’s based on the team’s subjective opinion, so we keep the choices pretty simple: No Value, Moderate Value, High Value, and Very High Value. We base this opinion on the reliability of the tests, or the tests’ ability to give accurate results each time they are executed, and the level of confidence the tests give the team for the quality of the system.

当我们谈论测试的价值时，它基于团队的主观意见，因此我们的选择非常简单：没有价值，中等价值，高价值和非常高价值。 我们将这种观点建立在测试的可靠性或测试每次执行时给出准确结果的能力，以及测试给团队提供系统质量的信心水平。

For example, some test suites are a must-have when it comes to making a decision, but the results are a little flaky, and when they fail sometimes for no apparent reason, a person has to re-execute those failing tests manually. We might still call this test suite High Value, but if it ran accurately every time, we would call it Very High Value.

例如，一些测试套件在做出决定时是必须的，但结果有点不稳定，有时当它们因为没有明显原因而失败时，一个人必须手动重新执行那些失败的测试。 我们可能仍然称这个测试套件为High Value，但如果它每次都准确运行，我们称之为非常高的价值。

On the other side of the coin, sometimes there is a test suite that gets executed because it’s part of a checklist, but no one really knows what the results mean. Maybe the original author has left the team and no one picked up the ownership of that suite. We would put that suite in the No Value category.

另一方面，有时会有一个测试套件被执行，因为它是清单的一部分，但没有人真正知道结果意味着什么。 也许原作者离开了团队，没有人获得该套件的所有权。 我们会将该套件放在No Value类别中。

The horizontal axis is easier to determine: It’s simply the number of minutes that it takes to execute that suite.

横轴更容易确定：它只是执行该套件所需的分钟数。

Now that you have an assessment of each suite, it’s time to think about improving your suites by making them more valuable (moving up in the diagram) or by helping them execute faster (moving to the left).

现在您已经对每个套件进行了评估，现在是时候考虑通过使它们更有价值（在图中向上移动）或帮助它们更快地执行（向左移动）来改进套件了。

For continuous integration, I like to target the tests into four categories: 对于持续集成，我喜欢将测试分为四类：

• 在十分钟或更短时间内，执行具有极高值的测试
  这些测试可以在每次构建时运行。 这些用于接受构建以进行进一步测试; 在这些测试通过之前，团队都应该考虑构建失败情况。 您的开发人员不希望等待超过十分钟才能获得构建结果。

• Tests with High Value or better that execute in an hour or less 在一小时或更短时间内，执行具有高价值或更高价格的测试
  These tests can run continuously. For example, you might configure these tests to execute every hour and start again as soon as they are complete. If there isn’t a new build ready yet, you can wait until the next build completes.这些测试可以持续运行。 例如，您可以将这些测试配置为每小时执行一次，并在完成后立即重新开始。 如果尚未准备好新构建，则可以等到下一个构建完成。

• Tests with High Value or better that take longer than an hour to execute超过一小时时，执行具有高价值或更高价值的测试
  These tests can run daily—or, usually, nightly, so that the results are available when the working day begins for your team.这些测试可以每天运行 - 或者通常每晚运行一次，以便您的团队在工作日时获得结果

• Tests with Moderate Value 具有中等价值的测试
  These tests can run once per week or once per release cycle. 这些测试可以每周运行一次或每个发布周期运行一次。

Notice that I did not include tests with No Value. These should be improved to add value, or just dropped from your execution. It doesn’t make sense to keep test suites that don’t add value.

请注意，我没有包含No Value的测试。 这些应该被改进以增加价值，或者只是从执行中删除。 留着不增加价值的测试套件是没有意义的。

I chose the time boundaries of ten minutes and one hour based on input from the development teams. They want to get quick feedback. You can imagine a developer waiting for the build results to complete successfully before going to lunch. Your timelines may vary based on your realities; this is just a framework to show the thought process behind selecting the tests that run with the build versus hourly.

我根据开发团队的意见选择了10分钟和1小时的时间边界。 他们希望获得快速反馈。 您可以想象一个开发人员等待构建结果在午餐前成功完成。 您的时间表可能因您的实际情况而异; 这只是一个框架，用于显示选择使用构建与每小时运行的测试背后的思考过程。

A huge benefit for executing the tests this frequently is that you are likely to have very few code changes between a successful test run and a failed test run, making it easier to isolate the change that caused the test to fail.

经常执行测试的一个巨大好处是，您可能在成功的测试运行和失败的测试运行之间进行很少的代码更改，从而更容易隔离导致测试失败的更改。

There are several strategies that have been useful for optimizing existing tests for the continuous integration suites. Here are five proven practices.

有几种策略可用于优化持续集成套件的现有测试。 以下是五种经过验证的做法

1. Create tiny but valuable test suites 创建小而有价值的测试套件

Choose the most important tests and pull them into a smaller suite that runs faster. These are usually very gross-level tests, but they’re necessary to qualify your system or app for further testing. If these tests don’t pass, it doesn’t make sense to proceed.

选择最重要的测试并将它们拉入一个运行速度更快的小套件中。 这些通常是非常粗略的测试，但它们是使您的系统或应用程序有资格进行进一步测试所必需的。 如果这些测试没有通过，那么继续进行是没有意义的。

A good starting point would be to create a new entity and perform the most important operation on that entity. For example, if it’s a note-taking app, the test would be to create a note, add text, close the app, then reopen and verify that the text was saved. If your note-taking app can’t save a note, there isn’t much use in proceeding to other tests.

一个好的起点是创建一个新实体并对该实体执行最重要的操作。 例如，如果它是一个笔记记录应用程序，测试将是创建一个笔记，添加文本，关闭应用程序，然后重新打开并验证文本已保存。 如果您的笔记记录应用程序无法保存笔记，则进行其他测试没有太大用处。

We often call these build acceptance tests or build verification tests. If you already have these suites, great; just make sure they execute quickly.

我们经常称这些为构建验收测试或构建验证测试。 如果你已经拥有这些套件，那就太好了; 只是确保他们快速执行。

2. Refactor the test setup 重构测试设置

Tests generally have a setup, then perform verification. For the note-taking example, to verify that the app opens with the previous text present, you first have to set up the test with the text. Examine how your tests are doing the test setup and see if there is a better way.

测试通常具有设置，然后执行验证。 对于笔记记录示例，要确认应用程序打开时是否存在上一个文本，您首先必须使用文本设置测试。 检查你的测试是如何测试设置的，看看是否有更好的方法。

For example, one team had a suite of UI-driven tests that took a long time to execute and had many false failures due to timing issues and minor UI tweaks. We refactored that suite to perform the test setup via API commands and do the verification through the UI. This updated suite had the same functional coverage, but it executed 70 percent faster and had about half the false failures caused by UI changes.

例如，一个团队有一套UI驱动的测试需要花费很长时间才能执行，并且由于时序问题和微小的UI调整而导致许多错误的失败。 我们重构了该套件，通过API命令执行测试设置，并通过UI进行验证。 这个更新的套件具有相同的功能覆盖范围，但它的执行速度提高了70％，并且大约有一半因UI更改而导致的错误故障。

3. Be smart with your wait times 等待时间要聪明设置

We have all done it: A flaky test keeps failing because the back end didn’t respond in time or some resource is still loading, so we put a sleep statement in. We intended that to be a temporary workaround, but that was a year ago now.

我们都已经完成了它：一个不稳定测试不断失败，因为后端没有及时响应或某些资源仍在加载，所以我们把睡眠语句考虑进来了。我们打算这是一个临时的解决方法，但那是一年以前。

Look for those dreaded sleep statements and see if you can replace them with a smarter wait statement that completes when the event happens, instead of a set period of time.

查找那些可怕的睡眠语句，看看是否可以用更智能的等待语句替换它们，该等语句在事件发生时完成，而不是设定的一段时间。

4. Trigger tests automatically 自动化触发测试

You may have several test suites that are normally initiated by a person during the test phase of a project. Often, it only takes a little shell scripting to be able to include these tests in the continuous integration suite.

您可能有几个测试套件，通常由一个人在项目的测试阶段启动。 通常，只需要一些shell脚本就可以将这些测试包含在持续集成套件中。

Security and performance tests are two examples of types of tests that might be performed by a specialist who is not part of the standard test team, so those tests might not be configured for automatic execution. The other benefit of running these tests frequently is that the problems that are found are frequently difficult to fix, so if the problem is identified sooner, the team has more time to fix it. These tests are often classified as Very Valuable, but they take more than an hour to execute, so they are typically executed daily.

安全性和性能测试是可能由不属于标准测试团队的专家执行的测试类型的两个示例，因此这些测试可能未配置为自动执行。 经常运行这些测试的另一个好处是发现的问题通常很难修复，因此如果问题得到更早发现，团队就有更多时间来解决它。 这些测试通常被归类为非常有价值，但它们需要一个多小时才能执行，因此它们通常每天执行。

5. Run tests in parallel 并发执行测试

Virtual machines and cloud computing services coupled with tools that help automatically set up environments and deploy your code make it much more affordable to run tests in parallel. Examine the test suites that take some time to execute and look for opportunities to run those tests in parallel.

虚拟机和云计算服务以及有助于自动设置环境和部署代码的工具，使得并行运行测试的成本更低。 检查需要一些时间执行的测试套件，并寻找并行运行这些测试的机会。

On one team, we had a very vital test suite that contained five hundred test cases. This suite took several hours to run, so we didn’t execute it very often. It was a very broad-based test, touching many different features. We were able to break that suite up into about a dozen different suites that could run in parallel, so we could run the tests more frequently (nightly instead of weekly), and we could tell more quickly where any problems lay because the new suites were organized by feature.

在一个团队中，我们有一个非常重要的测试套件，包含500个测试用例。 这个套件需要几个小时才能运行，所以我们没有经常执行它。 这是一个非常广泛的测试，涉及许多不同的功能。 我们能够将该套件分成几十个可以并行运行的不同套件，因此我们可以更频繁地运行测试（每晚而不是每周），我们可以更快地告诉我们问题所在，因为新的套件是 按功能组织。

Improving the value of your test suites and the time it takes to execute them can help you optimize your existing test suites to fit into a continuous integration program.

提高测试套件的价值以及执行测试套件所需的时间可以帮助您优化现有的测试套件，以适应持续集成计划。

关于作者

John Ruberto

John Ruberto采用工程方法将质量建设成软件。 他在Software Leadership Academy，在他的博客和会议发言中，教授应用于质量的领导原则。

John目前正在咨询质量工程实践和辅导工程师，以成为更好的领导者。 他曾担任First Data，Concur，Intuit，Alcatel，PhoenixBIOS和Boeing的质量工程副总裁。 他拥有Purdue的BSEE，华盛顿的MSCS和SJSU的MBA。 您可以在@JohnRuberto或blog.ruberto.com找到John。