测试意外：DevOps测试中的右移

目录

原链接

翻译内容

Summary(摘要):

正文

A Case for Shifting Right(转移权的案例)

Finding New Ways to Test(寻找新的测试方法)

Shaping the Future of Automation(塑造自动化的未来)

关于作者

自己的一点心得

---

原链接

https://www.stickyminds.com/article/testing-unexpected-shift-right-devops-testing

翻译内容

By Stefan Friese - September 26, 2016(Stefan Friese 写于2016年9月26日)

Summary(摘要):

When it comes to testing in DevOps, more than simple regression checking can be automated. By shifting right in the lifecycle and testing in production, you can analyze the undefined, unknown, and unexpected by relying on real traffic and unpredictable test input. With shorter implementation and release cycles, testing and production come closer together.

谈到在DevOps中进行测试时，远远不是实现简单的回归检查。 通过在生命周期中进行右移并在生产环境中进行测试，您可以依靠实际流量和不可预测的测试输入来分析未定义，未知和意外。 通过缩短实施和发布周期，测试和生产更加紧密。

正文

DevOps is not only about developing and releasing applications; testing is also an essential part of the practice. But the role of testing in DevOps is not as visible as the other practices, and there are often disputes about how testing should be performed.

DevOps不仅仅是关于开发和发布应用程序; 测试也是实践的重要组成部分。 但是，在DevOps中测试的作用并不像其他实践那样明显，并且经常存在关于如何执行测试的争议。

Due to the increasingly automated nature of the software delivery lifecycle in DevOps, continuous testing has become a popular movement. However, test automation gives many testers pause. Though automation can be helpful, particularly in a DevOps environment, it cannot (and should not) replace testers entirely.

由于DevOps中软件交付生命周期越来越自动化，连续测试已经变得流行。 但是，测试自动化让许多测试人员停下来。 虽然自动化可能会有所帮助，特别是在DevOps环境中，但它不能（也不应该）完全替代测试人员。

In particular, the context-driven testing community emphasizes that testing and automated checking are different and that validating expected results is not testing. Through exploring and experimenting, testers play a crucial role in ensuring the quality of software products.

特别是，上下文驱动的测试社区强调了测试和自动检查是不同的，并且验证预期结果并不是测试。 通过探索和实验，测试人员在确保软件产品质量方面发挥着至关重要的作用。

However, by insisting that real testing activities cannot be automated at all, testers are left out of the continuous testing conversation. That introduces the risk that important improvements in test automation will be shaped more by software engineers than by the testing community. There is a need to find an adequate response to the demand to orient testing toward an accelerating pace of development, with shorter implementation and release cycles. The question is not whether testing will change, but rather who will drive the innovations in testing.

但是，通过坚持真正的测试活动根本无法实现自动化，测试人员被排除在持续测试之外。 这引入了风险，即在测试自动化的重要改进方面，软件工程师可能比测试社区会做的更多。 需要找到对需求的充分响应，以便让测试加快开发速度，同时缩短实施和发布周期。 问题不在于测试是否会改变，而是谁将推动测试的创新。

A Case for Shifting Right(转移权的案例)

You’ve probably heard of the “shift left” movement in software testing, describing the trend toward teams starting testing as early as possible, testing often throughout the lifecycle, and working on problem prevention instead of detection. The goals are to increase quality, shorten test cycles, and reduce the possibility of unpleasant surprises at the end of the development cycle or in production.

您可能已经听说过软件测试中的“左移”运动，描述了团队尽早开始测试的趋势，在整个生命周期中经常进行测试，以及做工作来预防问题而不是检测问题。 目标是提高质量，缩短测试周期，并减少在开发周期结束或生产中出现令人不快的意外的可能性。

But there’s another new idea: shift right. In this approach, testing moves right into production in regard to functionality, performance, failure tolerance, and user experience by employing controlled experiments. By waiting to test in production, you may find new and unexpected usage scenarios.

但还有另一个新想法：右移。 在这种方法中，通过采用受控实验，测试可以在功能，性能，容错和用户体验方面直接进入生产阶段进行。 通过等待在生产中进行测试，您可能会发现新的和意外的使用场景。

The popular test strategy model of the four classifies tests based on whether they are business- or technology-facing and if they more guide development or critique the product. However, Gojko Adzic observed that "with shorter iterations and continuous delivery, it’s difficult to draw the line between activities that support the team and those that critique the product." As an alternative, he suggested using the distinction of whether you’re checking for expected outputs or looking to analyze the undefined, unknown, and unexpected:

存在四种流行的测试策略模型，其分类依据是根据它们是面向业务还是面向技术，以及它们是否更多地指导开发或评估产品。 然而，Gojko Adzic观察到“通过较短的迭代和持续交付，很难在支持团队的活动和评估产品的活动之间划清界线。” 作为替代方案，他建议使用您是检查预期输出，还是寻找分析未定义，未知和意外，来做区分：

Let’s look at some perhaps unfamiliar testing approaches that focus on the unknown and unexpected.

让我们看一些可能不熟悉的测试方法，这些方法专注于未知和意外。

Finding New Ways to Test(寻找新的测试方法)

Testing web services with live production traffic
The goal of testing is to ensure that the software product works for the consumer. Therefore, the best input for tests is the actual production traffic. This does not necessarily mean that the testing has to be done in production; by capturing, duplicating, and redirecting live production traffic to test environments, it is possible to use real-time production API calls as test input. By using live production traffic to test the new version of a web service prior to its release in production, it is possible to determine if unpredictable changes in API usage cause unexpected behavior, such as slower response times or deviations in the CPU consumption.

使用实时生产流量测试Web服务
测试的目标是确保软件产品适用于消费者。 因此，测试的最佳输入是实际生产流量。 这并不一定意味着必须在生产中进行测试; 通过捕获，复制和重定向实时生产流量到测试环境，可以使用实时生产API调用作为测试输入。在生产中发布之前， 通过使用实时生产流量来测试新版本的Web服务，可以确定API使用中的不可预测的更改是否会导致意外行为，例如响应时间较慢或CPU消耗偏差。

Twitter uses an automated testing tool called Diffy that finds bugs by acting as a proxy, receiving HTTP requests and directing them to an instance of the new version and to the production version, side by side. It then compares the responses and reports any regressions. Following this approach, API calls from production can be redirected to two product variations and an automated check can detect regression in their responses. For example, for an e-commerce search service, the tool could detect variations in the number of results and their ranking. It cannot determine if end-users would sense that the results are correct, as could be done by exploratory testing or by taking actual user feedback into account. But with the possibility to compare many samples based on unpredictable production requests, this automated check is a good basis to detect deviations and, thus, provide input for further manual analysis.

Twitter使用名为Diffy的自动化测试工具，它通过充当代理，接收HTTP请求并将它们并排引导到新版本的实例和生产版本来查找缺陷。 然后，它会比较响应并报告任何回归。 按照这种方法，生产中的API调用可以重定向到两个产品变体中，自动检查可以检测其响应中的回归部分。 例如，对于电子商务搜索服务，该工具可以检测结果数量及其排名的变化。 它无法确定最终用户是否会感觉到结果是正确的，这可以通过探索性测试或考虑实际的用户反馈来完成。 但是，有可能根据不可预测的生产请求，比较许多样品，这种自动检查是检测偏差的好方法，从而为进一步的人工分析提供输入。

A/B testing (A / B测试)
Compared to the previous approach, A/B testing goes one step farther because it is based on feedback from real users. It compares two versions of a product hosted in production. You split your website production traffic between the two versions to measure metrics such as the conversion rate. A/B testing is done in production by real customers, providing a way of critiquing the product.

与之前的方法相比，A / B测试更进了一步，因为它基于真实用户的反馈。 它比较了生产中产品的两个版本。 您可以在两个版本之间拆分网站生产流量，以衡量转化率等指标。 A / B测试由真实客户在生产中完成，提供了一种评估产品的方式。

Canary release(金丝雀发布)
The canary release strategy is used to reduce the risk of introducing a new software version in production. This is done by slowly rolling out the change to a small subset of instances before applying it to the entire infrastructure.

金丝雀发布策略用于降低在生产中引入新软件版本的风险。 这是将更改应用于整个产品用户之前，通过将更改缓慢地推广到一小部分用户来完成的。

The difference between a canary release and A/B testing is that A/B testing is for measuring functionality in the application, while the canary release does not look at user feedback. The canary release, as well as other release strategies such as the staged release, is about releasing new software safely, preferably detecting failures automatically and rolling back predictably.

金丝雀发布和A / B测试之间的区别在于，A / B测试用于测量应用程序中的功能，而金丝雀版本不考虑用户反馈。 金丝雀发布以及其他发布策略（例如分阶段发布）是关于安全发布新软件，最好是自动检测故障并可预测地回滚。

Testing in production(在生产中进行测试)
Why would you test in production and not only in dedicated test environments? Testing in production makes it possible to draw from real users and analyze use cases that are difficult to anticipate. In the end, the only way to prove that software is ready for production is to release it in production.

为什么要在生产中进行测试，而不仅仅是在专用测试环境中呢？ 生产中的测试可以从真实用户中实现用户画像，并分析难以预料的用例。 最后，证明该软件已准备好的唯一方法是在生产中发布它。

The traditional approach is to execute as many tests as possible prior to the release until confidence is reached that the product is robust and will not fail. For DevOps organizations, proper testing is also crucial, but they pay particular attention to the ability to fail small and fast if a failure happens in production. This can be achieved by deploying only small changes and releasing incrementally while monitoring closely that the application is healthy. It is important to be able to react quickly through automation if a failure happens. Therefore, released does not mean done; instead, there is the need to review and monitor.

传统方法是在发布之前执行尽可能多的测试，直到有信心达到产品健壮且不会失败。 对于DevOps组织而言，正确的测试也至关重要，但他们会特别注意，及时在生产中发生故障，也只会是小故障，并且可以快速修复的故障。 这可以通过仅部署较小的更改，并逐步发布，同时密切监视应用程序是否健康运行来实现。 如果发生故障，非常重要的是能够通过自动化快速做出反应。 因此，发布并不意味着完成; 相反，仍然需要复查和监控。

Monitoring can actually also rely on automated checks. Semantic monitoring uses actual tests to continuously evaluate the application in production. This approach can be useful to test the interaction of microservices at run-time, for example. There should be no clear distinction between monitoring and testing. Production tests interact with a live production system, as opposed to a controlled, predictable environment, so they are essential to ensuring a reliable production service.

监控实际上也可以依赖于自动检查。 语义监控使用实际测试来持续评估生产中的应用程序。 例如，该方法可用于在运行时测试微服务的交互。 监控和测试之间应该没有明显的区别。 生产测试和实时生产系统相互作用，而不是受控制的，可预测的环境，因此它们对于确保可靠的生产服务至关重要。

Fault tolerance testing(容错测试)
Chaos Monkey is a tool run in Amazon Web Services that intentionally creates production failures and determines if the system under test is still able to operate under those failure conditions. Failure will happen, so why not test possible failures in production yourself ahead of time? By failing quickly and inexpensively in a controlled experiment, the developers and system engineers are guided toward creating a system that is more robust and can cope with failures.

Chaos Monkey是一个在Amazon Web Services中运行的工具，它有意创建生产中的故障并确定被测系统是否仍能在这些故障条件下运行。 故障将会发生，那么为什么不提前测试生产中可能的故障呢？ 通过在受控实验中快速且低成本地模拟故障，开发人员和系统工程师将被引导创建一个更强大且能够应对故障的系统。

Shaping the Future of Automation(塑造自动化的未来)

These examples cannot give a comprehensive overview of automated testing in DevOps, but they show that more than just simple regression checking can be automated. Automation can also help analyze the undefined, unknown, and unexpected by relying on real production traffic and unpredictable test input. With shorter implementation and release cycles, testing and production come closer together. The testing community should not ignore such developments, but instead should see the opportunities to play an important role in helping drive test automation forward.

这些示例无法全面概述DevOps中的自动化测试，但它们表明，可以自动执行的不止简单的回归检查。 自动化还可以依靠实际生产流量和不可预测的测试输入，来帮助分析未定义，未知和意外的情况。 通过更短的实现和发布周期，测试和生产更加紧密。 测试社区不应忽视这些发展，而是应该看到在帮助推动测试自动化方面发挥重要作用的机会。

关于作者

Stefan Friese，14 years of experience as test manager, performance tester, software engineer, and quality engineer. In my current role, I lead a team that builds Continuous Integration & Delivery test capabilities for web services. You can find me on twitter as @y3key.

Stefan Friese，14年的工作经验，包括测试经理，性能测试员，软件工程师和质量工程师。 在我目前的职位上，我领导的团队为Web服务构建持续集成和交付测试功能。 你可以在Twitter上找到我@ y3key。

自己的一点心得

关于作者的一点观点，自己在实际项目中的应用情况：

• 在生产环境中做语义监控。监控case来自自动化case;
• 在生产中做测试。实际是用了生产中的数据来做最后一波的测试
• 容错测试。这是新服务、服务修改会进行的操作。通常单个服务内、服务间的容错都需要进行
• 关于A/B测试，Canary测试，根据业务需要，上线风险考虑，进行不同程度控制