Background

Software engineers classify ‘smoke’ tests as a quick health check on one or more software components, typically after a deployment, but also after a significant configuration change. This avoids wasted effort in moving a failed build or configuration change down the pipeline. After the successful smoke test, the engineers execute a more extensive integration test, feature or regression test.

But creating a good smoke test has always been a challenge—the ‘quick’ and the ‘check’ are at odds. Since most of us want a smoke test to complete in minutes, not hours, we typically trade off software coverage for execution speed. And if a smoke test is at the system or application level, we are usually talking GUI tests, i.e. slow relative to unit or API tests. We have to be very choosy. In our experience, system smoke test suites typically execute test cases numbering in the single digits, certainly not in the hundreds.

Introducing Boost

At Webomates, we didn’t like that tradeoff any more than you do, so we broke out of that box. We built a SaaS that executes hundreds (and soon thousands) of GUI and API tests in under 15 minutes. This is an add-on feature to the CQ (Continuous Quality) service we call Boost. This article describes how we built this capability on Amazon Web Services (AWS)

Boost-enabled cycles can be launched by a human from the CQ Portal or by a Continuous Integration/Continuous Deployment platform, e.g. Jenkins or Bamboo, via the Webomates CI/CD API. You can monitor the progress in real-time; results show live in the CQ Portal, or the CI/CD requester can poll periodically. Since the smoke mode is totally automated, we won’t have our engineers validate defects. You will analyze the results at the end and decide to proceed with your promotion or go back and fix issues, and try again.

The key differentiator between pre-Boost and Boost, is the management of the test nodes, or in the aggregate, the test fleet. The test fleet before Boost was non-elastic or static—manually provisioned to typically two nodes per customer product. The test fleet was almost always idle. Now with Boost, the fleet is elastic or dynamically created and destroyed as needed, on a per-test cycle basis.

Fundamentally Boost was a scaling challenge. If we want something to go faster, we need more Compute power, i.e. scale up (more nodes of the same size) or out (same number of nodes but each bigger). We used cloud providers compute elasticity and a test domain-specific architecture to solve this challenge. Read This Blog: Smoke Testing