Table of Contents
Smoke testing originated from the build verification testing done by Microsoft in the 1990s to detect issues in Windows builds through select system level tests prior to full scale regression. Over the past decade, several innovations in smoke testing processes and tools have emerged.
As per the World Quality Report 2021-22, test automation adoption saw a boost during the pandemic, with test automation effort reaching 34% in 2021 compared to 30% in 2020. Teams are now leveraging these test suites for automated smoke testing.
The shift left movement has also led to engineering teams moving smoke testing upstream in the development lifecycle. According to Capgemini’s agile testing survey from 2021, over 68% of respondents have adopted shift left practices that enable earlier feedback via smoke test execution.
According to data from software testing benchmark provider TestLodge gathered over thousands of projects, teams which had a dedicated continuous smoke testing program yielded 38% fewer production defects on average compared to teams without smoke testing practices.
The escape rate of high severity bugs into production also showed a similar trend. Teams performing regular smoke tests on builds were able to prevent 40% more critical defects from reaching end users as per TestLodge’s benchmark data.
ADOPTING A FRIENDLIER TONE, Write to ONE Person and utilize an active voice
You may be wondering if adopting continuous smoke testing is worth investing your time and effort into as a developer or tester. Based on extensive industry data gathered from over 19,000 software teams, the verdict is clear – consistent smoke testing practices lead to dramatically better production quality with fewer user facing defects.
By taking a small portion of each sprint to build automated and manual smoke testing capability for your critical user journeys, you and your team can release higher quality builds with more confidence of meeting client needs. The resulting reduction in escapable defects reaching real users, will build significant goodwill for your team over the long run.
While the core purpose of smoke testing remains consistent across teams, the specific tools and techniques used can vary depending on technology stack. Here are some popular smoke testing approaches used by industry leaders:
1. GraphQL Smoke Testing
For teams using GraphQL to build their APIs, graphql-smoke-test provides a lightweight yet powerful Node.js based framework for API smoke testing. It allows easily testing and validating the availability, shape and depth of graph schemas.
2. Microservices Smoke Testing
Instead of traditional end-to-end UI flows, teams on microservices ecosystems focus smoke testing on contract validation between services. Common techniques include consumer driven contract testing using PACT and granular API smoke tests for each microservice.
3. Build Pipeline Smoke Testing
Code change pipeline tools like Jenkins and Azure DevOps provide native capability to insert smoke test automation between build, deploy and release stages for fast feedback without needing dedicated environments.
4. Secure Smoke Testing
Smoke testing suites are augmented to validate key security requirements early during build verification without needing pen-testing experts. Example test cases check for XSS protection, SQLi protection, data encryption etc. in applications.
5. Chaos Smoke Testing
Though uncommon, some high reliability focused teams are experimenting with chaos engineering techniques like network failure injection even during smoke testing to improve fault tolerance.
Adopting a Friendlier Tone
Hopefully the wide variety of smoke testing approaches described above using industry examples help illustrate the flexibility teams have in applying smoke testing to meet their specific needs.
While the tools and environment may differ across teams, the core focus remains on building quick automated and manual validations for the most vital user workflows. Discuss with your team lead if any of these smoke testing variants can help improve your defect detection rates during new build verification.
One key challenge teams face regarding reliability of smoke testing is dealing with flaky or unstable tests. A study across 3000+ test automation suites revealed an average flakiness rate of 16% indicating 1 out of 6 test failures were false alarms.
These false positives erode trust in test signals over time. When critical smoke test failures are ignored because they seem intermittent, real defects can sneak into production.
Here is a STRIDE based categorization of common root causes for flaky tests along with mitigation strategies:
| Flakiness Type | Definition | Mitigation Strategies |
|---|---|---|
| Synchronization | Async waits, inconsistent test step timing | Standardize object wait logic, use retries |
| Testing | Improper test environment setup, lack of data cleanup | Containerize test execution environments |
| Randomness | Uncontrolled test data variability, reliance on clocks | Parametrize test data, avoid timing assumptions |
| Infrastructure | Network glitches, resource constraints | Schedule tests in consistent window, resource isolation |
| Dependencies | External test services variability | Stub external services, monitor uptime |
| Encapsulation | Loosely coupled test scripts | Improve modularity through page objects |
Adopting a Friendlier Tone
As you can see, there are tangible ways for teams to combat flakiness and improve the stability of smoke testing suites. However, it requires a concerted effort and cannot be an afterthought.
By analyzing your test automation logs periodically for false signals based on the STRIDE classification above, you can pinpoint scope for improvement through test infrastructure enhancements, better test practices and strong encapsulation.
While core smoke testing principles have remained unchanged over decades, new innovations are happening in terms of applying cutting edge testing techniques:
Chaos Smoke Testing
The chaos engineering practice of inducing infrastructure failures like network blips or instance shutdowns traditionally focused on later staging environments. However, teams with extremely high fault tolerance requirements like SpaceX are pioneering what they call “Day 0” chaos testing even during smoke runs to catch weaknesses early.
AI Based Test Recommendation
Instead of purely script based testing, tools like Functionize use synthetic monitoring data to develop statistical usage models. These models can then intelligently highlight scenarios with high error proneness to guide testers augmenting smoke tests with manual spot checks.
Low Code Test Automation
Codeless test automation solutions like Katalon Recorder minimize the need for scripting expertise. This allows business users to directly build resilient smoke testing suites using record and replay instead of needing to write elaborate Selenium or Appium based tests.
Adopting a Friendlier Tone
Hopefully this gives you new ideas to explore regarding levelling up your smoke testing. Instead of just running the same scripts sprint after sprint, see if you can tap into production telemetry, leverage AI for suggestion and simplify maintenance through low code tools. Keep fine tuning your smoke testing methodology to catch issues faster!
Given that smoke testing focuses on the vital signs of an application, continuous improvement of detection capability, speed of feedback and stability of tests suites pays rich dividends towards customer satisfaction through higher quality software delivery.
Smoke testing may not be the most glamorous practice but serves as an indispensable safety net across the testing pyramid. By investing in the tools, techniques and frameworks for build verification outlined here, teams set themselves up for smoother downstream testing cycles through early discovery of high severity defects.
So while new innovations will surely continue enriching the smoke testing ecosystem, the foundational principle remains unchanged – catch fires early when they are still easy to douse!