Automated testing and continuous testing are two buzzwords that get thrown around together more often than not. While they complement each other, they are in no way synonymous. Both automated testing and continuous testing have a monumental impact on DevOps and Continuous Delivery, each in its unique way. But, they are not a prerequisite nor post requisite of DevOps, Continuous Delivery, or even each other.
Bottom line, you can choose to do either automated testing or continuous testing on their own regardless of if you work in DevOps and CI/CD. Just like you can opt to drink milk despite also eating vegetables, irrespective of the fact that you run 5 miles a day. Both help your body individually and contribute together to keep everything working smoothly. Same goes for automated testing and continuous testing.
However, like any good physician, I’m going to argue for both as together they have a larger impact on the overall health and performance of your application.
What’s the Difference? Automated Testing vs. Continuous Testing
“Test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes.”
Automated testing or test automation is a process where you use software to automate a set of tasks. The reasons for automating processes in testing are the same for automating an assembly line. In a series of identical, repetitive tasks a machine can do more, faster, with fewer mistakes. It’s about achieving volume and velocity across a variety of different tasks.
Sidenote: I’ve seen some instances where test automation was used as an interchangeable term for continuous testing. I’m not going to stand on the distinction simply because it’s inherently confusing. When I say automated testing, in any shape or form, I mean automated testing.
“Continuous Testing goes beyond automation and encompasses all practices — including tooling and cultural change — that help mitigate risks before progressing to subsequent Software Development Lifecycle stages.”
Continuous testing is methodology focused on achieving continuous quality/ improvement. To that end, it can employ any number, of practices and/ or tools. Continuous testing is about veracity, just because you have massive amounts of data doesn’t mean that it is clean, consolidated, and consistent. Which is what you need in order to take actions to mitigate risk and improve quality.
Together They are More Than the Sum of Their Parts
As mentioned earlier, automated testing and continuous testing can be implemented separate from each other and still provide value. You don’t have to be using test automation to validate continuous testing and vice versa. Big or small (test) data, mitigating risk and improving quality should always be strived for. But automated testing exacerbates the need for continuous testing exponentially.
None or Minimal Automated Tests
One of the things we hear here, and there is “I don’t have a lot of automated tests, so I think that it’s too early for me to start investing in continuous testing.” I find this statement to be problematic because I don’t believe that you can improve and optimize what you can’t see. If you’re thinking at this point that test status, duration, etc., count as metrics, then I’m sorry to burst your bubble. As far as quality goes, those metrics are vanity metrics. Can you answer the following questions with the data you have today:
- Am I writing tests for my most active and uncovered code?
- Are my tests testing what they are supposed to test?
- Are my tests overlapping unnecessarily?
- Are all code changes tested?
Even when dealing with a small number of tests you will need to manage veracity. Theoretically, it should be easier to analyze and manage small data sets intuitively. In reality, you’ll most likely end up operating in the blind guided by vanity metrics and incur technical debt that you’ll need to deal with down the line.
New Quality Metrics That QA And R&D Team Leaders Must Know About
The Motherload of Automated Tests
Organizations that are knee deep in automated testing are on the Continuous Delivery bandwagon. Here rapid incremental releases, sophisticated software architecture, and a proliferation of testing frameworks and tools are the norm. In other words, volume, velocity, and variety are exacerbating veracity exponentially. This is where continuous testing proves its worth. With 4 V’s in play, intuitively or manually mitigating risk and improving quality, consistently is an exercise in futility. Without continuous testing, it is impossible to make data-driven decisions regarding test development, optimization, and refactoring. Meaning that your tests are neither as effective nor efficient as they could be.
In Closing
Automated testing is a given at this point, as is DevOps and Continuous Delivery. If you’re not there yet then you’ll be on your way soon. Continuous testing is a bit newer to the field but is already earmarked as the final step to tying together Continuous Integration and Continuous Deployment in order to achieve Continuous Delivery. The question of when to begin your investment in continuous testing is a question of proactive vs reactive quality.
In our opinion, quality needs to be approached proactively. Applications today have much more flexibility built in in order to keep up with business requirements. But that same flexibility holds the potential to bring the house crashing down on us. Shift left testing and test automation are no longer enough to proactively achieve for quality. Both and much more, must and will be encompassed into continuous testing in order to achieve and maintain high quality regardless of volume, velocity, variety and veracity.