Test Metrics

Don’t Let These 5 Problems Stop Your Regression Testing Strategy

Regression testing involves testing existing software to verify that working features are not broken by new development work. Regression testing is important because it can help catch defects very early on when the cost of remediation is still very low.

A regression testing strategy is your attempt to implement regression testing and make sure it is effective and providing value to the organization.

Many things can get in the way of a regression testing strategy. Five primary challenges with regression testing:

  1. It’s not “sexy” and considered uninteresting by testers and developers
  2. It’s perceived as having little value because you are retesting the same old features again and again
  3. It’s expensive and the cost is hard to justify
  4. It takes time to complete the tests and this slows down the agile velocity
  5. Regression testing and its value is not sufficiently visible to testing teams, developers, and management

Learn about these problems and how to solve them.

Problems Stemming from Culture

1. Regression Testing is Not “Sexy”

The Problem

There’s no way around it – regression testing involves running the same tests over and over again. This can demoralize testers and over time, they might miss tests, ignore or misinterpret them.

What Can Help?

  • Automate regression testing to reduce manual work.
  • Rotate the team and avoid having one person do the same work over and over again.
  • Educate testers about the importance of regression testing and the long-term contribution to the quality
  • Make regression testing metrics very visible to team and management.
  • Shell out! Give regression testers a bonus or award for staying on track and ensuring tests run consistently.

2. Regression Testing Perceived as Having Little Value

The Problem

Regression testing does not provide immediate value. Management asks: “why are we testing the same features we built months or years ago? Why not spend the efforts on new features that generate new revenue?” Testing teams often find themselves trying to justify the time spent on regression testing – and often end up neglecting it and spending time on mission-critical activities.

What Can Help?

  • Educate the team and your managers about the importance of the early discovery of bugs and other regressions from one version to the next – code quality, dev productivity, reduced time and effort for manual testers and acceptance tests.
  • Share with the team and your managers’ scenarios where basic capabilities (like login) will stop working and what are the implications of the company’s activity and brand.
  • Share statistics about the ROI of early bug detection. For example, the statistics below from an Ixia slide deck show that the cost of fixing a bug in production is 31X higher than fixing it at the build stage.

Problems Stemming from Technology

3. High Cost

The Problem

If regression testing is manual, it represents a high ongoing manpower cost. Organizations essentially need to pay one or more tester’s salary just to run regression tests. If it’s automatic, there is an upfront cost in development, troubleshooting the regression test suite, and tooling – unless you use open source, which usually translates into higher development and setup costs.

What Can Help?

  • If you’re still running tests manually, automation is the first thing to do.
  • Use risk-based testing – prioritize tests and focus on areas of the software that are the most prone to defects or changing frequently.
  • Ask the development team what they worked on or changed in the last release, or use quality intelligence platforms. They can help you on areas of the code which are really likely to have experienced regressions.

4. Time to Completion

The Problem

Regression tests need to be run after every development iteration. If they are done manually (or even semi-manually), they can take a few days to weeks for a complex product. In a world with dev sprints averaging two weeks, can grind progress to a halt. Even if regression testing is fully automated, it can slow down build times and become a bottleneck in the agile feedback cycle if they are not optimized.

What Can Help?

  • Automate! If done well, it will dramatically cut ongoing costs of regression testing.
  • Use risk-based testing – prioritize tests and focus on areas of the software that is most prone to defects or changing frequently.
  • After you automate, if your regression tests take 4 or 14 hours to run, requiring a nightly build and killing your continuous integration cycle, optimize them! There are many tools and best practices that can cut build time from hours to minutes. Consider TIA (test impact analysis) – execute per build only the relevant tests from your regression suite, where the code was changed.

5. Visibility

The Problem

Many organizations do not clearly define goals for their regression testing and the metrics behind those goals. Testers do regression testing, but it is not clear how well it is done, and what is the impact on the stability and quality of the software.

What Can Help?

  • Define goals and metrics.
  • Make sure those metrics are tied to results that interest the business, for example, reduction of production defects.
  • Try to quantify the ROI of resolving bugs earlier – for every real bug found early by testers, attach a Dollar amount of what it would cost to fix that bug much later in the process.
  • Quality Intelligence Platforms like SeaLights, which show which parts of your software are actually covered by tests, and which tests help achieve that coverage – this can help show the role and scope of regression testing to management and provide reasoning why and where additional tests should be developed for the regression suite.

Regression Testing: the Bigger Picture

Your regression testing strategy doesn’t exist in a vacuum. There are many other tests going on – UI automation, functional tests, integration and manual tests. What’s the role of regression testing in this bigger picture? How important is regression testing in reducing risks of production defects?

SeaLights provides a clear picture of how effective regression tests are, and what would happen if we neglected or stopped them. SeaLights is a Quality Intelligence Platform for software development that helps to increase the velocity of your engineering teams by using cross-stack quality analytics throughout your code activity, tests, and releases and focus your test development and execution on what matter most