What is Test Gap Analysis?
Test Gap analysis provides software engineering managers and QA managers a detailed overview of the test gaps that remain after the test phase. In other words, the analysis identifies changed code that was not tested before a release.
By releasing these high-risk areas you might experience frequent bugs in production and time-consuming hotfixes.
Not all test gaps are equally important which is why we divide them into three groups, each group representing a different level of risk.
- High-Risk: An untested method that was recently modified and used in production
- Medium Risk: An untested method that was recently modified and not used in the production
- Low Risk: An untested method that wasn’t modified recently and used in production
This division is valuable as it presents test gaps to managers and team leaders in a way that allows them to decide which actions to take in their sprint planning.
Background
There are many ways to increase sprint velocity. The common denominator is that to move faster, we must have some level of a technical debt.
Developers increasingly spend time on test automation and maintenance. This is a critical activity, but one which can slow down a team’s velocity. An important way to save time and increase velocity is to eliminate unnecessary tests.
Features that are rarely or never used in the current production version do not deserve additional tests. Even if you discover bugs, they will affect almost none of the software’s active users.
Moreover, code that has been stable for several sprints, or even weeks or months, and hasn’t broken, is unlikely to break now. Adding tests for code that isn’t actively worked on is probably a waste as end users already tested this code for you.
Writing and running integration tests: The Bane of Software Teams
To do their job right and help development teams avoid and reduce technical debt, they have to write, run, and maintain lots of integration tests for each software component. Furthermore, these integration tests must be executed before every release – making unit testing an arduous, time-consuming, and expensive process.
These challenges, among others, often create test gaps – areas of code that were either not tested in a certain time period or in a certain build. By their nature, test gaps are silos that result in high-risk technical debt and usually result from areas where code was recently changed or executed in production, but have not been tested in any regression cycles.
Maintain a Low-Risk Technical Debt as Part of Your Sprint Planning
Although the concept of technical debt is relatively new, it is founded on an old problem that has challenged software development teams for years.
Fundamentally, technical debt refers to the trade-off that occurs when development teams leverage the short-term benefits of increasing velocity to rapidly deliver software to clients, against the long-term value of developing software solutions that can easily be modified, repaired, and maintained.
Effective management of technical debt calls for an accurate analysis of test gaps in a software solution’s codebase to detect areas of high-risk so that adequate time and effort can be focused on them.
Using information and insights obtained from test gap analytics, development teams can adequately plan sprints and develop integration tests for such high-risk areas, thus increasing efficiency while reducing technical debt.
Such visibility into high-risk areas facilitates the exposure of poor design and coding practices, enabling agile teams to clean up the code, data, and all outdated assumptions, ensuring that codebase quality remains at optimal levels. This results in the effective management of technical debt, ensuring that it is low-risk and kept at manageable levels.
With insights from SeaLights’ Test Gap Analytics, software teams can plan their sprint based on the analysis of data gathered from multiple sources across the software development pipeline, make the right decisions of where their next tests should be developed, and put to rest the days of writing unneeded tests. These analytics allow developers, team leaders, and managers to maintain a low-risk technical debt to their advantage.