If someone sneezes in your production environment odds are you’re going to know about it, and in real-time. There is no shortage of tools like Datadog, New Relic, and AppDynamic to cover you on that end. The entire software development pipeline is made up of tools for planning, mocking, monitoring, frameworks, cloud services, deployment, etc., whose entire purpose is to increase velocity by turning time consuming activities into configurable services. But, there is one stage in our pipelines that is not fast, not visible and not robust. Testing.
If I were to tell you that if your code can run locally then you’re ready for production you’d laugh.
Code doesn’t exist in vacuum on your developers macs. But for some reason, in software testing if the tests have passed then the release is ready for production. Even though there is no visibility into testing gaps and no ability to measure how robust the testing was compared to end user executions. If you’re relying just on pass/fail then you’ll get false positive release readiness statuses like these:
- All your tests have passed but there’s new (and sometime unstable) code that hasn’t been tested at all.
- All your tests have passed but a highly trafficked code area is only covered by unit tests.
- Your team added 100 new tests but for dead code.
These false positives will turn into a spiral of declining quality and increasing hotfixes that will drag down your team’s velocity.
Top 4 Machine Learning Algorithms for Testing | | Download >>
Quality Intelligence is About Context
The impact of your tests on software quality and release readiness has a direct relationship to your team and code activities. Test metrics are meaningless until they’re put into the context of tests/code that have been developed/changed/deleted and with correlation to the end user usage. Once they’re given context and benchmarks metrics become intelligence.
| Metric | Intelligence |
|---|---|
| 13/14 integration tests passed | Regression identified in test 201 |
| Code coverage dropped by 3% | 22% of code added/changed has not been tested in any test stage |
Moving from metrics to intelligence is only part of the solution. To make the best release decisions software testing quality intelligence needs to incorporate real-time data from hundreds of thousands of tests executions, code changes, builds, historical data, and trends.
Using Software Testing Quality Intelligence to Make Better Release Decisions
With Quality Intelligence you’ll have visibility into:
- Your team’s quality efforts and progress.
- Identify code areas with untested code and quality drops and block them from production.
- Focus your scrum teams on the right code areas to invest in.
In order to achieve this without wasting time and effort to collect and analyze data from hundreds of sources, runs, and builds we developed SeaLights Quality Trend Intelligence. With it, R&D and QA managers get real-time visibility into the quality status of their builds.
Quality Trend Intelligence deeply impacts how determine release readiness and plan our sprints. I’ll be happy to share with you what it can do for your team’s velocity and
efficiency >> [email protected].