Regression Test Code Coverage

Today there is only one way to measure code coverage, and that’s only in unit tests. That said, in QA, functional coverage is tested with two methodologies. The first methodology measures how many of the total tests have run, passed and failed. It would be more correct to call this test execution status than functional coverage. The second measures the percent of the feature set tested, hence functional coverage. Neither method for functional coverage measures functional code coverage. Unit tests are granular and cannot take the place of functional tests, including regression, API, and end-to-end test.

Today’s applications are changing at such a rapid pace that it is practically impossible to track what changes affect what features. We no longer receive complete features that we then set out to test. We receive multiple builds each with incremental code changes. This reality of incremental code changes, especially in functional, API and regression testing in CI/CD pipelines needs Functional Test Code Coverage that can keep pace with it.

Everyone knows that code coverage is a critical indicator of quality. But today, there is no way to extract a finite coverage metric for all tests (including, unit, API, security, etc.) executed per build. This is where SeaLights steps in. We measure your exact code coverage calculated across all test tools and environments.

Measure Regression Test Code Coverage

Regression Test Code Coverage

Today there is only one way to measure code coverage, and that’s only in unit tests. That said, in QA, functional coverage is tested with two methodologies. The first methodology measures how many of the total tests have run, passed and failed. It would be more correct to call this test execution status than functional coverage. The second measures the percent of the feature set tested, hence functional coverage. Neither method for functional coverage measures functional code coverage. Unit tests are granular and cannot take the place of functional tests, including regression, API, and end-to-end test.

Today’s applications are changing at such a rapid pace that it is practically impossible to track what changes affect what features. We no longer receive complete features that we then set out to test. We receive multiple builds each with incremental code changes. This reality of incremental code changes, especially in functional, API and regression testing in CI/CD pipelines needs Functional Test Code Coverage that can keep pace with it.

Everyone knows that code coverage is a critical indicator of quality. But today, there is no way to extract a finite coverage metric for all tests (including, unit, API, security, etc.) executed per build. This is where SeaLights steps in. We measure your exact code coverage calculated across all test tools and environments.

Measure Regression Test Code Coverage

Quality Holes

Currently, the only way to validate quality is by the pass and fail results of our tests. While it is important that pre-existing tests pass they do nothing to ensure that untested code changes do not reach production. Quality Holes (Untested code changes) are one of the biggest challenges that companies who have adopted incremental code changes and agile CI/CD have to contend with.

Quality Holes are high-risk code changes because they are usually the result of bug fixes or new features. Meaning that they will affect trafficked features and if shipped without testing, have a high probability of resulting in usability or functionality issues. Or even worse a dead-on-delivery feature.

SeaLights identifies and tracks Quality Holes across both unit and functional test environments, providing both individual and aggregate environment analysis. Enabling you to redirect your testing efforts quickly without relying on manual reporting, release notes and commit notes.

Uncover Quality Holes

Quality Holes

Currently, the only way to validate quality is by the pass and fail results of our tests. While it is important that pre-existing tests pass they do nothing to ensure that untested code changes do not reach production. Quality Holes (Untested code changes) are one of the biggest challenges that companies who have adopted incremental code changes and agile CI/CD have to contend with.

Quality Holes are high-risk code changes because they are usually the result of bug fixes or new features. Meaning that they will affect trafficked features and if shipped without testing, have a high probability of resulting in usability or functionality issues. Or even worse a dead-on-delivery feature.

SeaLights identifies and tracks Quality Holes across both unit and functional test environments, providing both individual and aggregate environment analysis. Enabling you to redirect your testing efforts quickly without relying on manual reporting, release notes and commit notes.

Uncover Quality Holes

Build Quality Dashboard

The Build Quality Dashboard brings together QA, development, and DevOps and provides a centralized view of critical quality metrics per build. On a single dashboard you can see the complete status of each build: where was it tested, how many tests executed, in which environments/labs, how many tests passed/failed and code coverage. We not only collect all quality metrics such as, test status, coverage, and code changes from all your testing tools and environments, but perform a number of computations on the data that turn statistical noise into aggregated quality metrics and insights of your product. Creating a one-stop-shop for app, microservice, build, test and environment statuses.

With it, teams can drill down into their quality metrics from all environments and frameworks in one centralized quality dashboard always in comparison to the last build. There is also a Big Screen mode, a shared team view for quality metrics and build status. It is a straightforward way for teams to understand where they stand at any given moment and act fast to improve quality.

Launch Build Quality Dashboard

Build Quality Dashboard

The Build Quality Dashboard brings together QA, development, and DevOps and provides a centralized view of critical quality metrics per build. On a single dashboard you can see the complete status of each build: where was it tested, how many tests executed, in which environments/labs, how many tests passed/failed and code coverage. We not only collect all quality metrics such as, test status, coverage, and code changes from all your testing tools and environments, but perform a number of computations on the data that turn statistical noise into aggregated quality metrics and insights of your product. Creating a one-stop-shop for app, microservice, build, test and environment statuses.

With it, teams can drill down into their quality metrics from all environments and frameworks in one centralized quality dashboard always in comparison to the last build. There is also a Big Screen mode, a shared team view for quality metrics and build status. It is a straightforward way for teams to understand where they stand at any given moment and act fast to improve quality.

Launch Build Quality Dashboard

Quality Analytics

Incremental changes make it difficult to track and identify long-term trends that are indicative of the overall product quality. SeaLights makes it simple to view both historic and weekly trends and catch critical issues before they hit rock bottom.

SeaLights Quality Analytics are invaluable in identifying quality targets and building roadmaps. With it identifying areas of high frequency change but low coverage is easy and all that’s left is to set coverage goals.

Access Quality Analytics

Quality Analytics

Incremental changes make it difficult to track and identify long-term trends that are indicative of the overall product quality. SeaLights makes it simple to view both historic and weekly trends and catch critical issues before they hit rock bottom.

SeaLights Quality Analytics are invaluable in identifying quality targets and building roadmaps. With it identifying areas of high frequency change but low coverage is easy and all that’s left is to set coverage goals.

Access Quality Analytics

How Does SeaLights Work?

In order to identify and extract quality markers SeaLights runs two components in your pre-production environments: our Build Scanner and Test Listener.

The initial SeaLights onboarding takes between 30-60 minutes for the first application depending on the complexity of the configuration. Additional applications take only a few minutes to launch. As soon as you finish the Sealights configuration, continue testing through your frameworks as usual and all test and build quality metrics will be visible through the build quality dashboard.

Schedule a Live Demo