Even after following coding standards and methodologies, humans make mistakes, leading to errors and defects in a product. Thorough testing is what makes a product high quality. Unit testing is a good start, but it is sometimes skewed and is difficult to measure or improve.
Test measurement allows better product quality
So, how do you measure test effectiveness? Counting number of lines covered by tests or taking a functional approach and see if all the features are covered.
Optimizing Test Automation | Download White Paper >>
In this post, we’ll learn about two major test measurement methodologies and contrast them to understand which method is better suited to which product/software development method. The two popular methodologies are code coverage and test coverage.
Code Coverage
To begin with, let’s learn about code coverage. This metric aims to measure number of lines covered by the test cases. It reports total number of lines in the code and number of lines executed by tests. Think of it as the degree to which the source code of a program is executed when a test suite runs. The intent is, the higher the code coverage, the lower the chance of having undetected software bugs.
Subtypes
Coverage is further split into many subtypes – function coverage, branch coverage, condition coverage, loop coverage, statement coverage and parameter value coverage.
Let me define the subtypes for the sake of clarity.
- If tests cover all the function calls in code, function coverage is said to be 100%.
- Similarly, if all branches in the code, i.e. all the if – else conditions have been tested with every possible input, then branch coverage is said to be 100%.
- If all loop statements in the code are executed then loop coverage and statement coverage would be 100% respectively.
In ideal case, for condition coverage to be 100% all the conditions with all possible combination of condition parameters are tested. Parameter value coverage is checked by passing all common values to the parameters in function call.
Tools
Coverage testing tools are available today for all the major languages. Note, that most of them are relevant only for unit testing and not for the overall tests performed (Unit, integration, system, manual and more). Let’s take a look at few of them to understand the code coverage tools ecosystem.
Cobertura is a Java code coverage reporting tool. It is based on jcoverage and works on any platform with Java 5 or higher. You can use Cobertura with ant, maven or command line access. While instrumenting classes, Cobertura generates cobertura.ser file containing basic information about each class. This file is used when instrumenting, running, and generating reports. It also uses ASM when instrumenting. On running tests, it generates report containing all packages, each class within each package and corresponding line coverage, branch coverage and complexity.
For Python, a popular tool for code coverage measurement is Coverage.py. It works in three phases –
- execution, where it runs the code and monitors it to see what lines were executed.
- analysis, where it examines the code to determine what lines could have run.
- reporting is where it combines the results of execution and analysis to produce a coverage number and an indication of missing execution.
A measurement tool widely used for Go is gocov. This tool offers four major commands – test, convert, report and annotate. Using these commands, you can generate textual reports with annotations giving coverage information of each function in each go file. A GUI wrapper is also available for gocov named as GoCovGUI. It displays entire report generated by the tool in GUI which is easy to understand. It also has wrappers to generate reports in HTML and XML format.
Test Coverage
Test Coverage aims to measure of the effectiveness of testing in a qualitative manner. It determines whether the test cases are covering entire functional requirements. You can think of it as a kind of black box testing, where test cases are not written based on code but based on user requirements or expected functionality.
Subtypes
The common mechanisms used for test coverage measurement are unit testing, functional testing, performance testing, integration or system testing and acceptance testing.
Unit tests are written at a granular level to check if a function/method performs as expected. In functional testing, each functionality mentioned in the requirement document is tested. Performance testing is generally a way to stress test the code for its responsiveness and stability at different workloads. Integration testing or system testing is done to test if completely integrated product works in expected manner. Acceptance testing is generally done at the end of the development cycle. For acceptance testing, the product is handed over to the stakeholders and tested by them to determine whether the product is in acceptable state.
Tools
These test coverage tools aim to measure the testing effort in a qualitative manner. The idea here is to see if all requirements and functionalities are logically covered via testing. So, this is not straight-forward to measure as compared to code coverage. Methodologies like TDD (Test Driven Development) are also helpful to analyse test coverage and bring in the discipline to add tests from get go. To measure the impact of such tests, you will need to manually list out requirements and then analyse test case to understand which of those are covered.
Still, you need frameworks to write tests. Let’s see some testing frameworks available for various programming languages.
JUnit, the well known Java testing framework serves as a foundation to launch tests on the Java Virtual Machine. With support for assertions, assumptions, default test methods and other mechanisms to test functionality JUnit is a very good choice for testing Java based software.
For Python, framework called PyUnit is available. As the name indicates it is the JUnit equivalent for Python. But there are several other testing frameworks available to help you as well.
Go also offers a built in testing package for programs letting you easily write, organize and run tests. It also offers various profiling tools to understand software behaviour under varying loads.
Testing: The Next Bottleneck in Continuous Delivery | Download >>
Code Coverage vs Test Coverage
So, now we know that code coverage is a measure of how much code is executed during testing, while test coverage is a measure of how much of the feature set is covered with tests. An obvious question that comes to mind now, is what should you go for?
You’ll have to factor in several parameters and may still end up using bits of both. It is difficult to sum up, and say one is better. So, I will end this post with a list of pros and cons of each methodology, that will help you make a good decision.
Code Coverage Pros and Cons
Pros:
- Straightforward process, add tests and see if coverage improves.
- Easy to setup and measure coverage, once you find the right coverage tool.
- Helps capture bugs in program flow.
Cons:
- Most Code coverage tools are relevant for Unit test only. It is essential to track all test types coverage (unit, API, System and even manual).
- Good code coverage tools are difficult to find.
- Even if you find a good code coverage tool for the language your project is written in, it is an uphill task to set up and make sure it works well with your project.
Test Coverage Pros and Cons
Pros:
- As a black-box testing methodology this approach is aligned to the requirement specifications and has minimal interaction with code itself. Isolating tests from code allows skew-free tests.
- This tests the software features, making it a good candidate for other criteria like acceptance tests.
Cons:
- Most of the tasks in such testing methodology are manual. Since there are no tools to automate, it takes lot of effort to analyze the requirements and create test cases.
- There is no concrete way to measure such coverage, you can of course count features and then measure against number of tests, but that still leaves space for judgement errors. On the other hand code coverage is much more concretely measured.
Also keep in mind:
- Function and feature are different. Covering a single feature will cover several functions, while the required effort may be similar. Optimize for this.
- Negative tests may not improve coverage to a big extent, but help test the functionality in a big way. Ignoring such tests in not a good idea.
- Sufficiency of testing is much more complicated attribute than coverage can answer. So, look for signs from software users or if possible, use it yourself – that will help you get an idea if tests are really sufficient.
Beyond Just Coverage: Towards Holistic Test Scope
Coverage is an important but limited metric. Verifying testing breadth in the traditional sense is a useful concept, but one that is difficult to measure operationally. Most teams do not have a clear measure of the actual comprehensiveness of their testing efforts.
What would be truly useful is to measure actual test scope, going beyond unit tests to include integration tests, acceptance tests, and manual tests as well. Traditionally there has been no easy way to see a unified test coverage metric across all types of tests and all test systems in one place.
SeaLights is a continuous testing platform that makes this possible. It integrates with all your testing tools, and allows you to measure holistic test coverage across all types of tests. This provides an answer to a crucial question: how extensively is your product tested, and how much risk exists in terms of untested code changes that are shipping with your next release. Check out our free trial.