The SeaLights test metrics guide
for better and faster CI/CD
JAVA Code Coverage
Test measurement helps in identifying and minimizing bugs and design defects in your code. While there are various methodologies to measure test effectiveness, code coverage is one of the most popular. Not only does it help in analyzing code from overall feature completeness perspective but also serves as a good measure of software quality.
In simple terms, code coverage analyses each line in the code and confirms if it will be executed by a test case. There are several tools that do this and in this post, we will explore code coverage tools for Java.
Cobertura is based on jcoverage and works on platforms running Java 5 or higher. It calculates the percentage of code accessed by tests. Cobertura can be executed via ant, maven or command line. To run Cobertura on a code base, download the latest release from its website. Cobertura is also available via package managers, for example on Ubuntu (v15 and above) use the command:
sudo apt-get update
sudo apt-get install cobertura
Cobertura is also available as Maven, Eclipse, Jenkins and IntelliJ Plugin. While it supports several platforms, Cobertura follows the similar instrumenting and reporting mechanism.
First, you need to generate cobertura.ser file containing basic information about each class. This file is used when instrumenting, running, and generating reports.
To generate the instrumentation file using Cobertura command line interface, execute:
cobertura-instrument.bat [--basedir dir] [--datafile file]
[–auxClasspath classPath] [–destination dir] classes […]
Here, classes can be either be specified individually or via a directory tree containing multiple classes.
Next step is to run the tests, but before running tests make few changes to your classpath. Add cobertura.jar file path, instrumented classes, and uninstrumented classes directory. Note that the sequence must be followed, i.e. add instrumented class path before the uninstrumented class path. Also, it is not mandatory to have JUnit tests.
Execute your test file as below:
java -cp /path/to/cobertura.jar;Directory>;; -Dnet.sourceforge.cobertura.datafile=file in Project Directory>
Finally, get the report file for the test using:
cobertura-report.bat [--datafile file] [--destination dir] [--format (html|xml)] [--encoding encoding] source code directory [...] [--basedir dir file underneath basedir ...]
The report file is generated in either HTML or XML based on the input you provide. It contains all packages, each class within each package and corresponding line coverage, branch coverage and complexity. A typical report contains following fields:
- Package: Package names within the destination directory
- # Classes: Number of classes covered
- Line Coverage: Number of lines executed in the code (This excludes lines having curly braces or newlines, includes only useful lines in the code)
- Branch Coverage: test measurement of all the decision points within the code
- Complexity: It returns McCabe cyclomatic code complexity for your code
JaCoCo (Java Code Coverage) works on a platform with JRE version 1.5 and above. It is based on class files analysis. Class files getting measured must be compiled with debug information and they must have line number attributes. JaCoCo instruments the classes under test for collecting execution data. This adds two members to the class – A private static field $jacocoData and a private static method $jacocoInit(). Both members are marked as synthetic hence change your code to ignore synthetic members.
JaCoCo provides out of box tools for testing and can also be integrated with other tools. It provides tools like Ant Tasks, Maven Plugin, Java Agent and command line interface. You can integrate it with tools like Java Doc, Maven repository and XML Report DTD.
To get started, download JaCoCo from Github.
For command line interface, JaCoCo provides jacococli.jar file. Use this file to instrument the Java classes:
java -jar jacococli.jar instrument [ ...] --dest
java -jar jacococli.jar merge [ ...] --destfile [--help] [--quiet]
This creates single execution file which is used while generating reports. Generate the reports in any format using this execution file and Java class files:
java -jar jacococli.jar report [ ...] --classfiles [--csv ] [--encoding ] [--help] [--html
EMMA is an open-source toolkit for measuring and reporting coverage. It can instrument classes before they are loaded or on the fly. It is based on Java bytecode instrumentation and hence is quite fast, compared to other option. It does not require access to the source code and can instrument individual .class files or entire .jar
Using EMMA is quite easy, download the latest version of EMMA from its website. Download the zip file suited for your use. Unzip the file and start using it. To use EMMA on-the-fly from command-line for getting report file execute
java -cp emma.jar emmarun -jar
This will give you coverage data for your class, method, block, and line for all classes. Along with percentage report, it also has percentage ratio which helps in identifying areas that need more testing. As even if the coverage percentage for a package with 100 classes is same as that of a package with 10 classes, a package with 100 classes needs more attention. To generate the reports as a file, execute
java -cp emma.jar emmarun -r html -jar
This will generate report file in html format which can be viewed from your browser. To generate report file in xml format, replace html with xml in above command.
For offline instrumentation, take the copy of your jar or class file and instrument the original one by
java -cp emma.jar emma instr -m overwrite -cp
This updates the coverage.em file with metadata information and your jar file now has instrumented classes available for coverage. Add emma.jar to the application execution classpath. When you do this, coverage.ec file gets updated. Now combine metadata with runtime coverage data to produce coverage report,
java -cp emma.jar emma report -r html -in coverage.em,coverage.ec
This will generate report file in html format with all the details mentioned earlier.
In this article, we learned about Java code coverage. We started with Cobertura, one of the most popular tools, and saw how to set it up and generate coverage reports. We learned about various Cobertura plugins like Maven, Eclipse etc. as well. Next, we got into JaCoCo, the open source testing platform. We learned how to install and deploy JaCoCo to an instrument and report code coverage.
Finally, we learned about Emma. Since Emma instruments the code at the bytecode level, it is relatively faster to report the coverage and it can instrument classes before they are loaded or even on the fly.