JaCoCo: Code Coverage Analysis for Java

JaCoCo is a tool for code coverage analysis for Java. What this means is that it instruments your code, watches it as it runs, and tracks and computes coverage, or what statements the run actually executed. Most usefully, it can also aggregate coverage data from multiple runs and tell you which statements were executed, and which were not, over all of the runs together.

For white box testing, tools like this are imperative to use, so that you can tell what code your tests covered and what code they did not. Then you can add tests to cover the missing code.

This tutorial will use JaCoCo as a command line tool, but if you go to the website, you can see that there is Eclipse integrate with the EclEmma version of the tool.

Downloading JaCoCo

The main page referenced above has a list of JaCoCo versions. You just need the main zip file, not the source file. You should unpack this file in some directory where you can install tools, such as /home/yourusername/tools/jacoco-0.8.10 ; note that the zip file does not contain a top level directory (like jacoco-0.8.10), so you need to create it first.

Software Under Test (SUT)

For this example we use a simple Java class:

public class Bigger
{
    private int val;
    
    public Bigger(int v)
    {
        val = v;
    }
    
    // returns true if param is bigger than object value, false otherwise
    public boolean isBiggerThanMe(int otherVal)
    {
        if (otherVal >= val)
            return true;
        else
            return false;
    }
    
    public static void main(String args[])
    {
        int argVal = Integer.parseInt(args[0]);
        Bigger big = new Bigger(10);
        System.out.println("Is " + argVal + " bigger than 10? " +
              big.isBiggerThanMe(argVal));
    }
}

This code can be compiled using a simple:

javac Bigger.java

and run like:

java Bigger 4
Is 4 bigger than 10? false

which outputs the second line above.

Running a Code Coverage Capture

JaCoCo has several modes of operation, but the easiest is to load it as a Java Agent and let it work its magic at run time. Its “magic” is to instrument the code so that it sets a boolean flag every time each statement (actually, basic block) is executed. It outputs this data into a file named jacoco.exec, although you can specify a different name if you want.

So here’s the same run above, but with JaCoCo instrumentation:

java -javaagent:/home/jcook/tools/jacoco-0.8.8/lib/jacocoagent.jar Bigger 4
Is 4 bigger than 10? false

After that run, we now have a jacoco.exec data file. We then need to use another JaCoCo tool to generate a report that is readable by us. The easiest report to use and generate is an HTML report:

java -jar /home/jcook/tools/jacoco-0.8.8/lib/jacococli.jar report jacoco.exec --classfiles . --html html --sourcefiles .

The options on that line tell JaCoCo to look in the current directory (".") for both the compiled class files and the original Java source files, and to put the HTML report into a directory named “html”.

After generating the report, use your favorite web browser to visit the file link with whereever your files are, such as file:///home/jcook/ws/please-gitlab/mywork/cs371/examples/simplemax/html/index.html

It will start with an overview, but if you click down into the Bigger class you will eventually see this:

This shows that not all the code was executed, and if you click through to the source code, you can see exactly which lines were not executed!

Aggregating Data from Multiple Runs

There are different ways of telling JaCoCo to aggregate data, but the easiest is: do nothing! When you run JaCoCo and it sees that there already exists a jacoco.exec data file where it would create one, then it automatically adds to that data file with coverage stats from the new run. This makes it easy to compute coverage over a whole test set – just run all the tests using JaCoCo and make sure they are writing out to the same jacoco.exec file!

When you are ready to see the new coverage stats, then you need to just re-run the report-generating command. After running with an argument of “14” and regenerating the report, now we see 100% coverage:

Neither the “4” test nor the “14” test alone would give us 100% coverage, but with the combined data from both tests, we have full 100% coverage.

That’s great!

Offline Coverage Analysis

The example above uses online coverage instrumentation—which means that JaCoCo instruments the class file(s) when the class is loaded into the JVM at runtime. This works fine but it means the instrumentation is done every time you run the program. For small programs this may not be an issue, but for big programs, this can be quite a bit of overhead.

JaCoCo, fortunately, has the ability to instrument and then write out (save) the instrumented class files before you actually run them. The command examples below are templates for doing this, so adapt them appropriately.

1. First, compile your code and put the class files somewhere else, e.g.,:

javac -d classdir *.java

2. Instrument the class files and place the instrumented versions elsewhere:

java -jar /path/to/jacococli.jar instrument classdir/*.class --dest covinst

3. Run the program using the instrumented class files:

java -cp /path/to/jacocoagent.jar:covinst MainClass \[program-arguments\]

4. Use JaCoCo to generate a report in an “html” subdirectory:

java -jar /path/to/jacococli.jar report jacoco.exec --classfiles classdir --sourcefiles . --html html

Using JaCoCo with JUnit

You can use both online and offline JaCoCo instrumentation with JUnit, the main “trick” is to run JaCoCo as a java agent and then tell it to not instrument JUnit itself.

If you have pre-instrumented (offline) class files, then run JUnit and tell JaCoCo to not instrument anything:

java -javaagent:/path/to/jacocoagent.jar=excludes=* -jar /path/to/junit-platform-console-standalone-1.9.0.jar -cp covinst:classdir -c JUnitTestClass

If you want to do online instrumentation, then tell JaCoCo to ignore JUnit:

java -javaagent:/path/to/jacocoagent.jar=excludes=*junit* -jar /path/to/junit-platform-console-standalone-1.9.0.jar -cp classdir -c JUnitTestClass

All other compiling and JaCoCo reporting will be the same.