View on GitHub

Saga

Better code coverage tool for JavaScript

Flattr this
Download this project as a .zip file Download this project as a tar.gz file

Saga is looking for a new maintainer

Read more on the related issue page

Screenshot of example report

What? Another coverage tool?

Indeed. I've been using JSCoverage a lot - and it proved to be a pain to use in a Continuous Integration environment. In desperation, I wrote the jscoverage-maven-plugin, a Maven wrapper around JSCoverage, which did the job, except for some huge drawbacks:

  1. It directly depended on JSCoverage.
  2. It only generated the statements/executed/coverage report, even though JSCoverage itself made it possible to see a line-by-line coverage report. Of course, I simply used jscoverage-server when I needed to see the lines I missed when writing tests, but, again, it's a hassle.
  3. The user had to provide the path to the jscoverage binary, which most people would frown upon - it's just too much of a hassle. Of course, I could ship the binaries for all platforms together with the plugin and decide which one to use during runtime, but this just sounds sketchy every time I think about it...
  4. It wasn't deployed to any Maven repository, so everyone would have to build it themselves.

Having JSCoverage as a dependency proved to be painful as well:

  1. It's native - that alone makes it a pain when it comes to CI.
  2. JSCoverage, as most (all?) tools only provide the total coverage report, whereas I often find myself wishing I could also get a coverage report for every test run.
  3. It doesn't seem to be under any active development

The good things about Saga

  1. It's Java. Wait, scratch that, that's not necessarily a good thing. However, it's written in Java for a reason: the idea was that it should be extremely simple to integrate it with Maven and any other build tool. Another reason is the fact that it heavily relies on HtmlUnit and its internals for running tests and instrumentation.
  2. It can generate both total coverage and per-test reports. The good thing about it is that you get more accurate results - unit tests are designed to cover specific units. If your total coverage report shows you high coverage for a unit, that does not necessarily mean that you've covered the unit that much. Actually, it doesn't mean you've covered it at all - it just means that some test might have invoked some parts of that unit as a side effect.
  3. It's the only tool that lets you generate accurate coverage reports by allowing you to specify files that you absolutely want to see in your coverage report (via the "sourcesToPreload" parameter)
  4. It has an awesome coverage report, especially after implementing the suggestions of Marat Dyatko :)
  5. It tries to make the most of your processor power by running tests and generating coverage concurrently by default.
  6. It can generate coverage even for inline scripts.
  7. It's already in Maven Central and constantly synced, so adding it to your build cycle is pretty straightforward.
  8. It has Maven, Gradle and CLI implementations to suit your needs
  9. Support for HTML, raw LCOV, CSV and PDF reports (more to come)
  10. It just works. There's very little configuration, and I'd like to think that the defaults are sensible enough.

Maven plugin usage

Add the following piece of code to your POM:

<plugin>
    <groupId>com.github.timurstrekalov</groupId>
    <artifactId>saga-maven-plugin</artifactId>
    <version>1.5.3</version>
    <executions>
        <execution>
            <phase>verify</phase>
            <goals>
                <goal>coverage</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <baseDir>${testsBaseDir}</baseDir>
        <includes>
            **/set1/*-TestRunner.html,
            **/set2/*-TestRunner.html
        </includes>
        <outputDir>${project.build.directory}/coverage</outputDir>
    </configuration>
</plugin>

And that's it. Wait, you also have to do

mvn verify

As of 1.5.3, baseDir can be a webpage URL, like this:

<plugin>
    <groupId>com.github.timurstrekalov</groupId>
    <artifactId>saga-maven-plugin</artifactId>
    <version>1.5.3</version>
    <executions>
        <execution>
            <phase>verify</phase>
            <goals>
                <goal>coverage</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <baseDir>http://localhost:8234</baseDir>
        <outputDir>${project.build.directory}/coverage</outputDir>
    </configuration>
</plugin>

Of course, there are some more configuration options, if you feel like it:

Parameter name Description Default value
baseDir The path to the base directory for the test search OR the URL of the web page with the tests. none
includes A comma-separated list of Ant-style patterns to include in the search for test runners

Note that this parameter only makes sense if baseDir is a filesystem URL.
none
excludes A comma-separated list of Ant-style patterns to exclude from the search for test runners

Note that this parameter only makes sense if baseDir is a filesystem URL.
none
outputDir The output directory for coverage reports none
outputInstrumentedFiles Whether to output instrumented files. Will be written to ${outputDir}/instrumented false
noInstrumentPatterns A list of regular expressions to match source file paths to be excluded from instrumentation none
cacheInstrumentedCode Whether to cache instrumented source code. It's entirely possible that two tests might load some of the same resources - this would prevent them from being instrumented every time, but rather cache them for the whole coverage run. true
outputStrategy One of TOTAL, PER_TEST or BOTH. Pretty self-explanatory. TOTAL
threadCount The maximum number of threads to use.
Runtime
  .getRuntime()
  .availableProcessors()
includeInlineScripts Whether to include inline scripts into instrumentation false
backgroundJavaScriptTimeout How long to wait for background JS jobs to finish (in milliseconds) 300000
sourcesToPreload A comma-separated list of Ant-style patterns to exclude from the search for sources to preload (useful to generate total coverage even though your tests might not reference certain files, especially when you simply don't have tests for classes but you DO want to see them). Paths are expected to be provided relative to the base directory. none
sourcesToPreloadEncoding Encoding to use when preloading sources. UTF-8
browserVersion Determines the browser and version profile that HtmlUnit will simulate. This maps 1-to-1 with the public static instances found in com.gargoylesoftware.htmlunit.BrowserVersion.

Some valid examples:
  • FIREFOX_17
  • CHROME
  • INTERNET_EXPLORER_7
  • INTERNET_EXPLORER_8
  • INTERNET_EXPLORER_9
FIREFOX_17
reportFormats A comma-separated list of formats of the reports to be generated. Valid values are:
  • HTML
  • RAW
  • CSV
  • PDF
HTML, RAW
sortBy The column to sort by, one of 'file', 'statements', 'executed' or 'coverage'. coverage
order The order of sorting, one of 'asc' or 'ascending', 'desc' or 'descending'. ascending
skipTests Enables skipping coverage reporting. false

Tested on Maven version 2.2.1 & 3.x.x

Using no-instrument patterns

It might be a little confusing, but the no-instrument patterns are actually regular expressions which full script paths are matched against. So, if you had a test which loaded a script like this

<script src="someScript.js"></script>

If you wanted to exclude "someScript.js" from instrumentation, you would have to specify a pattern as follows:

<!-- ... -->

<noInstrumentPatterns>
    <pattern>.+/someScript\.js</pattern>
</noInstrumentPatterns>

<!-- ... -->

That is, if you don't care about which file named "someScript.js" gets excluded. If you do (say, you have multiple files named "someScript.js"), you would have to provide more specific patterns, such as

<!-- ... -->

<noInstrumentPatterns>
    <pattern>.+/onlyExcludeThisOne/someScript\.js</pattern>
</noInstrumentPatterns>

<!-- ... -->

And so forth.

Example projects

Just unzip them and run your regular "mvn clean verify".

jasmine-maven-plugin

Using jasmine-maven-plugin? Good, you should be!
After all, the amazing guys behind it have gone to great lengths to make integration with Saga easier, and have even created a page with an example on how to integrate saga- and jasmine-maven-plugin in a perfect ensemble! If you are too lazy to read it, there's a Jasmine with Saga sample project for that as well.

Some other testing framework

Check out the generic sample project.

Gradle support

As of version 1.3.0, Saga has a Gradle plugin. Even though using it is very similar to using Maven, I'll still provide a short example.

Add the following piece of code to your build.gradle:

buildscript {
    repositories {
        mavenCentral()
    }

    dependencies {
        classpath group: 'com.github.timurstrekalov',
            name: 'gradle-saga-plugin', version: '1.5.0'
    }
}

apply plugin: 'saga'

saga {
    baseDir = 'tests'
    outputDir = new File('build/coverage')
    includes = '**/*Test.html'
}

Now, if you want INFO-level logger output (which you see during the Maven plugin run), you have to run gradle with the -i (--info) switch:

gradle -i coverage

Configuration options are called exactly the same as those of Maven (default values are the same), though you have to sometimes be aware of their types

Parameter name Type
baseDir java.lang.String
includes java.lang.String
excludes java.lang.String
outputDir java.io.File
outputInstrumentedFiles java.lang.Boolean
noInstrumentPatterns java.util.List
cacheInstrumentedCode java.lang.Boolean
outputStrategy java.lang.String or com.github.timurstrekalov.saga.core.OutputStrategy
threadCount java.lang.Integer
includeInlineScripts java.lang.Boolean
backgroundJavaScriptTimeout java.lang.Long
sourcesToPreload java.lang.String
sourcesToPreloadEncoding java.lang.String
browserVersion java.lang.String
reportFormats java.lang.String
sortBy java.lang.String
order java.lang.String

There's also a sample project on how to use the Gradle plugin..

Other build tools

Using some other build tool? Raise a ticket!

Command-line tool

There's also a command-line utility for those not using Maven. It has pretty much the same configuration options and works just as well.

usage: java -jar saga-cli-<version>-jar-with-dependencies.jar
    -b <arg>
    -i <arg>
    -o <arg>
    [-d] [-f] [-h]
    [-e <arg>]
    [-n <arg>]
    [-s <arg>]
    [-t <arg>]
    [-j <arg>]
    [-p <arg>]
    [--preload-sources-encoding <arg>]
    [--report-formats <arg>]
    [--sort-by <arg>]
    [--order <arg>]

 -b,--base-dir <arg>                            Base directory for test search

 -d,--include-inline-scripts                    Whether to include inline scripts into
                                                instrumentation by default (default is
                                                false)

 -e,--exclude <arg>                             Comma-separated list of Ant-style
                                                paths to the tests to exclude from run

 -f,--output-instrumented-files                 Whether to output instrumented files
                                                (default is false)

 -h,--help                                      Print this message

 -i,--include <arg>                             Comma-separated list of Ant-style
                                                paths to the tests to run

 -n,--no-instrument-pattern <arg>               Regular expression patterns to match
                                                classes to exclude from
                                                instrumentation

 -o,--output-dir <arg>                          The output directory for coverage
                                                reports

 -s,--output-strategy <arg>                     Coverage report output strategy. One
                                                of [PER_TEST, TOTAL, BOTH]

 -t,--thread-count <arg>                        The maximum number of threads to use
                                                (defaults to the number of cores)

 -j,--background-javascript-timeout <arg>       How long to wait for background JS 
                                                jobs to finish

 -p,--preload-sources <arg>                     Comma-separated list of Ant-style
                                                paths to files to preload

    --preload-sources-encoding <arg>            Encoding to use when preloading
                                                sources

    --report-formats <arg>                      A comma-separated list of formats of
                                                the reports to be generated. Valid values
                                                are: HTML, RAW, CSV, PDF

    --sort-by <arg>                             The column to sort by, one of 'file',
                                                'statements', 'executed' or 'coverage'
                                                (default is 'coverage')

    --order <arg>                               The order of sorting, one of 'asc'
                                                or 'ascending', 'desc' or 'descending'
                                                (default is 'ascending')

Download a pre-built package containing the latest executable jar with dependencies.

Need an older version?

Where's the changelog?

I try to minimize the number of things I have to do, so I simply make use of GitHub's milestones in order to both organize the issues and (sort of) keep a changelog. You can find the list of closed milestones here.

Why the name?

Why not? Actually, it was suggested by Timur's wife after she started reading a book on Viking history. Plus, Maven Central search only finds this project's artifacts, so there's no ambiguity.

Why doesn't Internet Explorer render my report as nicely?

If you're looking at a JS code coverage report with IE, then I'll put it gently and not start calling you names, even though I should. In short, Saga will never support IE-compatible reports. No graceful degradation, no nothing. Reports may work or they may not - if they do, it's purely coincidental and I cannot be held responsible for it.