Saga is looking for a new maintainer
Read more on the related issue page
What? Another coverage tool?
Indeed. I've been using JSCoverage a lot - and it proved to be a pain to use in a Continuous Integration environment. In desperation, I wrote the jscoverage-maven-plugin, a Maven wrapper around JSCoverage, which did the job, except for some huge drawbacks:
- It directly depended on JSCoverage.
- It only generated the statements/executed/coverage report, even though JSCoverage itself made it possible to see a line-by-line coverage report. Of course, I simply used jscoverage-server when I needed to see the lines I missed when writing tests, but, again, it's a hassle.
- The user had to provide the path to the jscoverage binary, which most people would frown upon - it's just too much of a hassle. Of course, I could ship the binaries for all platforms together with the plugin and decide which one to use during runtime, but this just sounds sketchy every time I think about it...
- It wasn't deployed to any Maven repository, so everyone would have to build it themselves.
Having JSCoverage as a dependency proved to be painful as well:
- It's native - that alone makes it a pain when it comes to CI.
- JSCoverage, as most (all?) tools only provide the total coverage report, whereas I often find myself wishing I could also get a coverage report for every test run.
- It doesn't seem to be under any active development
The good things about Saga
- It's Java. Wait, scratch that, that's not necessarily a good thing. However, it's written in Java for a reason: the idea was that it should be extremely simple to integrate it with Maven and any other build tool. Another reason is the fact that it heavily relies on HtmlUnit and its internals for running tests and instrumentation.
- It can generate both total coverage and per-test reports. The good thing about it is that you get more accurate results - unit tests are designed to cover specific units. If your total coverage report shows you high coverage for a unit, that does not necessarily mean that you've covered the unit that much. Actually, it doesn't mean you've covered it at all - it just means that some test might have invoked some parts of that unit as a side effect.
- It's the only tool that lets you generate accurate coverage reports by allowing you to specify files that you absolutely want to see in your coverage report (via the "sourcesToPreload" parameter)
- It has an awesome coverage report, especially after implementing the suggestions of Marat Dyatko :)
- It tries to make the most of your processor power by running tests and generating coverage concurrently by default.
- It can generate coverage even for inline scripts.
- It's already in Maven Central and constantly synced, so adding it to your build cycle is pretty straightforward.
- It has Maven, Gradle and CLI implementations to suit your needs
- Support for HTML, raw LCOV, CSV and PDF reports (more to come)
- It just works. There's very little configuration, and I'd like to think that the defaults are sensible enough.
Maven plugin usage
Add the following piece of code to your POM:
<plugin>
<groupId>com.github.timurstrekalov</groupId>
<artifactId>saga-maven-plugin</artifactId>
<version>1.5.5</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>coverage</goal>
</goals>
</execution>
</executions>
<configuration>
<baseDir>${testsBaseDir}</baseDir>
<includes>
**/set1/*-TestRunner.html,
**/set2/*-TestRunner.html
</includes>
<outputDir>${project.build.directory}/coverage</outputDir>
</configuration>
</plugin>
And that's it. Wait, you also have to do
mvn verify
baseDir can be a webpage URL, like this:
<plugin>
<groupId>com.github.timurstrekalov</groupId>
<artifactId>saga-maven-plugin</artifactId>
<version>1.5.5</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>coverage</goal>
</goals>
</execution>
</executions>
<configuration>
<baseDir>http://localhost:8234</baseDir>
<outputDir>${project.build.directory}/coverage</outputDir>
</configuration>
</plugin>
Of course, there are some more configuration options, if you feel like it:
Parameter name | Description | Default value |
---|---|---|
baseDir | The path to the base directory for the test search OR the URL of the web page with the tests. | none |
includes |
A comma-separated list of
Ant-style patterns to include in
the search for test runners
Note that this parameter only makes sense if baseDir is a filesystem URL. |
none |
excludes |
A comma-separated list of
Ant-style patterns to exclude from
the search for test runners
Note that this parameter only makes sense if baseDir is a filesystem URL. |
none |
outputDir | The output directory for coverage reports | none |
outputInstrumentedFiles | Whether to output instrumented files. Will be written to ${outputDir}/instrumented | false |
noInstrumentPatterns | A list of regular expressions to match source file paths to be excluded from instrumentation | none |
cacheInstrumentedCode | Whether to cache instrumented source code. It's entirely possible that two tests might load some of the same resources - this would prevent them from being instrumented every time, but rather cache them for the whole coverage run. | true |
outputStrategy | One of TOTAL, PER_TEST or BOTH. Pretty self-explanatory. | TOTAL |
threadCount | The maximum number of threads to use. |
Runtime .getRuntime() .availableProcessors() |
includeInlineScripts | Whether to include inline scripts into instrumentation | false |
backgroundJavaScriptTimeout | How long to wait for background JS jobs to finish (in milliseconds) | 300000 |
sourcesToPreload | A comma-separated list of Ant-style patterns to exclude from the search for sources to preload (useful to generate total coverage even though your tests might not reference certain files, especially when you simply don't have tests for classes but you DO want to see them). Paths are expected to be provided relative to the base directory. | none |
sourcesToPreloadEncoding | Encoding to use when preloading sources. | UTF-8 |
browserVersion | Determines the browser and version profile that HtmlUnit will simulate.
This maps 1-to-1 with the public static instances found in
com.gargoylesoftware.htmlunit.BrowserVersion.
Some valid examples:
|
FIREFOX_17 |
reportFormats |
A comma-separated list of formats of the reports to be generated. Valid values are:
|
HTML, RAW |
sortBy | The column to sort by, one of 'file', 'statements', 'executed' or 'coverage'. | coverage |
order | The order of sorting, one of 'asc' or 'ascending', 'desc' or 'descending'. | ascending |
skipTests | Enables skipping coverage reporting. | false |
Tested on Maven version 2.2.1 & 3.x.x
Using no-instrument patterns
It might be a little confusing, but the no-instrument patterns are actually regular expressions which full script paths are matched against. So, if you had a test which loaded a script like this
<script src="someScript.js"></script>
If you wanted to exclude "someScript.js" from instrumentation, you would have to specify a pattern as follows:
<!-- ... -->
<noInstrumentPatterns>
<pattern>.+/someScript\.js</pattern>
</noInstrumentPatterns>
<!-- ... -->
That is, if you don't care about which file named "someScript.js" gets excluded. If you do (say, you have multiple files named "someScript.js"), you would have to provide more specific patterns, such as
<!-- ... -->
<noInstrumentPatterns>
<pattern>.+/onlyExcludeThisOne/someScript\.js</pattern>
</noInstrumentPatterns>
<!-- ... -->
And so forth.
Example projects
Just unzip them and run your regular "mvn clean verify".
jasmine-maven-plugin
Using jasmine-maven-plugin? Good, you should be!
After all, the amazing guys behind it have gone to great lengths to make integration with Saga easier, and have even created a page with an
example on how to integrate saga- and jasmine-maven-plugin in a perfect ensemble! If you are too lazy to read it, there's a Jasmine with Saga sample project for that as well.
Some other testing framework
Check out the generic sample project.
Gradle support
As of version 1.3.0, Saga has a Gradle plugin. Even though using it is very similar to using Maven, I'll still provide a short example.
Add the following piece of code to your build.gradle:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath group: 'com.github.timurstrekalov',
name: 'gradle-saga-plugin', version: '1.5.5'
}
}
apply plugin: 'saga'
saga {
baseDir = 'tests'
outputDir = new File('build/coverage')
includes = '**/*Test.html'
}
Now, if you want INFO-level logger output (which you see during the Maven plugin run), you have to run gradle with the -i (--info) switch:
gradle -i coverage
Configuration options are called exactly the same as those of Maven (default values are the same), though you have to sometimes be aware of their types
Parameter name | Type |
---|---|
baseDir | java.lang.String |
includes | java.lang.String |
excludes | java.lang.String |
outputDir | java.io.File |
outputInstrumentedFiles | java.lang.Boolean |
noInstrumentPatterns | java.util.List |
cacheInstrumentedCode | java.lang.Boolean |
outputStrategy | java.lang.String or com.github.timurstrekalov.saga.core.OutputStrategy |
threadCount | java.lang.Integer |
includeInlineScripts | java.lang.Boolean |
backgroundJavaScriptTimeout | java.lang.Long |
sourcesToPreload | java.lang.String |
sourcesToPreloadEncoding | java.lang.String |
browserVersion | java.lang.String |
reportFormats | java.lang.String |
sortBy | java.lang.String |
order | java.lang.String |
There's also a sample
project on how to use the Gradle plugin..
Other build tools
Using some other build tool? Raise a ticket!
Command-line tool
There's also a command-line utility for those not using Maven. It has pretty much the same configuration options and works just as well.
usage: java -jar saga-cli-<version>-jar-with-dependencies.jar
-b <arg>
-i <arg>
-o <arg>
[-d] [-f] [-h]
[-e <arg>]
[-n <arg>]
[-s <arg>]
[-t <arg>]
[-j <arg>]
[-p <arg>]
[--preload-sources-encoding <arg>]
[--report-formats <arg>]
[--sort-by <arg>]
[--order <arg>]
-b,--base-dir <arg> Base directory for test search
-d,--include-inline-scripts Whether to include inline scripts into
instrumentation by default (default is
false)
-e,--exclude <arg> Comma-separated list of Ant-style
paths to the tests to exclude from run
-f,--output-instrumented-files Whether to output instrumented files
(default is false)
-h,--help Print this message
-i,--include <arg> Comma-separated list of Ant-style
paths to the tests to run
-n,--no-instrument-pattern <arg> Regular expression patterns to match
classes to exclude from
instrumentation
-o,--output-dir <arg> The output directory for coverage
reports
-s,--output-strategy <arg> Coverage report output strategy. One
of [PER_TEST, TOTAL, BOTH]
-t,--thread-count <arg> The maximum number of threads to use
(defaults to the number of cores)
-j,--background-javascript-timeout <arg> How long to wait for background JS
jobs to finish
-p,--preload-sources <arg> Comma-separated list of Ant-style
paths to files to preload
--preload-sources-encoding <arg> Encoding to use when preloading
sources
--report-formats <arg> A comma-separated list of formats of
the reports to be generated. Valid values
are: HTML, RAW, CSV, PDF
--sort-by <arg> The column to sort by, one of 'file',
'statements', 'executed' or 'coverage'
(default is 'coverage')
--order <arg> The order of sorting, one of 'asc'
or 'ascending', 'desc' or 'descending'
(default is 'ascending')
Download a pre-built package containing the latest executable jar with dependencies.
Need an older version?
Where's the changelog?
I try to minimize the number of things I have to do, so I simply make use of GitHub's milestones in order to both organize the issues and (sort of) keep a changelog. You can find the list of closed milestones here.
Why the name?
Why not? Actually, it was suggested by Timur's wife after she started reading a book on Viking history. Plus, Maven Central search only finds this project's artifacts, so there's no ambiguity.
Why doesn't Internet Explorer render my report as nicely?
If you're looking at a JS code coverage report with IE, then I'll put it gently and not start calling you names, even though I should. In short, Saga will never support IE-compatible reports. No graceful degradation, no nothing. Reports may work or they may not - if they do, it's purely coincidental and I cannot be held responsible for it.