wiki:TestingXSLT

Testing XSLT

Overview

Creating a working stylesheet may seem like an end in itself, but once it’s written you may want it to run faster or you may not be sure that the output is correct (And if you are sure, how sure are you?).

Profilers, unit test frameworks, and other tools of conventional programming are similarly available for XSLT but are not widely used. This presentation surveys the available tools for ensuring the quality of your XSLT.

There is no one-size-fits-all solution when looking for tools. For example, if you are using Saxon and Ant, then you are looking for a different set of tools than if you are using libXSLT and Makefiles.

Profilers

Profilers are in wide use for conventional programming, and several XSLT and XSL profilers are available. However, it is a truism of XSLT programming that different XSLT processors have their own strengths and weaknesses, so if you are profiling your stylesheet, it is important to profile it running on the same XSLT processor as you will use in production.

Some XSLT processors, such as Saxon and libXSLT, have their own profiling mechanisms, and several XSLT editors or IDEs, such as <oXygen/>, Stylus Studio, and XML Spy, provide built-in profilers what work with a number of different XSLT processors.

Profiling XSLT is not an exact science since:

  • The execution time for a template includes the execution times of all of the templates that it calls; the processor and IDE vendors do their best to separate the two when reporting timing.
  • Results depend on the state of the machine; running the stylesheet multiple times in succession generally means the later runs are faster than the earlier as the program and its libraries are already in memory.
    • Both Saxon and xsltproc provide a command-line switch for running the transformation time multiple times to counteract irregularities in the timing of a single run.
  • The time taken by a particular template may depend as much on the current node list as on the structure of the template.

The presentation will describe the features and applicability of the different XSLT and XSL profilers available as well as a few hints and tips about how to get the best out of your profiler: such as what to do when the profiler reports that your hottest hotspot is a literal result element instead of the complicated XPath selector that you expected.

XML IDEs

XML IDEs such as <oXygen/>, Stylus Studio, and XMLSpy support profiling and debugging of XSLT. Figure 1 shows <oXygen/> profiling information display.

oXygen XSLT profiling. Figure 1 – <oXygen/> XSLT profiling

xsltproc

The xsltproc XSLT 1.0 processor supports a --profile switch that triggers a dump of profiling information.

See xsltproc

Saxon

Saxon provides a profile through a two part process.

See Saxon

Test Frameworks

Not all testing frameworks are unit testing frameworks, and not all tests are unit tests.

Unit tests – tests written from the perspective of the programmer – came to prominence with the rise in popularity of the Extreme Programming (XP) methodology in the late 1990s. Programmers have always recognised that they should write tests for their code, but that hasn’t always meant that they do. Writing tests before writing the code is central to XP, so the publicity about XP brought unit testing to the attention of many programmers, irrespective of whether they adopted all, some, or none of the XP methodology. The practise can be separated from the rest of the XP bag of tricks and referred to as Test-Driven Development (TDD).

Unit testing is perhaps most commonly associated with the Java programming language since XP’s creators also wrote the JUnit unit testing framework and since Ant, the ubiquitous Java-based build tool, makes it easy to run JUnit tests and generate HTML reports of the results. Unit testing tools are available for a wide variety of programming languages, including XSLT, but current awareness of XSL and XSLT unit testing tools is limited.

Your choice of XSLT unit testing framework depends less on your XSLT processor than it does on your XSLT version and your testing approach. There are already several unit testing frameworks specific to XSLT 2.0 (which will limit your choice of XSLT processor), but otherwise your choice depends on whether you want to work purely in XSLT, within Ant, with Ant and JUnit, with just JUnit. Again, your choice will depend on what other tools you are using.

XSLT unit testing frameworks include:

A testing framework of a different kind is the XSLV static validation tool available from  http://www.brics.dk/XSLV/.

XSpec (  http://code.google.com/p/xspec/) is a Behavior Driven Development (BDD) framework for XSLT.

Effectiveness of Unit Testing

Few people, if any, would claim that unit testing is the silver bullet that will kill all your software defects. The industry data summarised in Software Estimation: Demystifying the Black Art [MCC06] indicates it is most often effective in removing 30% of the defects in the code being tested.

Removal Step Lowest Rate Modal Rate Highest Rate
Informal design reviews25%35%40%
Formal design inspections45%55%65%
Informal code reviews20%25%35%
Formal code inspections45%60%70%
Modelling or prototyping35%65%80%
Personal desk checking of code20%40%60%
Unit test15%30%50%
New function (component) test20%30%35%
Integration test25%35%40%
Regression test15%25%30%
System test25%40%55%
Low-volume beta test (<10 sites)25%35%40%
High-volume beta test (>1,000 sites)60%75%85%

Table 1 – Defect-removal Rates (from [MCC06])

As with so many things, you mileage may vary, and historical data from your own organisation will be a better indication of how effective this, or any technique, is for you.

Unit Testing Wish-List

My personal wish-list for an XSLT unit testing framework includes:

  • Ability to assert that a message was (or was not) output.
    • If all an xsl:template does is emit a message, the result from the template will, in XSLT 2.0 terms, be just an empty sequence. There are times when it is useful to check that the template took the path less travelled and emitted a message rather than just failed to produce a result.
  • Works with keys
    • When the unit testing framework and test input data are both in a stylesheet that imports the style sheet being tested, there is no input document for which to generate key data. Testing with uninitialised keys just leads to frustration.
  • Test stylesheet as a whole
    • A framework should be able to test by running the whole stylesheet as well as by exercising individual templates.
  • Report results from multiple unit test files
    • Most of the unit test frameworks are designed to run a single file containing the unit tests. When there are a lot of tests or when the spec for the transform has multiple parts, it makes sense to split the unit tests into multiple parts, each in a different file. A framework should be able to run multiple files of unit tests in one invocation and report on the combined results of all the tests.
  • Maintainers respond to bug reports and enhancement requests
  • Test data and assertions written in XML
    • Several of the unit testing frameworks use tests written in Java and/or the tests are run using Ant. However, I do more than just jobs that use one or both of Java and Ant (what I do use at any point is often what my clients are currently using), so I personally prefer that tests can be written in XML and to have the option of running tests from the command line so that it’s not necessary for users to know a particular programming language or use a particular build tool to be able to create and run unit tests.

Why use more that XSLT for testing XSLT?

Why should you prefer an XSLT unit testing framework that uses more that just XSLT? Two reasons: xsl:message and multiple output documents.

Most XSLT unit testing frameworks are written purely in XSLT; that is, the framework uses an XSLT processor and XSLT stylesheet to run the tests (or a massaged form of the tests) on the stylesheet under test. The test result output is typically XML that is processed by another stylesheet to produce HTML for viewing.

Using XSLT to test XSLT may be partly explained by XSLT being an XSLT practitioner’s favourite tool and partly by Eric van der Vlist’s XSLT-based XSLTunit being the first known framework. Jeni Tennison’s possibly unnamed framework is also purely XSLT (though XSLT 2.0 instead of the XSLT 1.0 in XSLTunit), and it inspired the Tennison Tests framework, which automates running the multiple tests but hasn’t tinkered with the XSLT nature of the tests, and also XTC.

By comparison, the frameworks that rely on compiling code to run tests are, it seems, much less well known. They include Juxy, which compiles Java code for JUnit tests, and XMLUnit, which has versions in Java for use with JUnit and C# for use with NUnit.

IMO, the extra-XSLT frameworks have the advantage that you can (or should be able to) make assertions about aspects of the stylesheet to which a pure XSLT framework is either blind or oblivious.

Inasmuch as you typically use xsl:message to output messages when there’s an error in the source document or there’s something that the stylesheet can’t handle, you should want to make assertions about whether or not a message has been emitted. This is particularly true when you are writing “dirty” tests that you expect will trigger error handling (as opposed to “clean” tests that you expect to work), and even more true when you expect to execute a xsl:message that has ‘terminate="yes"‘.

Pure XSLT frameworks don’t provide a way to tell when a message is emitted, and any <xsl:message terminate="yes"> will terminate the testing framework along with the stylesheet under test, leaving you with nothing. A good extra-XSLT framework will let you make assertions about xsl:message and will not evaporate when the stylesheet terminates itself.

When a stylesheet may create multiple result documents, you probably want to make assertions about the existence and content of those result documents. That could be left to a post-process, but if the file names and the content relate to the content of the source document, it is simpler and more self-contained when you could make your assertions in the same unit tests as you use to make assertions about the rest of the operation of the stylesheet. A pure XSLT framework can’t provide any indication that a result document was or was not created, though it may be possible to access a secondary result document as part of the unit tests (provided the XSLT processor finishes writing the secondary result document while the transformation-that-is-the-running-of-the-unit-tests is still in progress). An extra-XSLT framework typically lets you make assertions after the stylesheet-under-test has finished.

So while there are some aspects of XSLT processing, such as what happens on a xsl:message, that you as a stylesheet writer don’t need to concern yourself about, you as a stylesheet tester do want to know about them, so you need more that just an XSLT processor in the framework for your unit tests.

XSLTunit

XSLTunit by Eric van der Vlist of Dyomedea is the grandfather of XSLT unit testing frameworks. It has influenced the development of several other testing projects.

See XSLTUnit.

Juxy

Juxy describes itself as “a library for unit testing XSLT stylesheets from Java”. It works with JUnit. It states that it is best suited for the projects where both Java and XSLT are used simultaneously.

See Juxy.

Unit Testing XSLT

Jeni Tennison’s unit testing framework, available from  http://www.jenitennison.com/xslt/utilities/unit-testing/ under the title “Unit Testing XSLT”, is a pure XSLT 2.0 solution where unit tests may be either in the stylesheet being tested or in a separate file.

See Unit Testing XSLT.

Should be considered to be superseded by XSpec. See XSpec.

tennison-tests

The tennison-tests project on SourceForge (  http://tennison-tests.sourceforge.net/) marries Jeni Tennison’s unit testing stylesheets with an Ant task for running one or more unit test files and producing reports.

See Tennison Tests.

Should be considered to be superseded by XSpec. See XSpec.

<XmlUnit/>

<XmlUnit/>, from  http://xmlunit.sourceforge.net/, is available in two forms: a Java framework (for use both with and without JUnit) and a less well-developed C# framework for use with NUnit.

See XmlUnit.

Unit Testing Framework – XSLT (UTF-X)

UTF-X, available from  http://utf-x.sourceforge.net/, is a Java-based framework where tests can be run from the command line, from an Ant build file, or using JUnit. Tests are written as XML.

See UTF-X.

XTC

XTC, by Florent Georges, is available from  http://www.fgeorges.org/xslt/xslt-unit/. XTS is an XSLT 2.0 framework that is capable of testing both XSLT 2.0 and XQuery. A single test comprises an assertion of the expected result followed by a sequence constructor.

XTC should be considered superseded by XSpec, of which Florent Georges is the current maintainer.

See XTC.

XSpec

XSpec (  http://code.google.com/p/xspec/), by Jeni Tennison and contributors, is a Behavior Driven Development (BDD) framework for XSLT. It is based on the Spec framework of RSpec, which is a BDD framework for Ruby.

See XSpec.

<jxsl/>

<jxsl/> (  http://code.google.com/p/jxsl/) puts a Java wrapper around XSpec tests so they can be used with JUnit and, in particular, used in a Continuous Integration (CI) setup.

Static Tests

XSLV Static Validation Tool

The XSLV tool for static validation of XSLT from the University of Aarhus, available online at  http://www.brics.dk/XSLV/, is able to check that all output of a stylesheet at runtime is valid according to a specified output schema, assuming that the input is valid according to its specified schema. Schemas can be written as DTD, XML Schema, or a subset of RELAX NG.

See XSLV.

xslqual.xsl

xslqual.xsl (  http://gandhimukul.tripod.com/xslt/xslquality.html), by Mukul Gandhi, runs a number of XSLT code quality rules. Each rule in this file is in a form of an XML fragment, something like following:

<rule name="DontUseDoubleSlashOperatorNearRoot">
    <message>
        Avoid using the operator // near the root of a large tree
    </message>
    <xpath>
        //(@match | @select)[starts-with(., '//')]
    </xpath>
    <priority>2</priority>
    <example>
        <!-- VIOLATION -->
        <xsl:variable name="x" select="//x/y/z" />

        <!-- REPAIR -->
        <xsl:variable name="x" select="/a/b/x/y/z" />
    </example>
</rule>

XSLT Metrics

XSLT Metrics (  http://code.menteithconsulting.com/wiki/XSLTMetrics), by Tony Graham, is a stylesheet for some descriptive, rather than prescriptive or proscriptive, metrics about what's in a stylesheet. The purpose of the metrics is finding out what's in a stylesheet rather than telling you how to write your stylesheet.

See XSLT Metrics.

debugxslt

debugxslt (  http://code.google.com/p/debugxslt/), by James Fuller, is an XSLT lint checker that uses Schematron.

XSLStyle

XSLStyle (  http://www.cranesoftwrights.com/resources/xslstyle/index.htm), by Crane Softwrights, is a system for embedding documentation in an XSLT stylesheet that also includes consistency checks on the XSLT.

Coverage

Testing your XSLT is good. Knowing how much of your XSLT you've tested is even better. A coverage tool is able to report which portions of a stylesheet have been exercised and which haven't.

[wiki:TestingXSLT/XSpec

XSpec (  http://code.google.com/p/xspec/) has the only XSLT coverage utility currently known. See XSpec Coverage.

Bibliography

[MCC06]

Steve McConnell, Software Estimation: Demystifying the Black Art, ISBN 0-7356-0535-1, Microsoft Press, Redmond, Washington, 2006.