wiki:XTech2007/MyStylesheetRunsBut

My Stylesheet Runs But…

Tony Graham
Menteith Consulting

Presented at XTech 2007 in Paris, France, on 18th May 2007.

Overview

Creating a working stylesheet may seem like an end in itself, but once it’s written you may want it to run faster or you may not be sure that the output is correct (And if you are sure, how sure are you?).

Profilers, unit test frameworks, and other tools of conventional programming are similarly available for XSLT but are not widely used. This presentation surveys the available tools for ensuring the quality of your XSLT.

There is no one-size-fits-all solution when looking for tools. For example, if you are using Saxon and Ant, then you are looking for a different set of tools than if you are using libXSLT and Makefiles.

…but it’s too slow

Profilers are in wide use for conventional programming, and several XSLT and XSL profilers are available. However, it is a truism of XSLT programming that different XSLT processors have their own strengths and weaknesses, so if you are profiling your stylesheet, it is important to profile it running on the same XSLT processor as you will use in production.

Some XSLT processors, such as Saxon and libXSLT, have their own profiling mechanisms, and several XSLT editors or IDEs, such as <oXygen/>, Stylus Studio, and XML Spy, provide built-in profilers what work with a number of different XSLT processors.

Profiling XSLT is not an exact science since:

  • The execution time for a template includes the execution times of all of the templates that it calls; the processor and IDE vendors do their best to separate the two when reporting timing.
  • Results depend on the state of the machine; running the stylesheet multiple times in succession generally means the later runs are faster than the earlier as the program and its libraries are already in memory.

o Only xsltproc provides a command-line switch for running the transformation time multiple times to counteract irregularities in the timing of a single run.

  • The time taken by a particular template may depend as much on the current node list as on the structure of the template.

The presentation will describe the features and applicability of the different XSLT and XSL profilers available as well as a few hints and tips about how to get the best out of your profiler: such as what to do when the profiler reports that your hottest hotspot is a literal result element instead of the complicated XPath selector that you expected.

XML IDEs

XML IDEs such as <oXygen/>, Stylus Studio, and XMLSpy support profiling and debugging of XSLT. Figure 1 shows <oXygen/> profiling information display. This presentation, however, does not go into details of IDE support since XSLT profiling support is likely to be just a small part of why you’d choose to use a particular IDE.

oXygen XSLT profiling. Figure 1 – <oXygen/> XSLT profiling

xsltproc

The xsltproc XSLT 1.0 processor supports a --profile switch that triggers a dump of profiling information. The example below shows the first few lines of output from profiling one of the code-generating stylesheets for the xmlroff XSL formatter:

xsltproc --profile --stringparam dump-info dump-info.xml fo-context-dump.xsl xslspec.xml
number               match                name      mode  Calls Tot 100us Avg

    0                     property-to-context-property-merge
                                                            172 426224   2478
    1                        fo-context-c-file                1 315848 315848
    2                     property-to-slist-foreach-if
                                                            172 143712    835
    3                     property-to-context-property-copy
                                                            172 107006    622
...

Saxon

Saxon 8 provides a profile through a two part process:

  1. Run Saxon with the -TP switch, writing the error output to a file
    java -jar saxon8.jar -TJ xslspec.xml fo-context-dump.xsl  dump-info=dump-info.xml 2> profile.xml
    
  2. Run the timing-profile.xsl stylesheet from the saxon-resources.zip file (available from  http://www.saxonica.com/) to create a HTML report:
    java -jar saxon8.jar profile.xml timing-profile.xsl  > profile.html
    

Figure 2 shows a screenshot of the report generated from running the same stylesheet as in the xsltproc example. Notice the first two “hotspots” are the same as for xsltproc, but after that the result start to diverge.

Saxon profile report. Figure 2 – Saxon profile report ...but why is a simple <hr/> a hotspot?

Hotspot display detail. Figure 3 – Hotspot detail

If an XSLT processor doesn’t provide hooks for measuring timing, or if the profiler uses a general technique that works with multiple processors, the best the profiler can do is to periodically pause the XSLT processor and determine what is currently happening.

However, if the sampling is too infrequent, then it’s almost random what is happening when the profiler pauses the processor. Conversely, if sampling is too frequent, the overhead of the profiler will skew the results.

If your hotspots aren’t quite what you expect, you may need to run the stylesheet for longer. xsltproc provides the --repeat switch, for use with the --profile switch or the –timing switch, which repeats the transformation 20 times. If you’re not using xsltproc, you can use a stylesheet that imports your original stylesheet and runs it multiple times:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="1.0">
  <xsl:import href="original.xsl"/>
  <xsl:variable name="randomNodes" select="//node()"/>
  
  <xsl:template match="/">
    <xsl:variable name="root" select="/"/>
    <xsl:for-each select="$randomNodes[position() &lt; 20]">
      <xsl:apply-templates select="." name="apply-imports"/>
    </xsl:for-each>
  </xsl:template>
  
  <xsl:template name="apply-imports">
    <xsl:apply-templates select="." mode="apply-imports"/>
  </xsl:template>
  
  <xsl:template match="/" mode="apply-imports">
    <xsl:apply-imports/>
  </xsl:template>
</xsl:stylesheet>

...So what about compiling or hardware?

Your options include:

  • XSLTC, part of Apache Xalan, compiles stylesheets to “translets”
  • Ambrosoft offers the Gregor XSLT compiler
  • IBM (was Datapower) and Intel (was Sarvega) offer hardware accelerators for XML and XSLT processing.

…but how do I know it’s correct?

Unit tests – tests written from the perspective of the programmer – came to prominence with the rise in popularity of the Extreme Programming (XP) methodology in the late 1990s. Programmers have always recognised that they should write tests for their code, but that hasn’t always meant that they do. Writing tests before writing the code is central to XP, so the publicity about XP brought unit testing to the attention of many programmers, irrespective of whether they adopted all, some, or none of the XP methodology. The practise can be separated from the rest of the XP bag of tricks and referred to as Test-Driven Development (TDD).

Unit testing is perhaps most commonly associated with the Java programming language since XP’s creators also wrote the JUnit unit testing framework and since Ant, the ubiquitous Java-based build tool, makes it easy to run JUnit tests and generate HTML reports of the results. Unit testing tools are available for a wide variety of programming languages, including XSLT, but current awareness of XSL and XSLT unit testing tools is limited.

Your choice of XSLT unit testing framework depends less on your XSLT processor than it does on your XSLT version and your testing approach. There are already several unit testing frameworks specific to XSLT 2.0 (which will limit your choice of XSLT processor), but otherwise your choice depends on whether you want to work purely in XSLT, within Ant, with Ant and JUnit, with just JUnit. Again, your choice will depend on what other tools you are using.

XSLT unit testing frameworks include:

A testing framework of a different kind is the XSLV static validation tool available from  http://www.brics.dk/XSLV/.

Effectiveness of Unit Testing

Few people, if any, would claim that unit testing is the silver bullet that will kill all your software defects. The industry data summarised in Software Estimation: Demystifying the Black Art [MCC06] indicates it is most often effective in removing 30% of the defects in the code being tested.

Removal StepLowest RateModal RateHighest Rate
Informal design reviews25%35%40%
Formal design inspections45%55%65%
Informal code reviews20%25%35%
Formal code inspections45%60%70%
Modelling or prototyping35%65%80%
Personal desk checking of code20%40%60%
Unit test15%30%50%
New function (component) test20%30%35%
Integration test25%35%40%
Regression test15%25%30%
System test25%40%55%
Low-volume beta test (<10 sites)25%35%40%
High-volume beta test (>1,000 sites)60%75%85%

Table 1 – Defect-removal Rates (from [MCC06])

As with so many things, you mileage may vary, and historical data from your own organisation will be a better indication of how effective this, or any technique, is for you. Unit Testing Wish-List

My personal wish-list for an XSLT unit testing framework includes:

  • Ability to assert that a message was (or was not) output.
    • If all an xsl:template does is emit a message, the result from the template will, in XSLT 2.0 terms, be just an empty sequence. There are times when it is useful to check that the template took the path less travelled and emitted a message rather than just failed to produce a result.
  • Works with keys
    • When the unit testing framework and test input data are both in a stylesheet that imports the style sheet being tested, there is no input document for which to generate key data. Testing with uninitialised keys just leads to frustration.
  • Test stylesheet as a whole
    • A framework should be able to test by running the whole stylesheet as well as by exercising individual templates.
  • Report results from multiple unit test files
    • Most of the unit test frameworks are designed to run a single file containing the unit tests. When there are a lot of tests or when the spec for the transform has multiple parts, it makes sense to split the unit tests into multiple parts, each in a different file. A framework should be able to run multiple files of unit tests in one invocation and report on the combined results of all the tests.
  • Maintainers respond to bug reports and enhancement requests
  • Test data and assertions written in XML
    • Several of the unit testing frameworks use tests written in Java and/or the tests are run using Ant. However, I do more than just jobs that use one or both of Java and Ant (what I do use at any point is often what my clients are currently using), so I personally prefer that tests can be written in XML and to have the option of running tests from the command line so that it’s not necessary for users to know a particular programming language or use a particular build tool to be able to create and run unit tests.

XSLTunit

XSLTunit by Eric van der Vlist of Dyomedea is the grandfather of XSLT unit testing frameworks. When asked, Eric described it as “stable, rustic and mature”. It has influenced, both positively and negatively, the development of several other testing projects.

XSLTunit tests are written in a stylesheet. The stylesheet imports both the stylesheet being tested and the XSLTunit “xsltunit.xsl” stylesheet. The unit tests are written within a template that matches the document root (so it is the first rule executed when the combined stylesheets are run).

This example from the XSLTunit web site illustrates how elements in the XSLTunit namespace do the work and an xsl:apply-template invokes the stylesheet being tested to produce a result that is to be compared against the expected result.

<xsl:template match="/">
  <xsltu:tests>
    <xsltu:test id="test-title">
      <xsl:call-template name="xsltu:assertEqual">
        <xsl:with-param name="id" select="'full-value'"/>

        <xsl:with-param name="nodes1">
          <xsl:apply-templates select="document('library.xml')/library/book[isbn='0836217462']/title"/>
        </xsl:with-param>
        <xsl:with-param name="nodes2">
          <h1>Being a Dog Is a Full-Time Job</h1>
        </xsl:with-param>
      </xsl:call-template>
    </xsltu:test>
…
  </xsltu:tests>
</xsl:template>

The result from running XSLTunit is an XML document indicating the success or failure of each test. When a test asserting equality with an expected result fails, the output also includes a diff of the expected and actual results. The following is the result from running the sample files from the XSLTunit web site:

<xsltu:tests xmlns:xsltu="http://xsltunit.org/0/">
  <xsltu:test id="test-title">
    <xsltu:assert id="full-value" outcome="passed"/>
  </xsltu:test>
  <xsltu:test id="test-title-reverted">
    <xsltu:assert id="non-empty-h1" outcome="passed"/>
  </xsltu:test>
  <xsltu:test id="XPath-expressions">
    <xsltu:assert id="h1" outcome="passed"/>
    <xsltu:assert id="value" outcome="passed"/>
  </xsltu:test>
</xsltu:tests>

Juxy

Juxy describes itself as “a library for unit testing XSLT stylesheets from Java”. It states that it is best suited for the projects where both Java and XSLT are used simultaneously.

In contrast to XSLTunit and its descendants, Juxy tests are written in Java. Tests can be written to run standalone, for use with JUnit, or for use with any other (probably Java-based) testing framework. Tests for use with JUnit are the most likely use for Juxy, both because the tests are less verbose and so are easier to both read and write and because many people are already using JUnit for testing Java code.

The input being tested can be from a file, a Document object or a String, as shown in the following JUnit-specific example excerpted from the Juxy website:

public class SampleTestCase extends JuxyTestCase {
  public void testListTransformation() {
      newContext("stylesheet.xsl");
      context().setDocument("" +
          "<list>" +
          "	<item>item 1</item>" +
          "	<item>item 2</item>" +
          "	<item>item 3</item>" +
          "</list>");
      Node result = applyTemplates();
      xpathAssert("text()", "item 1, item 2, item 3").eval(result);
  }
}

Unit Testing XSLT

Jeni Tennison’s unit testing framework, available from  http://www.jenitennison.com/xslt/utilities/unit-testing/ under the title “Unit Testing XSLT”, is a pure XSLT 2.0 solution where unit tests may be either in the stylesheet being tested or in a separate file. The following example from Jeni’s web site shows a simple test of an XSLT 2.0 function:

<test:tests>
  <test:test>
    <test:param name="number" select="2" />
    <test:expect select="4" />
  </test:test>
</test:tests>
<xsl:function name="eg:square" as="xs:double">
  <xsl:param name="number" as="xs:double" />
  <xsl:sequence select="$number * $number" />
</xsl:function>

A stylesheet containing tests and templates is transformed using a provided stylesheet to generate a new, standalone stylesheet that contains only the tests and that imports the original stylesheet. Running this stylesheet (irrespective of what you use for input) runs the tests and produces an XML result file. This result is then transformed using another provided stylesheet to produce a HTML report (defaulting to using Jeni’s distinctive purple and green colour scheme) that summarises the results.

Figure 4 shows the report generated using the above test plus a second test that is forced to fail (since 2 * 2 does not equal 5) to show the details that are provided for failed tests.

Unit Testing XSLT report. Figure 4 – Unit test report

tennison-tests

The tennison-tests project on SourceForge marries Jeni Tennison’s unit testing stylesheets with an Ant task for running one or more unit test files and producing reports. This framework is most likely to be useful to someone used to using Ant and already using it for unit testing non-XSLT code. As the tennison-tests web page puts it:

The Tennison Tests (XSLT Unit Testing) project aims to harvest the best of both worlds, allowing XSLT Developer's to write their tests in XML and appease the nUnit camp by providing an easy integration into automated build tools, specifically Ant.

Using tennison-tests requires adding the task’s definition and then using it in an Ant target, for example:

<!-- ============================================================= -->
<!-- Targets for running unit tests                                -->
<!-- ============================================================= -->

  <!-- Defines the 'tennison-tests' custom Ant task. -->
  <taskdef name="xslttest"
	   classname="com.jenitennison.xslt.unittest.XSLTTest"
	   classpath="${basedir}/ant-xslttest-1.0.0/ant-xslttest-1.0.0.jar"/>

  <!-- Executes all unit tests in 'test'. -->
  <target name="test" depends="init">
    <xslttest src="${basedir}/ant-xslttest-1.0.0/main/src/xslt"
	      target="build/test"
	      generate="true">
      <fileset dir="${basedir}/test">
	<include name="*.xml" />
      </fileset>
      <factory name="net.sf.saxon.TransformerFactoryImpl"/>
    </xslttest>
  </target>

  <!-- Executes a single unit test. -->
  <!-- Example usage:
          ant -Dtest=2-6.xml test.single
  -->
  <target name="test.single" depends="init">
    <xslttest src="${basedir}/ant-xslttest-1.0.0/main/src/xslt"
	      target="build/test"
	      generate="true">
      <fileset dir="${basedir}/test">
	<include name="${test}" />
      </fileset>
      <factory name="net.sf.saxon.TransformerFactoryImpl"/>
    </xslttest>
  </target>

The generated reports appear identical to those produced by Jeni’s original stylesheets (apart from using a different colour scheme). While the Ant task can run multiple unit test files on one invocation, as yet it does not produce a summary report of all the individual unit test file’s results.

<XmlUnit/>

<XmlUnit/>, from  http://xmlunit.sourceforge.net/, is available in two forms: a Java framework (for use both with and without JUnit) and a less well-developed C# framework for use with NUnit. The Java framework can test assertions about XML documents (and even badly-formed HTML), validate documents, and compare two documents as well as test assertions about the result of XSLT transformations.

Tests are written as methods of Java (or C#) classes. The following abbreviated example from the <XmlUnit/> documentation shows a test where the result of an XSLT transformation is compared to the expected output.

public void testXSLTransformation() throws Exception {
    String myInputXML = "...";
    File myStylesheetFile = new File("...");
    Transform myTransform = new Transform(myInputXML, myStylesheetFile);
    String myExpectedOutputXML = "...";
    Diff myDiff = new Diff(myExpectedOutputXML, myTransform);
    assertTrue("XSL transformation worked as expected", myDiff.similar());
}

This example shows input and expected output coming from a String or a File. They may instead be a DOM node or a SAX InputSource?.

Unit Testing Framework – XSLT (UTF-X)

UTF-X, available from  http://utf-x.sourceforge.net/, is a Java-based framework where tests can be run from the command line, from an Ant build file, or using JUnit. Tests are written as XML. The following example from from the UTF-X web site asserts that the result of processing the content of the utfx:source element will match the content of the utfx:expected element.

<utfx:test>
  <utfx:name>sect1 with title only</utfx:name>
  <utfx:assert-equal>
    <utfx:source validate="yes">
      <section id="section1">
        <heading>Section 1</heading>
      </section>
    </utfx:source>
    <utfx:expected validate="yes">
      <a name="section1" />
      <h1>Section 1</h1>
    </utfx:expected>
  </utfx:assert-equal>
</utfx:test>

UTF-X includes a test generator that can generate a test definition file for an existing stylesheet. XTS

XTS, by Florent Georges, is available from  http://www.fgeorges.org/xslt/xslt-unit/. XTS is an XSLT 2.0 framework that is capable of testing both XSLT 2.0 and XQuery. A single test comprises an assertion of the expected result followed by a sequence constructor, as in the following example from the XTS web site that illustrates testing a function from the stylesheet under test:

<t:tests>
  <t:title>hello-world()</t:title>
  <t:test>
    <t:expect select="'Hello, world!'"/>
    <xsl:sequence select="hw:hello-world()"/>
  </t:test>
</t:tests>

The sequence constructor – a single xsl:sequence in this case, though it could be more complicated – is compared for equality with the expected result. Alternatively, the expected result could be written as an XPath expression to evaluate or, when used with Florent’s ‘error-safe’ extension for Saxon, an assertion of an error that should be thrown when evaluating the sequence constructor.

The XML file containing the tests is transformed into a test stylesheet that imports the stylesheet being tested. That stylesheet, when run, ignores its XML input and runs named templates corresponding to the tests in the original test XML file.

The output of the test stylesheet is an XML file that can be transformed into a HTML report. Figure 5 shows the sample report from the XTS web site.

xts report. Figure 5 – XTS report

XSLV Static Validation Tool

The XSLV tool for static validation of XSLT from the University of Aarhus, available online at  http://www.brics.dk/XSLV/, is able to check that all output of a stylesheet at runtime is valid according to a specified output schema, assuming that the input is valid according to its specified schema. Schemas can be written as DTD, XML Schema, or a subset of RELAX NG.

Since XSLT is Turing complete, determining validity for all possible stylesheets is undecidable, so the tool applies some approximations. However, the designers wanted to be able to guarantee correctness, so the approximations err on the safe side: the tool reports some valid output as being invalid, but should never report invalid output as valid.

I consider that this type of static validation complements rather than replaces unit testing, firstly since the stylesheet output is likely to be invalid from its inception up until it is largely complete, and secondly since its all too easy to create output that’s valid but still incorrect. …but how do I know I’m testing enough?

That is the role of coverage tools. I’m still looking for a coverage tool, and have plans to write my own.

…but how can I check my FO?

The open source XSL formatters are the best sources for tools for testing formatted documents. The xmlroff XSL formatter (  http://xmlroff.org/) has a framework for rasterising PDF or PostScript output and comparing formatted pages against a reference, and the FOP XSL formatter (  http://xmlgraphics.apache.org/fop/) has a framework for comparing the XML renditions of its area tree against a reference.

xmlroff Testing Module

This module serves multiple purposes for the xmlroff XSL formatter:

  • Comparing results between different versions of xmlroff
  • Comparing results between xmlroff’s different output formats
  • Comparing xmlroff output to files produced another way, e.g., from a different formatter – FO or otherwise

The testing module uses a combination of Makefile, Perl, and commonly available utilities to:

  • Run an XSL formatter on one or more test suites conforming to the format originally defined by the W3C XSL FO subgroup for XSL 1.0 Candidate Recommendation testing. The test definition for a single test is shown below:

<test id="expression1" fo="multiple" xml="inherited-property-value.fo" results="inherited-property-value.pdf">Use inherited-property-value() in expressions for font-size property, including using function with property other than font-size and using as part of a larger expression.</test>

  • Rasterise the PDF or PostScript output
  • Compare the rasterised output against a reference
  • Produce a report summarising all tests and an individual report for each test

xmlroff test report. Figure 6 – xmlroff individual test report

  • Produce a “stereo” image combining the result and reference for any page that is different from its reference version

Detail of xmlroff 'stereo' report. Figure 7 – Detail of “stereo” comparison, where the red channel is from the current result and the blue channel is from the reference

When the reports are served by a web server, the record of the test results – which also conforms to the format defined by the W3C XSL FO subgroup – can be updated to indicate whether the result agrees with the expected result or to add a comment about the test result. The test result for the example is shown below:

<testresult id="expression1" results="expression1.pdf" agreement="full" specproblem="no" testproblem="no"/>

FOP Layout Engine Tests

FOP’s layout engine tests are integrated with other JUnit tests that are run in the FOP build process. Each test includes a complete FO document, and assertions are made (and checked) about the FOP-specific area-tree that results from formatting the FO document with FOP.

The following is the example test document from  http://wiki.apache.org/xmlgraphics-fop/HowToCreateLayoutEngineTests:

<testcase>
  <info>
    <p>
      This test checks <something>.....
    </p>
  </info>
  <variables>
    <img>../../resources/images/bgimg300dpi.jpg</img>
  </variables>
  <fo>
    <fo:root xmlns:fo="http://www.w3.org/1999/XSL/Format" xmlns:svg="http://www.w3.org/2000/svg">
      <fo:layout-master-set>
        <!-- etc. etc. -->
        <fo:block-container background-image="##img">
        <!-- etc. etc. -->
    </fo:root>
  </fo>
  <checks>
    <eval expected="0 0 360000 360000" xpath="/areaTree/pageSequence/pageViewport/@bounds" desc="page size"/>
    <true xpath="/areaTree/pageSequence/pageViewport/page[1]"/>
    <true xpath="not(/areaTree/pageSequence/pageViewport/page[2])"/>
    <eval expected="0 0 360000 360000" xpath="/areaTree/pageSequence/pageViewport/page[1]/regionViewport/@rect" desc="region body area"/>
  </checks>
</testcase>

Bibliography

[MCC06]

Steve McConnell, Software Estimation: Demystifying the Black Art, ISBN 0-7356-0535-1, Microsoft Press, Redmond, Washington, 2006.