Conformance testing

David Marston

As part of the OASIS XSLT/XPath Conformance Testing project, we are cataloging all aspects of processor behavior that are allowed to vary at the discretion of the processor's development team. We also have identified dozens of gray areas in the spec and passed them to the W3C Working Group for creation of errata. As a byproduct, the test suite could probably be used to identify the subset of the XSLT syntax that is "portable" in the sense that all conformant processors must behave the same way.

Francis Norton made a remark about a lint-like utility for stylesheets that has inspired some discussion. I think anything that passed cleanly would be unrealistically constrained. To summarize, stylesheet portability would require a tactic for each of the following:

In my OASIS activities, I've been concerned with a strict definition of conformance, because failing a conformance test would be a bad thing for any processor. Thus, we cannot enforce "should" verbiage in the specs, but a more "realistic" lint might do so. We'll examine each of the above 3 topics in that regard after dealing with one other notion.

Let OASIS give you the test suite for conformance, and you can pressure the XSLT vendors to make their respective processors fully conformant because you want your fully portable stylesheet to work on their processor. Write stylesheets to the spec, then let them bring their processor up to spec. That's the whole reason we have vendor-neutral standards. Furthermore, all the authors of XSLT books want to explain the aspects that are common to all XSLT processors, serving as guides for writing portable stylesheets.

Okay, now let's deal with each form of variation that may occur within fully-conformant XSLT processors.

1. Most of the broad grants of discretion ("should" verbiage) appear in Chapter 16 of the XSLT spec, concerning output. There are minor(?) potential variations in 2.2-2.3 (version numbers), 7.7.1 (xsl:number language and character-set support), 10 (sorting of text data in various languages), 12.1 (document() function awareness of URI schemes). XPath 3.6 has a minor grant of discretion regarding string equivalence. If you relax the portability criterion to say that a stylesheet is considered portable if it expects that processors do everything the specs say they "should" do as well as what they "must" do, then your stylesheet can at least deal with serialized output as opposed to DOM-tree output. In other words, you can use the xsl:output element and disable-output-escaping and avoid processors that refuse to generate a file or character stream according to the "should" verbiage in Chapter 16. You'd still need to know which processors support numbering and sorting in all the languages that matter to you, and if you pass "unusual" URIs to document(), you'd need to know which processors support those protocols.

2. Specific grants of discretion typically say something like "An XSLT processor may signal the error; if it does not signal the error, it must recover by..." followed by one or two very specific recovery actions. (There are only two when talking about encoding; it can pick either UTF-8 or UTF-16.) The OASIS committee is nearly finished with its inventory and naming of all such items, numbering about 50. You may be tempted to insist that a portable stylesheet avoid all these areas, but one of them is template conflict resolution in XSLT 5.5. Do you really want to assign an explicit priority to every single (match) template, AND guarantee that no two of the same priority will ever otherwise conflict? A more realistic goal is to know what decisions were made by each vendor, and steer away from the "signal an error" scenario on all the items that are not true errors of stylesheet writing. Your portable stylesheet should not commit any of the errors where the non-signal choice is to discard the potential output entirely. The OASIS test cataloging system will help in further analysis of these behavior variations. Test cases for these variations are still "conformance" tests, but they depend on vendor information about the design decisions they made. (If a vendor says they chose to raise an error, we apply the test that expects an error; if they say they chose to ignore the token, we apply the test that expects the token to be missing on output while processing continues, etc.)

3. Gray areas in the specs are unintentional, and sincere people can disagree about whether certain verbiage is too vague or not. One area that the Working Group apparently found too vague is the original XSLT verbiage about xsl:number being given negative numbers, zero, NaN, and Infinity for alphabetic or Roman-numeral formatting. They issued an errata that tightens up the spec, although in the process they granted specific error/ignore discretion in a new place. Not all XSLT vendors have upgraded their processors to conform to the errata. Does your portable stylesheet have to assume the worst, that the processor isn't up-to-date (but still conforms to the original XSLT 1.0)? What if vendor A insists that the spec is perfectly clear in some area where vendor B says there's a gray area? I think that a portable stylesheet could assume that the processor implements a "common sense" approach to each gray area, because that's generally what the errata will later require.

The above squeezes away various exceptions and details. I wanted it to serve as a scoping vehicle for the notion of assessing stylesheet portability. To see more about the OASIS test suite effort, go to http://www.oasis-open.org/committees/xslt/index.shtml and watch that page for a catalog of discretionary items. If anyone really wants to write XStyLinT(TM), please consider joining the OASIS Technical Committee because you'll be deeply involved in the relevant issues!

G. Ken Holman

There is a vendor-neutral organization in the process of producing a test suite that will be used by authorities who wish to provide certification services. OASIS has technical membership involved in the development of tests, but is not interested in the provision of a service of the certification of processors. Those interested in such services will be welcome to utilize the fully publicly available completed work of the committee (anticipated Spring 2001).

Please see Oasis for more details.

Our mail list has been archived where you can see what we've been up to.