xmlroff-testing Package

Perl scripts and XSLT stylesheets for automating testing of xmlroff.

Testing xmlroff

xmlroff testing uses:

  • Test cases from the XSL 1.0 CR test suite, NIST, and Sun

  • XSLT stylesheet for making shell scripts to run xmlroff on the test cases

  • Perl script for running the test scripts and diffing the outputs

  • XSLT stylesheets for making summary and individual test result HTML files

Test Cases

The test cases have not yet been set up on the SourceForge system.

The XSL Working Group (WG) had to demonstrate that every feature of XSL had been implemented before XSL 1.0 advanced from Candidate Recommendation (CR) to Proposed Recommendation (PR).

In 2001, the WG and NIST (National Institute of Science and Technology in the U.S.) developed a DTD for both test suite descriptions and test suite results, and NIST and the XSL formatter implementors contributed both test suites and their descriptions.

When working on the XSL 1.0 CR test suite results with Max Froumentin of the W3C, we developed a simple, well-formed document type for referring to multiple test suite descriptions.

The XSL 1.0 CR test suite was put on the web. NIST also continued to develop tests.

The current test cases used for testing xmlroff are the XSL 1.0 CR test cases, an interim release of NIST's later test cases, and some test cases developed at Sun.

test2sh.xsl

This is run manually whenever the test cases are modified.

Usage:

   xsltproc --stringparam testsuites \
   ../../TestSuite/coverage/testsuites.xml test2sh.xsl test2sh.xsl

The testsuites parameter is the filename of the XML document referring to multiple test suite descriptions.

The stylesheet expects that the current directory will have a subdirectory with the same name as the subdirectory of each of the test suite description files referred to by the file referred to by the 'testsuites' parameter. I.e., if testsuites.xml refers to ../NIST/testsuite.xml, then test2sh.xsl expects to be able to write to a NIST subdirectory of the current directory.

In each subdirectory corresponding to the subdirectories of the referred-to test suite descriptions, test2sh.xsl writes a test.sh file containing the shell commands to run xmlroff on each test case described in the test suite description.

For each test case, test2sh.xsl writes shell commands to:

  • Run xmlroff on the test case input file or files, with output written to a PDF file named after the input file and error output written to a .err file also named after the input file.

  • Run ImageMagick's convert program on the generated PDF file to convert the PDF to PNG files, one PNG file for each page in the PDF.

  • Run ImageMagick's composite program with the 'stereo argument on each generated PNG file for which there is a corresponding PNG file in the 'ref' subdirectory. The output, which is written to a like-named file in the stereo subdirectory, shows the differences between the generated file and the reference version in the ref subdirectory.

The test.sh file also runs a simple command-line Perl script on the error files to remove process IDs from the error messages in the .err files so the files can be meaningfully diffed.

dotests.pl

This Perl script is run nightly as part of a cron job.

It executes the shell scripts created by test2sh.xsl.

This script is run from the directory containing the subdirectories with the test.sh shell scripts.

The Perl script scans the subdirectories of the current directory for test.sh files.

In each subdirectory containing a test.sh file, the Perl script:

  • Runs test.sh, with output written to test.out in the current subdirectory.

  • Following completion of test.sh, for each file in the ref subdirectory of the current subdirectory, the Perl script:

    • Diffs the file in ref with the same-named file in the current subdirectory, with output written to the diff subdirectory of the current subdirectory.

    • Outputs a message if the file in diff is not zero bytes.

  • If no differences were found, outputs a message saying so.

testsuccess.pl

This Perl script is run after dotests.pl.

It generates an XML file listing file names and file sizes of test result files that is used as input to the XSLT stylesheet that generates the HTML reports. Using this script saves having to worry about trying to find and use XSLT processor-specific extensions for accessing file details.

testsucess.pl scans the subdirectories containing the test results and outputs a well-formed XML document listing the file names and file sizes for all the files relating to each PDF file.

testresults.xml

This is the record of xmlroff's results for each test. It conforms to the DTD from the XSL WG.

This file should have a new entry added to it for every new test case added to the test suite.

testsuccess.xsl

This is run after testsuccess.pl has generated the XML file listing the test result file names and file sizes.

It generates HTML reports: one large report summarizing all the test cases and their results, plus an individual report for each test case.

Usage:

   xsltproc -o testsuccess.html --novalid \
   --stringparam testsuites /usr/local/src/TestSuite/coverage/testsuites.xml \
   --stringparam success testsuccess.xml \
   testsuccess.xsl testresults.xml

The summary report and the individual reports include information from both the test case description and the xmlroff test results.

The test case description information is obtained by following file references in the file referred to by the testsuites parameter to individual test suite decription files.

The xmlroff test results information is obtained from both testresults.xml and testsuccess.xsl.

The summary report is a large table with one row for each test case. Each row contains:

  • The test case ID.

  • An indication of the value of the test result attributes from testresults.xml for the test.

  • The test result's descriptive text from testresults.xml.

  • A link to the PDF output, if any.

  • A link to the report HTML for the individual test.

  • An indication of the number of output pages and the types of the result files present.

The individual reports contain:

  • The "path" to the test case

  • The test case information:

    • Test description

    • Which FO is being tested, which could be multiple

    • Input XML file, which could be an FO file

    • Input XSLT stylesheet, if any

    • Sample result file, if any provided by test case author

  • The test result information, as an HTML form so the results can be updated from the individual report:

    • Whether or not the test result agrees with the test case description and/or sample results

    • Whether or not the XSL formatter will in the future support the FOs and properties used in the test

    • Whether or not any differences between the test result and the test case description and/or sample results indicates a problem with interpreting the XSL 1.0 Recommendation

    • Whether or not any differences between the test result and the test case description and/or sample results indicates a problem with the test.

    • Any further textual description

  • For each page in the test result PDF:

    • The PNG for the page

    • If there are differences between the current PNG and the corresponding ref PNG:

      • The PNG for the page from the ref subdirectory

      • The PNG for the stereo composite showing the differences between the two PNGs

    • If, instead, there are no differences between the current PNG and the corresponding ref PNG:

      • text stating Reference is identical to current.

Selecting the 'Update' button in the 'Result' form both updates the test result information in testresults.xml and copies the current PNG files to the ref subdirectory. Unfortunately, it does not update the form because the current XSLT stylesheet generates both the summary report and all the individual reports, all of which takes a long time. Ideally the summary report and the individual reports should be XHTML that can be simply modified, not regenerated, whenever a test's results are updated.

The summary report should also be broken into multiple HTML files, one or more for each test suite, but that also hasn't been done yet.

update-results.pl and update-result.xsl

update-results.pl is the CGI script called when the 'Update' button in an individual report is selected. It modifies testresults.xml using the update-result.xsl stylesheet, copies the current test result's PNG files to its ref subdirectory, zeroes the corresponding files in the diff subdirectory, and removes the corresponding files in the stereo subdirectory.


DocBook Libxslt SourceForge