Input to automatic generation of test running dashboard

Hi all,

we discussed today the compilation of test results, see
http://www.w3.org/2012/12/17-mlw-lt-minutes.html#item03

In case it might help, below is an overview how we did that for ITS 1.0:

1) have a matster file that lists all output files, with a unique 
relation to an input file and to an implementation
http://www.w3.org/International/its/tests/input-file-list.xml
see e.g.
<inputFile name="Translate1.xml" datacat="translate" 
relatedTest="translate-global-embedded-rules">
  <resultFile location="test1/Translate1-result.xml" 
implementation="test1"/>
  <resultFile location="test2/Translate1-result.xml" 
implementation="test2"/>
  <resultFile location="test3/Translate1-result.xml" 
implementation="test3"/>
</inputFile>

2) have a process
http://www.w3.org/International/its/tests/compare.xsl
that runs through input-file-list.xml and for each result file does a 
comparison with the expected output
<xsl:call-template name="compareResult">
  <xsl:with-param name="expectedResult" as="item()">
   <xsl:copy-of 
select="doc(concat('expected/',substring-before(../@name,'.xml'),'-result.xml'))"/>
  </xsl:with-param>
  <xsl:with-param name="actualResult" as="item()">
  <xsl:copy-of select="doc(string(@location))"/>
  </xsl:with-param>
  <xsl:with-param name="implname">
  <xsl:value-of select="@implementation"/>
</xsl:with-param>
  <xsl:with-param name="inputfilename"><xsl:value-of 
select="../@name"/></xsl:with-param>
  <xsl:with-param name="relatedTest"><xsl:value-of 
select="../@relatedTest"/></xsl:with-param>
</xsl:call-template>

3) the result of running that process is at
http://tinyurl.com/its10-testsuite-dashboard
and also linked from
http://www.w3.org/International/its/tests/
(see the "click here" link)
then there is an additional stylsheet
http://www.w3.org/International/its/tests/compareResultHtml.xsl
that generates the HTML table used in
http://www.w3.org/International/its/tests/

The a bit tricky part of the process was in
http://www.w3.org/International/its/tests/compare.xsl
since we had the XML based output format and had to do a comparison of 
the XML files. With the text based format in the ITS 2.0 test suite a 
simple text based comparison should do it. But a file listing input from 
the various implementors like
http://www.w3.org/International/its/tests/input-file-list.xml
might be hepful. One could then auto-generate from this also the main 
test suite page
https://github.com/finnle/ITS-2.0-Testsuite/blob/master/its2.0/its-testsuite.html

Just some thoughts.

Best,

Felix

Received on Monday, 17 December 2012 21:11:06 UTC