Re: requirements for test sample management system


At 15:38 30/09/2008, Tim Boland wrote:

>The W3C Mobile Web Activity has a mobile test harness [1], which may 
>be relevant to our discussions on this topic.

The test harness, and especially the way a reviewer checks test 
results (i.e. you get output in red or green or compare something 
against a reference image), reminds me of an important difference 
between the WCAG 2.0 test suites I've worked on and most of the W3C 
test suites I've seen:
most W3C test suites move along a single axis: the list of features 
of a technology;
a test suite for WCAG 2 has two such axes: the accessibility 
requirements on the one hand, and the features of a technology 
(sometimes multiple technologies, like HTML and JavaScript) on the other.[1]

This difference has influenced the design of the test samples, the 
metadata format, and the processes and tools around these things.

[1] (That is also one of the reasons why reviewing test samples 
during the conference call isn't the quick and dirty process that 
Michael would like it to be; the other reason is that we need more 
concrete review recommendations that the group can quickly accept or reject.)

Best regards,


>Also there was a W3C QAWG Test FAQ document [2], which may contain 
>information to stimulate further discussions on this topic.
>The Matrix of W3C Specifications [3], although older, may contain 
>links to other development work which may be of interest.
>Thanks and best wishes
>Tim Boland NIST
>At 12:48 PM 9/29/2008 +0200, you wrote:
>>I have an (unrecorded) action item to write up draft requirements 
>>for a test sample management system. This has become more urgent 
>>because of the migration of the wiki.
>>Here's a list of requirements:
>>* an overview (which I will call "status list" from here on) of all 
>>the test samples; this overview should show:
>>   - test sample ID,
>>   - a link to the HTML version of the XML metadata,
>>   - links to the structure reviews (possibly with identification 
>> of the reviewers or reviewers),
>>   - links to the content reveiws (possibly with identification of 
>> the reviewers or reviewers),
>>   - the status of the test sample,
>>   - the submitter: organization (if any) and contact information 
>> (e-mail is sufficient),
>>   - links to issues that need to be fed back to WCAG WG.
>>   (This mirrors what we have at 
>> <>.)
>>   It should also be possible to filter and sort this overview by 
>> at least ID and status.
>>   Nice to have in the status list:
>>     - filtering based on "features" used in the test samples,
>>     - filtering based on techniques and failures covered by the 
>> test samples.
>>   (The status list at <> has some sorting 
>> options but no filtering.)
>>* an HTML view of the metadata of each test sample;
>>   see the links in the column "test sample ID" at 
>> <>.
>>   - This HTML view also contains links to both the metadata file 
>> and the HTML file or files.
>>   - For the sake of greater automation of the structure reviews, 
>> this view should also link
>>     to validation reports (or funcationality to generate them) for 
>> both XSD validation and
>>     ISO Schematron validation.
>>   - The section "relevant technique/failure" should also provide 
>> the title of the technique
>>     or failure instead of only the identifier.
>>   - Links to WBS questionnaires where the test sample was surveyed.
>>Notes regarding the status list:
>>- The XML metadata contains an element for status 
>>   but reviewers have only edited the status in the wiki. Ideally, 
>> status is tracked in the XML
>>   metadata and the status list pulls the status from the XML, so 
>> that the status list does not
>>   get out of sync with the XML metadata.
>>- The status list in the wiki contains several tables: one for the 
>>current WCAG 2 draft,
>>   "the previous table" (for test samples that map to the May 2007 
>> draft and test samples
>>   for the April 2006 draft which still needed to be updated to a 
>> more recent draft), and
>>   a table with "obsolete test samples" (test samples with the 
>> status "deprecated" because there
>>   was no matching technique or failure, and JSP test samples).
>>   So we would also need to be able to filter and sort the test 
>> samples by WCAG 2 draft.
>>What we already have:
>>- XSLT to generate HTML view of each test sample's XML metadata.
>>- Ant script (using more XSLT) to generate test sample status list.
>>- XSD for the XML metadata (TCDL 2.0; see RDDL file at 
>>   but is there a PHP implementation?
>>   (I don't see this at 
>> <>
>>- ISO Schematron (see how-to doc at 
>>   but there seems to be no PHP implementation. Maybe we can use 
>> the Amara XML Toolkit
>>   (<>), an open-source 
>> collection of Python tools that
>>   has an ISO Schematron implementation. Amara 1.2a2 requires 
>> Python 2.4 and 4Suite XML 1.0
>>   (<>), which is also 
>> open source.
>>   Alternatively, we can just rely on the XSTL implementation 
>> (discussed in my how-to doc) and
>>   PHP's XSLT support: 
>> <>.
>>   If we choose the option of using the XSLT produced by the XSLT 
>> implementation
>>   (e.g. to use it in PHP code), we need to take into account that 
>> there is currently no stylesheet
>>   that produces an HTML report at 
>> <> or at
>>   <>.
>>This is just a first draft. Please comment on the list.
>>We can then put the requirements on a page under 
>>Best regards,
>>Christophe Strobbe

Christophe Strobbe
K.U.Leuven - Dept. of Electrical Engineering - SCD
Research Group on Document Architectures
Kasteelpark Arenberg 10 bus 2442
B-3001 Leuven-Heverlee
tel: +32 16 32 85 51
Please don't invite me to LinkedIn, Facebook, Quechup or other 
"social networks". You may have agreed to their "privacy policy", but 
I haven't.


Received on Tuesday, 30 September 2008 14:08:47 UTC