W3C home > Mailing lists > Public > public-wai-ert-tsdtf@w3.org > February 2009

Re: Test samples with multiple techniques

From: Christophe Strobbe <christophe.strobbe@esat.kuleuven.be>
Date: Tue, 24 Feb 2009 12:57:01 +0100
Message-Id: <>
To: TSDTF <public-wai-ert-tsdtf@w3.org>

At 19:34 17/02/2009, Christophe Strobbe wrote:
>I did a quick search to find test samples with more than one technique.
>In the 3rd BenToWeb test suite, we had at least 27 of these, for example:
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc1.2.3_l2_004
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc1.4.7_l3_200
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc2.2.2_l2_001
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc2.3.1_l1_001
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc2.4.5_l2_001
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc2.4.6_l2_010
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc3.1.3_l3_001
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc3.3.1_l1_004
>(three techniques)
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc3.3.3_l2_001
>(three techniques)
>(three techniques)
>* http://www.bentoweb.org/ts/XHTML1_TestSuite3/metadata/sc4.1.1_l1_012
>All fail/pass statements were based on the success criterion, not 
>the test procedure in the technique/failure.

And in the TSD TF repository:

* sc1.2.3_l2_022 (G9 G93)
* sc1.4.7_l3_200 (G146 C14)
* sc2.3.1_l1_001 (G15 G19)
* sc2.3.1_l1_003 (G15 G19)
* sc2.3.1_l1_004 (G15 G19)
* sc2.3.1_l1_005 (G15 G19)
* sc2.3.1_l1_008 (G15 G19)
* sc2.3.1_l1_009 (G15 G19)
* sc3.3.1_l1_018 (G83 SCR18 G139)
* sc3.3.1_l1_019 (G83 G85 SCR18 G139)
* sc3.3.1_l1_020 (G83 G85 SCR18 G139)
* sc3.3.1_l1_021 (G83 G85 SCR18 G139)
* sc3.3.1_l1_022 (G83 G85 SCR18 G139)
* sc3.3.1_l1_023 (G83 G85 SCR18 G139)
* sc3.3.1_l1_024 (G83 G85 SCR18 G139)
* sc3.3.1_l1_026 (G83 G85 SCR18 G139)
* sc3.3.1_l1_027 (G83 G85 SCR18 G139)
* sc3.3.1_l1_028 (G83 G85 SCR18 G139)
* sc3.3.1_l1_029 (G83 G85 SCR18 G139)
* sc3.3.1_l1_030 (G83 G85 SCR18 G139)
* sc3.3.1_l1_031 (G83 G85 SCR18 G139)
* sc3.3.1_l1_032 (G83 G85 SCR18 G139)
* sc3.3.1_l1_033 (G83 G85 SCR18 G139)
* sc3.3.1_l1_034 (G83 G85 SCR18 G139)
* sc3.3.2_l2_004 (G83 G84 G85 G139 SCR18)
* sc3.3.4_l2_200 (G131 G89 H44)
* sc3.3.4_l2_201 (G131 H44)

(Note that not all BenToWeb test cases have been migrated to the TSD 
TF repository, so there could be more of these.)

It is possible that some of the BenToWeb test case authors were 
somewhat too generous with relevant techniques.
However, this does not explain why we have we have 27 test samples 
with more than one technique.
E.g. sc3.3.1_l1_026 is about a mandatory text input field with error 
correction. SC 3.3.1 (Error Identification) says: If an input error 
is automatically detected, the item that is in error is identified 
and the error is described to the user in text. sc3.3.1_l1_026 
references the following techniques:
* G83: Providing text descriptions to identify required fields that 
were not completed
* G85: Providing a text description when user input falls outside the 
required format or values
* SCR18: Providing client-side validation and alert
* G139: Creating a mechanism that allows users to jump to errors

If we changed this test sample in order to map to only one technique, 
e.g., SCR18: Providing client-side validation and alert, would we 
then still make sure that the test sample meets SC 3.3.1?
If the answer is yes, there may be no problem. But if the answer is 
no, how does this affect the test sample? If the test sample only 
passes the test procedure in SCR18, we can no longer state that it 
passes SC 3.3.1, and we will need to build in a kind of disclaimer 
about this. (This would be a lot of extra work, unless we do this 
systematically for every test case, in which case we can automate 
it.) Such test samples would not be very useful as examples of good practice.

Best regards,


Christophe Strobbe
K.U.Leuven - Dept. of Electrical Engineering - SCD
Research Group on Document Architectures
Kasteelpark Arenberg 10 bus 2442
B-3001 Leuven-Heverlee
tel: +32 16 32 85 51
Please don't invite me to LinkedIn, Facebook, Quechup or other 
"social networks". You may have agreed to their "privacy policy", but 
I haven't.

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
Received on Tuesday, 24 February 2009 11:57:46 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:06:02 UTC