Re: Various Test/Spec questions from WCAG

At 09:55 AM 2/17/04 +0900, olivier Thereaux wrote:
>Hello QAWG,

Hi Olivier,


>As you know, the WCAG working group is taking QA seriously and have
>been trying to follow our guidelines as closely as possible. I usually
>try to answer question from Jenae Andershonis (QA contact within WCAG
>WG), but she recently sent me a batch for which I could use your
>insight,

These are only my own interpretations -- no QAWG standing (would QAWG 
consensus answers be a good idea?).

>and I thought you'd be interested in the feedback they
>provide, too...
>
>Specl GL:
>>I have a question about Checkpoint 7.1. Use conformance key words. The
>>latest structure for WCAG 2.0 is Principle | Guideline | Level (1, 2
>>or 3) Success Criteria.

I'm looking at:

[0] http://www.w3.org/TR/WCAG20/#perceivable

I see some Success Criteria.  But I don't see Levels, Principles, etc.  So 
I assume I'm not looking at "the latest structure".

>>My question is at what level should the RFC 2119 keywords be applied?
>>Should the keywords be used in the statement for Principle, Guideline
>>or Success Criteria? Or at all three? My guess is that it should be at
>>the Success Criteria level.
>
>So do I. What do you think?

However, looking at that /TR/ version ... yes, the RFC keywords would 
potentially be applicable within the Success Criteria.  EXCEPT ... WCAG has 
chosen a different style to express its basic testable statements.  They 
have chosen a conventional "test assertion" (TA) style.  RFC keywords would 
not be applicable unless they changed to a (SpecGL-like) "conformance 
requirements" style.  E.g.,

"1.   non-text content that can be expressed in words has a text-equivalent 
explicitly associated with it."

would become,

"1.   non-text content that can be expressed in words MUST have a 
text-equivalent explicitly associated with it".

Note that CP7.1 allows the current WCAG style (TAs), as long as the style 
is explained and the reason for using it instead of RFC keywords is 
explained.  (E.g., "less fierce" or "don't shout" or ...).

Aside.  As a result of the CR trial implementation and review period for 
SpecGL and OpsGL, there are newly re-opened issues, challenging the 
RFC-keywords style should be so strongly preferred in all cases.  My 
personal recommendation (for which QAWG may lynch me!) ... don't change 
this aspect of WCAG20, at least not until QAWG has had the chance to sort 
this issue again.


>Test GL (based on TR version)

Btw, we hope to publish (/TR/) a new version of TestGL next week, and I 
think it will be much changed from the current /TR/ version that Jenea is 
using (dated May 2003) .  The new version's most recent editor-draft 
ancestor is:

[1] http://www.w3.org/QA/WG/2003/10/TestGL-20031020.html

>>Question #1
>>Checkpoint 4.3. Automation of testing is encouraged.
>>
>>We weren't sure if this meant that the test cases should be
>>automatable, (i.e. doesn't involve a human opinion). Or if it means
>>that we should engage with vendors of tools like A-Prompt and Bobby,
>>or does it mean something even different than these two
>>interpretations.
>
>Not sure what Jenae means by "engaging" here, but my interpretation of
>the GL is that the tests should be automatable when possible. (which,
>for WCAG, is not always/necessarily/easily possible)

I'm not sure I understand the question exactly.  So I'll say some things 
and see if anything hits a chord.

I see two possible applications of automation:

1.) the test case itself is automated (doesn't involve human intervention);

2.) the test *framework* is automated.

I think that the /TR/ version of TestGL CP4.3 is about automation of the 
framework, #2.  I.e., even if the individual test case involves human 
judgement and intervention (WCAG rule assessment, VBWG subjective test, 
OpsGL CP conformance), there can be an automated framework (e.g., harness) 
that sequences the tests, presents the tester with a nice form-driven 
questionnaire, assembles the results, and reports "the final score".

We were talking about this very thing, this morning, in QAWG telecon, 
considering "what are Test Materials for GL documents like OpsGL?"

Aside.  The W3C Markup Validator is a pure example of #1.  As I understand 
Bobby, it also is a #1-ish of test tool (i.e., it's a WCAG checker, 
right?  Reads the document and tries to analyze it from perspective of WCAG 
rules?).


>>Question #2
>>Checkpoint 6.1. Organize conformance testing activities.
>>Would the definition of vendor be like Watchfire (Bobby tool)?
>
>I think this is an interesting one... AFAIK vendors here mean
>implementors of the specification, which does not necessarily mean
>anything for Guideline specifications... A wording warning for the QAF,
>I suppose.

Interesting question.  Let's see, the Class of Product (CoP) of WCAG -- 
i.e., the object or target of its conformance requirements -- is Web 
documents.  Therefore, an "implementation" would be an instance of a Web 
document?  And a "vendor of an implementation" would be an author/owner of 
Web document instance(s)?

I think that's a sound chain of inference.  That said, might one generalize 
"author" to authoring tool?  Yes, I know that there is an ATAG.  I'm not 
thinking about all of the aspects of an authoring tool that ATAG 
addresses.  I'm thinking of a single aspect -- the WCAG-compliance of the 
documents it produces.

Yes ... we should look at QAF and make sure that the "content" CoP (which 
is WCAG's CoP) fits into all of our GL & CP about testing, test materials, etc.

By the way, these test-activity checkpoints of that 5/2003 TestGL draft 
have be moved into OpsGL:

[2] 
http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/guidelines-chapter#Ck-publish-test-results


>No "vendor" entry in the QA glossary it seems:
>http://www.w3.org/2003/glossary/keyword/qa-glossary.rdf/?keywords=vendor

*Lots* of stuff missing.  WebOnt and JC have made some serious comments 
about that.


>>Question #3
>>Checkpoint 6.2. Encourage Vendors to publish test results.
>>Is the preferred "Special place" within the W3C site or a link to the
>>vendor's site?
>>And should we create something like the 508 VPAT for the vendor to
>>complete?
>
>IIRC, for test result publication, W3C is preferred, though not
>strongly. Correct?

Yes.  See [2].

508 VPAT?  Not sure what that is.  Sounds like a standardized results 
reporting form?  This brings to mind SpecGL's Implementation Conformance 
Statement (ICS), remembering that "implementation" here is "Web page instance":

[3] 
http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/guidelines-chapter#Ck-require-ICS


>>Question #4
>>
>>Our proposed charter [2] states that we would commit to attaining QA
>>level three. Now that the conformance structure has been changed to A,
>>AA and AAA, what would level three equate to? And do we need to list
>>out of conformance goals for all three of the QA specs.
>>
>>[2] http://www.w3.org/2003/08/wcag-charter.html
>
>I think this has been simplified and corrected in Ops, correct?

Yes.  In the LC version of OpsGL, that table bore no relationship to the 
requirements of the rest of the QAF family, and it was heavily 
criticized.  Therefore we revised it to:  tell us how you intend to conform 
to OpsGL, to SpecGL, and to TestGL.

>http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/guidelines-chapter#Gd-address-QA-in-charter
>I do not, however, recall what level three would be equivalent to (A or
>AA I guess), could anyone tell me?

Although the current GL1 checkpoints are simpler and clearer, it isn't a 
simple mapping between them and the old LC version of OpsGL.  Old Level 3 
required, for example, a set of Test Assertions.  That is P3 in SpecGL 
(hence only needed for AAA SpecGL conformance), and P1/P2 in TestGL (so 
needed for A/AA TestGL conformance).



>Hope this list of question provides you with interesting feedback, and
>that you can help me give good answers to this very motivated group.

Again ... would it be a good idea to generate a WG consensus response?

-Lofton.

Received on Wednesday, 18 February 2004 20:03:10 UTC