RE: draft review of TSDTF test samples assigned to me - my action item

Hi Tim,

Some inline comments about the first review, but I think that applicable to all of them:

> GENERAL COMMENTS:
> 
> (1) Need a way to handle versioning of WCAG drafts and techniques
>   in the metadata -  the techniques links in the metadata are dated
> from the May WCAG draft, even though the December draft has been published
> and the
> WCAG techniques links have been updated?
>
> (2) Need a way to give the title of the SC, not just the number, in the
> metadata, because the number may change between successive WCAG drafts due
> to
> addition/removal of SCs between versions, so the referenced number may
> become
> incorrect.  I know, because I recently updated the WCAG1-WCAG2 mapping,
> and
> found that older SC numbers had changed, or were deleted, from earlier
> WCAG
> versions

I would also add that we also need an updating plan for keeping metadata consistence each time WCAG is updated (also related to the next point).
 
> (3) there needs to be some mention of what software is needed to correctly
> play
> the files in the testfiles.  One of the criteria is that the testfiles
> "work correctly",
> but that is partially dependent upon the users' environment and installed
> software?
> What is the test environment for these tests?

That's a good point.

> (4) In the structure review following, does the "test files" portion refer
> just to the
> actual test file, or to both the metadata file and the test file?  The
> distinction can be
> confusing - for example, there are links in both the metadata file and the
> test file, so
> the checklist for "links working correctly" can be applied to both
> files?  Similarly, in
> the "metadata" portion, the checklists for "titles being accurate" can be
> applied to both
> metadata file and test files?    Should the labelling of the structure
> review match progression
> through first the metadata file and then the associated test file (if not
> already done)?

My understanding is that the "test files" section is intended to refer only to the actual test files (not metadata) and the "metadata" is intended to refer only the metadata file.

> (5) The naming convention for these files is "l1", etc., for levels of the
> WCAG SCs, but WCAG SCs
> now use levels A, AA, AAA, so this may be confusing?

Don't think soo, they are just files names and I think that xxxx_lAA_xxxx is not going to be much more significative.

Additionally I think l1, l2 and l3 is also valid as we have 3 levels that are named whatever. As a final point I don't think we are at time of changing the naming convention.

> ---------------------------------------
> 
> SPECIFIC REVIEWS:
> 
> ----------------------------------------
> 
> Structure Review for Test Sample sc1.2.1_l1_001
> 
> Contact Information
> 
> Review Criteria - name and email address of the submitter are available
> Review Result - fail?
> Comment - I could not find it in the metadata file?

They are at the Test Sample Status List
[http://www.w3.org/2006/tsdtf/TestSampleStatusList]

We may need to clarify this in the structure review process
Is there a need to have them in the metadata file?

> Review Criteria - organization on whose behalf the test sample was
> submitted
> Review Result - pass
> Comment - I found it in the metadata file (ERTWG?)

This information is also in the Test Simple Status List.
Until now the submitter is always BTW, not ERTWG.

>[...]
>   the actual testfile
> "sc1.2.1_l1_001.html" contains "html" as the file type (but the "primary"
> technology is xhtml from the
> doctype?) - what should the relationship be between file type and "primary
> (what does that mean)"
> technology??
> In terms of directory structure, the metadata file seems to comply (the
> "xhtml/metadata" part)
> but are subdirectories allowed in this structure?  The actual test file
> has
> "html" listed as a file type
> this seems OK, but what does this imply in term of "primary technology"
> (doctype is "xhtml1/strict")?

Not sure what you mean with "primary technology". I don't remember we use this terminology anywhere.

>    The "xhtml/testfiles" part of the path seems OK, but
> then there is a subdirectory "resources/video.." which seems inconsistent
> with the listing under
> "directory structure"?  Also, under "testfiles" in the process document,
> there are "resource" subdirectories,
> but after "testfiles" in this case there is the actual files" - is this
> inconsistent (should there
> be a "resource" part before the actual file for consistency)?

I think that the only inconsistence here is that the "video" subdirectory is directly under the "testfiles" one, and the rest of resources (applets, audio, flash...) are under the "resources" directory.

According to the "Directory Structure" section [http://www.w3.org/WAI/ER/tests/usingTCDL#structure] of the "Using TCDL" document is the rest of the world (applets, audio, flash...) and not video who are in the wrong place.
 
> Review Criteria - all the files include valid markup unless otherwise
> required by the test
> Review Result - fail?
> Comments - metadata file validates as "well formed XML (1 warning)"
> according to the W3C validator,not all the files validate according to the
> W3C validator (may
> need to document the exceptions?).  The actual testfile fails validation
> with 4 errors (according
> to the W3C validator).

IMO It's difficult here to say if the validation errors are required by the test or not, as the test is about captions.
 
> Review Criteria - all the files include correct links unless otherwise
> required by the test
> Review Result - cannot tell?
> Comments - need to check all the links?  Checked a few in the metadata
> file, and they seemed OK,
> but need to check all the schema links for correctness..  What is the
> definition of a "correct link"?

I think we need to check all of them (although several may be done in an automatic way)

My interpretation of correct link is one that is not broken (unless it's required by the test purpose) and get to what is expected at first.

> Video in testfile seemed to play OK for me, and I got the sound OK ..
> but some of the buttons were "grayed out" (last four)
> - also should there be a "test purpose" somehow included in the html file
> -
> there were no captions
> in the video file for me but someone watching may forget what the metadata
> file says.

It supposed that metadata is somehow part of the test, so you shouldn't ignore it. Anyway the plan is also to expose some of the metadata.
 
> Review Criteria - all the files include correct spelling unless otherwise
> required by the test
> Review Result - pass?
> Comments - could not find any spelling errors, but again, what is
> definition of "correct spelling"?
> Which dictionary is being used?

I wouldn't complicate too much things regarding spelling. I vote just for an "obvious misspelling" criterion.

> Metadata
> 
> Review Criteria - all the dates and other integer or literal values have
> the correct format
> Review Result - cannot tell?
> Comments - what is the definition of "correct format"?

We have discussed this before. It seems obvious that we should include this information directly in the documentation.
 
[...]
> Review Criteria - all titles, descriptions, and other required fields are
> included and accurate
> Review Result - cannot tell
> Comments - what is definition of "accurate"?  There is a "title" tag in
> the
> metadata file, which
> accurately says "a video with no captions", but in the actual testfile
> there is a "title" tag that
> says "prerecorded multimedia with captions", which does not seem accurate
> and is a contradiction
> with the title tag in the metadata (I did not notice any captions in the
> video when played)...

I also noted that there's a certain grade of inconsistence between metadata titles and the test sample titles, and I agree than in some cases this inconsistence may be relevant.

May consistence between both titles be required?

[...]
>   the only id attribute I found in the metadata file was for the technique
> tag, which
> contained "F8" (I think there should be a technique description as well -
> see my earlier comments -
> rather than just "F8", because if techniques change "F8" may no longer be
> correct, and it may be
> difficult to determine correctness in the future - document management
> issue)...
> Also the "may"
> reference is no longer correct, because there is now a "december"
> reference.. No id attributes
> found in html testfile..  Same goes for the "F8" reference after
> "location"
> tag under "rule" element previous - there
> is also a "may" reference there.. the "F8" here has no path to give it
> context.

That's true, but I think our main issue here is been "condemned" to work with such unstable references. I think that it's really useful and make sense to have SC and Techniques solid references, is there anything the WCAG WG can do on the subject?

>[...]
> What happens
> if my environment doesn't support "wmv"?

Good point!
Additionally, what about the copyright of the videos, how can we be sure about it?

  I compared metadata in metadata
> file against metadata in
> process document..  Two "technical specs" sections - is the "testelement"
> section still correct in
> light of recent WCAG evolution?  

It was there waiting for the "Baseline concept" evolution. Maybe it's time to take a final decision on this (IMO it doesn't make sense any more due to the evolution that the Baseline concept has experimented).

>The "complexity" attribute on the
> "testcase" element - is that
> still important in light of recent telecon discussion on that
> attribute? 

No, it shouldn't.

> In the process document,
> it says that required "file" element MUST contain one of four "optional"
> choices (wording seems
> confusing - even though none of the options is included in this example,
> wording in metadata document
> seems confusing - what happens to MUST?)

What do you mean with "the process document"?
I think that both, the metadata document [http://www.w3.org/WAI/ER/tests/usingTCDL] and the TCDL specification read that the file element CAN contain one of four "optional" choices. 

That's all as far as I am concerned.
Regards,
 CI.

__________________

Carlos Iglesias

Fundación CTIC
Parque Científico-Tecnológico de Gijón
33203 - Gijón, Asturias, España

teléfono: +34 984291212
fax: +34 984390612
email: carlos.iglesias@fundacionctic.org
URL: http://www.fundacionctic.org

Received on Monday, 4 February 2008 19:02:19 UTC