Re: CR exit criteria

See embedded...

At 11:55 AM 7/31/2006 +0200, Chris Lilley wrote:

>On Friday, July 28, 2006, 11:04:28 PM, Benoit wrote:
>
>BB> Hi,
>
>BB>   I've been thinking about the CR exit criteria discussion. Here's my
>BB>   opinion on it.
>
>BB>   First, I think Chris' request is reasonable (that is two
>BB>   successful passes for each 1.0 and 2.0 tests).
>
>To be clear, I'm asking for testing 2.0. Some of the tests to do that are 
>currently 1.0 tests. I assume that they would need to be identified as 
>WebCGM 2.0 files,though.

I have been thinking about this question.

One way to identify them as 1.0 files is to put them in a place that says, 
"all of these 1.0 tests are valid 2.0 tests".  The other way is to change 
the "1.0" to "2.0" in the MetDesc element in the CGM, to also make that 
change in the graphical text that labels and appears in the legend block of 
the CGM, and regenerate all of the PNG files.

I have done something like the latter before, when adapting tests from ATA 
to WebCGM.  It's a bit of work.

While the latter is more appealing from the perspective of "goodness" of 
the test suite, and ought to be done eventually, on the other hand it is 
somewhat "cosmetic".  I would like to think that, for now, we can live with 
the former for CR interoperability purposes ("two pass").

Thoughts?

One more...


>BB>  There is however a
>BB>   down side to it, and that is it could slow us down in our progress
>BB>   to Rec.
>
>BB>   Is there middle ground that can be reached? Probably.
>
>BB>   I think we would have to agree that no new 1.0 tests can be created.
>BB>   Dealing with the existing one is plenty for now. If the CGM Open TC
>BB>   wants to create more tests later; that's up to them, but from a W3C
>BB>   perspective, we are only dealing with existing 1.0 tests. Ok?
>
>That was my proposal, yes.
>
>
>BB>   Also, I don't think the working group should be trying to regroup
>BB>   the two test suites into a single one. That would be wasted cycles
>BB>   (in my opinion).
>BB>
>BB>   Creating the matrix itself wouldn't take much time, the name of each
>BB>   test is available in the ICS pro-forma. This is mostly copy/paste
>BB>   work.
>
>BB>   What is time consuming is if vendors provide inaccurate results;
>
>Right.
>
>BB>   this can't happen. Also each vendor would have to be able to provide
>BB>   beta versions of their product for someone like Chris to verify the
>BB>   results.
>
>Perhaps we could do some of that at the f2f meeting?

That's a thought.

Unfortunately, two of the implementors won't be there (unless we invited 
them, which from QAWG experience is apparently legitimate for a "public" 
group;  or maybe we could get a block of Zakim time and join them in if 
there is need for discussion with them.)

Btw, the reason I objected earlier to a test by test, vender by vender 
verification of all results, as opposed to spot-check or focus on reported 
problematic tests...  Does it make a statement about our view of the 
self-reporting by venders?  Does anyone else in W3C do this?  Do we thereby 
start to establish new criteria for the two-pass convention, "independently 
verified interop data"?    (Plus ... it would dump on Chris about 1000 
individual test claim verifications:  4*250.  I'm sure he wouldn't mind 
that in his spare time  :-)   )

-Lofton.


>BB>  If we get a commitment from all the vendors to provide
>BB>   prompt and accurate results for each tests, it may be doable in
>BB>   relatively little time; if that's not the case, I'm afraid we'd be
>BB>   stuck in CR for a long time.
>
>BB>   Thoughts on this?
>
>
>
>
>
>--
>  Chris Lilley                    mailto:chris@w3.org
>  Interaction Domain Leader
>  Co-Chair, W3C SVG Working Group
>  W3C Graphics Activity Lead
>  Co-Chair, W3C Hypertext CG

Received on Monday, 31 July 2006 14:20:29 UTC