Re[2]: CR exit criteria

Monday, July 31, 2006, 10:20:28 AM, Lofton Henderson wrote:

> See embedded...

> At 11:55 AM 7/31/2006 +0200, Chris Lilley wrote:

>>On Friday, July 28, 2006, 11:04:28 PM, Benoit wrote:
>>
>>BB> Hi,
>>
>>BB>   I've been thinking about the CR exit criteria discussion. Here's my
>>BB>   opinion on it.
>>
>>BB>   First, I think Chris' request is reasonable (that is two
>>BB>   successful passes for each 1.0 and 2.0 tests).
>>
>>To be clear, I'm asking for testing 2.0. Some of the tests to do that are 
>>currently 1.0 tests. I assume that they would need to be identified as 
>>WebCGM 2.0 files,though.

> I have been thinking about this question.

> One way to identify them as 1.0 files is to put them in a place that says,
> "all of these 1.0 tests are valid 2.0 tests".  The other way is to change
> the "1.0" to "2.0" in the MetDesc element in the CGM, to also make that
> change in the graphical text that labels and appears in the legend block of
> the CGM, and regenerate all of the PNG files.

> I have done something like the latter before, when adapting tests from ATA
> to WebCGM.  It's a bit of work.

> While the latter is more appealing from the perspective of "goodness" of
> the test suite, and ought to be done eventually, on the other hand it is
> somewhat "cosmetic".  I would like to think that, for now, we can live with
> the former for CR interoperability purposes ("two pass").

> Thoughts?
I'm ok with the former.

> One more...

>>BB>  There is however a
>>BB>   down side to it, and that is it could slow us down in our progress
>>BB>   to Rec.
>>
>>BB>   Is there middle ground that can be reached? Probably.
>>
>>BB>   I think we would have to agree that no new 1.0 tests can be created.
>>BB>   Dealing with the existing one is plenty for now. If the CGM Open TC
>>BB>   wants to create more tests later; that's up to them, but from a W3C
>>BB>   perspective, we are only dealing with existing 1.0 tests. Ok?
>>
>>That was my proposal, yes.
>>
>>
>>BB>   Also, I don't think the working group should be trying to regroup
>>BB>   the two test suites into a single one. That would be wasted cycles
>>BB>   (in my opinion).
>>BB>
>>BB>   Creating the matrix itself wouldn't take much time, the name of each
>>BB>   test is available in the ICS pro-forma. This is mostly copy/paste
>>BB>   work.
>>
>>BB>   What is time consuming is if vendors provide inaccurate results;
>>
>>Right.
>>
>>BB>   this can't happen. Also each vendor would have to be able to provide
>>BB>   beta versions of their product for someone like Chris to verify the
>>BB>   results.
>>
>>Perhaps we could do some of that at the f2f meeting?

> That's a thought.

> Unfortunately, two of the implementors won't be there (unless we
> invited them, which from QAWG experience is apparently legitimate
> for a "public" group; or maybe we could get a block of Zakim time
> and join them in if there is need for discussion with them.)
Would they be willing to send a beta version to you and/or Chris?

> Btw, the reason I objected earlier to a test by test, vender by
> vender verification of all results, as opposed to spot-check or
> focus on reported problematic tests... Does it make a statement
> about our view of the self-reporting by venders? Does anyone else in
> W3C do this? Do we thereby start to establish new criteria for the
> two-pass convention, "independently verified interop data"? (Plus
> ... it would dump on Chris about 1000 individual test claim
> verifications: 4*250. I'm sure he wouldn't mind that in his spare
> time :-) )
I can only comment on what I've seen in the SVG Working Group.
Basically, it's not in the vendors best interest to claim a PASS if
the actual result is a FAIL. To have an independent reviewer is useful
for those tests that 'unfortunately' require some interpretation.

> -Lofton.

>>BB>  If we get a commitment from all the vendors to provide
>>BB>   prompt and accurate results for each tests, it may be doable in
>>BB>   relatively little time; if that's not the case, I'm afraid we'd be
>>BB>   stuck in CR for a long time.
>>
>>BB>   Thoughts on this?
>>
>>--
>>  Chris Lilley                    mailto:chris@w3.org
>>  Interaction Domain Leader
>>  Co-Chair, W3C SVG Working Group
>>  W3C Graphics Activity Lead
>>  Co-Chair, W3C Hypertext CG

-- 
Regards,
 Benoit   mailto:benoit@itedo.com

This e-mail and any attachments are confidential and may be protected
by legal privilege. If you are not the intended recipient, be aware
that any disclosure, copying, distribution or use of this e-mail or
any attachment is prohibited. If you have received this e-mail in
error, please notify us immediately by returning it to the sender and
delete this copy from your system. Thank you for your cooperation.

Received on Monday, 31 July 2006 14:55:06 UTC