Re: CR exit criteria

At 05:04 PM 7/28/2006 -0400, Benoit Bezaire wrote:

>Hi,
>
>   I've been thinking about the CR exit criteria discussion. Here's my
>   opinion on it.
>
>   First, I think Chris' request is reasonable (that is two
>   successful passes for each 1.0 and 2.0 tests).

I won't dispute "reasonable".  But as I expressed in telecon, there are 
equally reasonable alternative views of it.  Effectively, the functionality 
of the 1.0 subset of 2.0 is being subject to higher quality criteria to be 
reaffirmed in 2006, than were applied to accept it as Recommendation in 
1999 and 2001.  The appropriateness of that can be debated (W3C Process is 
silent about such specifics).

>There is however a
>   down side to it, and that is it could slow us down in our progress
>   to Rec.

And it is this potential that worries me.  If the suggested quality 
criteria for the 1.0 subset had occurred to us earlier, we could have been 
working on this for the last year or so, in parallel with developing the 
new 2.0 tests.  (In which case it wouldn't be an issue.)


>   Is there middle ground that can be reached? Probably.
>
>   I think we would have to agree that no new 1.0 tests can be created.
>   Dealing with the existing one is plenty for now. If the CGM Open TC
>   wants to create more tests later; that's up to them, but from a W3C
>   perspective, we are only dealing with existing 1.0 tests. Ok?
>
>   Also, I don't think the working group should be trying to regroup
>   the two test suites into a single one. That would be wasted cycles
>   (in my opinion).

I don't understand this comment, "regroup the two suites into a single 
one".  Explain?

>
>   Creating the matrix itself wouldn't take much time, the name of each
>   test is available in the ICS pro-forma. This is mostly copy/paste
>   work.

Trivial.  (Actually, I can generate the *empty* new matrix easily from a 
list of test names, which I have.)

However, once we go from 40 tests to 250+ tests, it is questionable whether 
manual construction and maintenance of the new matrix (editing an HTML 
table) is practical or reliable.  Some automation is likely 
necessary.  E.g., vendors submit their results in a rigorous format -- even 
a careful XHTML table would suffice -- that can be merged automatically by 
XSLT.

I can write such XSLT, but haven't done so yet because ... too lazy, and 
too hard to get people to adopt rigor in submitting results, so that 
hand-editing 40 row HTML table has been easier, especially with few and 
infrequent changes.


>   What is time consuming is if vendors provide inaccurate results;
>   this can't happen. Also each vendor would have to be able to provide
>   beta versions of their product for someone like Chris to verify the
>   results.

I object to that suggestion, unless you mean "spot check".  I don't have a 
problem with spot-check.  But we should NOT require a full verification of 
results, if that's what you mean.  That is inappropriate (IMO), and without 
precedent (AFAIK).   (Not to mention more work and delay.)

>If we get a commitment from all the vendors to provide
>   prompt and accurate results for each tests, it may be doable in
>   relatively little time; if that's not the case, I'm afraid we'd be
>   stuck in CR for a long time.
>
>   Thoughts on this?

No strong opinion.  I'm slightly pessimistic about anything involving 5 
vendors/implementations and 250+ (1.0) tests being done quickly.

For now, I'm preceding with the "lumpy" composite table, i.e., the one at 
the granularity of the ICS.  I'll have something soon.  (And possibly the 
issue will become moot, except for the labor to do the new 1.0-subset table.)

-Lofton.

Received on Friday, 28 July 2006 22:44:35 UTC