Re: CR exit criteria

Okay, here is a first-cut at composite ICS:
http://www.w3.org/Graphics/WebCGM/WG/Group/2006/draft-ics-10-composite.html

This is based on results reported by 4 implementers in Apr-May 2003.  I did 
a little adjustment of the reported results of ITEDO -- they gave the 
reasons for "No" answers so I was able to check against present Beta.

Btw, I was thinking more about the need for automation in a big (250+) 1.0 
matrix.  That might not be necessary.  I can automatically create an 
initial version with "Ok" in all slots.  I would anticipate that each 
vender only has a small handful of "Fail" or "Partial", and that could 
easily be hand edited and maintained.

-Lofton.

P.S. I didn't know where was a good place to put this.  At first I thought 
it was best in member space for now -- it is a quick and unverified work in 
progress.  If you have a better idea (esp. Thierry), feel free to suggest 
or to relocate it.


At 04:44 PM 7/28/2006 -0600, Lofton Henderson wrote:

>At 05:04 PM 7/28/2006 -0400, Benoit Bezaire wrote:
>
>>Hi,
>>
>>   I've been thinking about the CR exit criteria discussion. Here's my
>>   opinion on it.
>>
>>   First, I think Chris' request is reasonable (that is two
>>   successful passes for each 1.0 and 2.0 tests).
>
>I won't dispute "reasonable".  But as I expressed in telecon, there are 
>equally reasonable alternative views of it.  Effectively, the 
>functionality of the 1.0 subset of 2.0 is being subject to higher quality 
>criteria to be reaffirmed in 2006, than were applied to accept it as 
>Recommendation in 1999 and 2001.  The appropriateness of that can be 
>debated (W3C Process is silent about such specifics).
>
>>There is however a
>>   down side to it, and that is it could slow us down in our progress
>>   to Rec.
>
>And it is this potential that worries me.  If the suggested quality 
>criteria for the 1.0 subset had occurred to us earlier, we could have been 
>working on this for the last year or so, in parallel with developing the 
>new 2.0 tests.  (In which case it wouldn't be an issue.)
>
>
>>   Is there middle ground that can be reached? Probably.
>>
>>   I think we would have to agree that no new 1.0 tests can be created.
>>   Dealing with the existing one is plenty for now. If the CGM Open TC
>>   wants to create more tests later; that's up to them, but from a W3C
>>   perspective, we are only dealing with existing 1.0 tests. Ok?
>>
>>   Also, I don't think the working group should be trying to regroup
>>   the two test suites into a single one. That would be wasted cycles
>>   (in my opinion).
>
>I don't understand this comment, "regroup the two suites into a single 
>one".  Explain?
>
>>
>>   Creating the matrix itself wouldn't take much time, the name of each
>>   test is available in the ICS pro-forma. This is mostly copy/paste
>>   work.
>
>Trivial.  (Actually, I can generate the *empty* new matrix easily from a 
>list of test names, which I have.)
>
>However, once we go from 40 tests to 250+ tests, it is questionable 
>whether manual construction and maintenance of the new matrix (editing an 
>HTML table) is practical or reliable.  Some automation is likely 
>necessary.  E.g., vendors submit their results in a rigorous format -- 
>even a careful XHTML table would suffice -- that can be merged 
>automatically by XSLT.
>
>I can write such XSLT, but haven't done so yet because ... too lazy, and 
>too hard to get people to adopt rigor in submitting results, so that 
>hand-editing 40 row HTML table has been easier, especially with few and 
>infrequent changes.
>
>
>>   What is time consuming is if vendors provide inaccurate results;
>>   this can't happen. Also each vendor would have to be able to provide
>>   beta versions of their product for someone like Chris to verify the
>>   results.
>
>I object to that suggestion, unless you mean "spot check".  I don't have a 
>problem with spot-check.  But we should NOT require a full verification of 
>results, if that's what you mean.  That is inappropriate (IMO), and 
>without precedent (AFAIK).   (Not to mention more work and delay.)
>
>>If we get a commitment from all the vendors to provide
>>   prompt and accurate results for each tests, it may be doable in
>>   relatively little time; if that's not the case, I'm afraid we'd be
>>   stuck in CR for a long time.
>>
>>   Thoughts on this?
>
>No strong opinion.  I'm slightly pessimistic about anything involving 5 
>vendors/implementations and 250+ (1.0) tests being done quickly.
>
>For now, I'm preceding with the "lumpy" composite table, i.e., the one at 
>the granularity of the ICS.  I'll have something soon.  (And possibly the 
>issue will become moot, except for the labor to do the new 1.0-subset table.)
>
>-Lofton.
>
>
>
>
>

Received on Sunday, 30 July 2006 16:52:20 UTC