Re: Linking test suites to specs

On Jun 27, 2013, at 3:15 PM, Rebecca Hauck wrote:

> 
> On 6/27/13 2:40 PM, "Tab Atkins Jr." <jackalmage@gmail.com> wrote:
> 
>> On Thu, Jun 27, 2013 at 2:06 PM, Rebecca Hauck <rhauck@adobe.com> wrote:
>>> This never made it on to the already full agenda of yesterday's meeting
>>> &
>>> I'm not even sure it requires that precious time.  If no one objects to
>>> me
>>> making this proposed change to the spec header, I'll proceed
>>> (approximately
>>> this time tomorrow).
>>> 
>>> To reiterate, the changes I'll be making are:
>>> 
>>> Add the Test Suite section to all of the specs ­ those with no tests
>>> labeled
>>> "none yet"
>>> Link to the test suite cover page per fantasai's suggestion
>> 
>> What'll the markup look like?  Either a "none yet" string, or a link
>> to the test suite cover page?  What text should the link contain -
>> just the link itself, or something else?
> 
> Yes, either "none yet" or a link to the cover page. For example:
> 
> http://test.csswg.org/suites/css3-flexbox/nightly-unstable
> 
> And just the link is fine.
> 
> However, these test suite links have versions in them - is that going to
> be problematic? Input from Peter is welcome here as I'm not super familiar
> (yet) with these things are built and labeled.

The 'nightly-unstable' suites are simply the result of the nightly build and contain whatever was present in the repository at the time.

The dated test suites are snapshots capturing a specific state of the test suite, generally the copy that was used to meet the CR exit criteria of the spec and are saved for posterity's sake (and are linked from the CR implementation reports). 

For some suites, like CSS2.1, we had a number of 'Release Candidate' snapshots prior to exiting CR. Mostly those are from before we had nightly builds, but also, due to the size of the suite, we had a lot of submissions from people running the suites outside the harness and submitting implementation reports. The snapshots were a way for the implementation reports to be linked to the versions of the tests (the harness tracks results per test revision).

Given that once we exit CR the goal of the test suite shifts from spec testing to conformance testing, it's probably best for specs to always point to the nightly build.

Peter

Received on Friday, 28 June 2013 01:29:51 UTC