Re: Documentation update

On Jan 10, 2012, at 7:46 AM, Robin Berjon wrote:

> Hi all,
> 
> I've taken a first stab at writing up the documentation aimed at test suite maintainers using the Test Framework. It should eventually land at:
> 
>    http://w3c-test.org/framework/docs/maintainer/
> 
> whenever the cron job syncs it, but in the meantime it's at:
> 
>    http://dvcs.w3.org/hg/testframework/raw-file/tip/docs/maintainer/index.html
> 
> It probably needs a few more iterations before it's really usable — in order to get there your feedback is very much welcome. Note that I would really appreciate this being double-checked, the content there is mostly based on reading the code and toying with it a little — and my brain's PHP implementation is only so good.
> 
> There are a few things of specific note:
> 
>    • I'm not sure what those fields are for test suite manifests: references, revision, credits.

The references field is for reftests. It's a comma separated list of the names of the references that the test must be compared against. If the reference name begins with a '!' then it's a mismatch reference (i.e. its rendering must NOT match the testcase).

The revision field is an id of the revision of the testcase. This is generally the revision number or changeset id of the most recent commit of the file into it's canonical source repository. For reference tests or tests that have support files (stylesheets, images, etc.), it should be the most recent revision of the test or any of its related files. This field is important because all results are stored by the revision of the test active at the time. When tests are revised the proper results are associated with each revision. It's also possible to have different revisions of a test active in different test suites at the same time.

The credits field is for the name and contact info of the test author(s). I don't believe it's currently used by the harness but the purpose is to be able to present this data to the user. It's a comma separated list of name and url pairs, e.g. 'Peter Linss <mailto:peter.linss@hp.com>, Bob Smith <http://www.smith.com/Bob>'

Note that the test suite build system from the CSS test suite automatically generates this manifest file with all the data in place. At some point this code will be generalized to be used for other suites as well.

Also note that the CLI test case import script has a comparison feature. If you re-import a test suite and supply the paths to the previous version as well as the new version, it will perform a normalized comparison of the test files whose revisions have changed. This will build a revision equivalence table for tests that have non-normative differences only (e.g. metadata changes that don't impact the test). This way previously stored results can be shared among different test revisions where the test itself is functionally unchanged.

The CSS test suite uses the CLI import script and the comparison feature for the nightly build/import cycle.

> 
>    • Specifications can be imported from the CLI using a spec manifest — is this exposed to the Web interface? (I don't think so).

I don't believe so. I wrote all the CLI import scripts, Mike wrote the Web interface versions. 

> Should it be?

Possibly, or the web interface should be expanded to include the functionality.

> Is this how multipage specifications are supposed to be handled? Right now I haven't documented this aspect.

Yes, the CLI version can handle multipage specs.

> 
>    • The test runs I could find seem to necessitate manual intervention (you see each test one by one, submit the result you see). Yet I see a number of mentions of automated test suite submission. Is this something that's planned for but not supported, or did I miss something? Is there agreement on how it ought to work?

There's some experimental support for automatic submission of script tests that use test harness.js. This will be expanded on. At some point there may also be automatic comparison of reference tests.

Received on Tuesday, 10 January 2012 18:55:03 UTC