Re: Framework next steps

On Jul 22, 2011, at 6:50 AM, Philippe Le Hegaret wrote:

> Mike, Francois,
> 
> Some thoughts on the new framework.
> 
> now that the framework is deployed, we need to start improving and
> documenting along the way as well.
> 
> Improvement:
> 
> Looking at
> http://w3c-test.org/framework/suite/nav-timing-default/
> 
> Mike has been missing some metadata. I added some on
> http://w3c-test.org/webperf/tests/
> but it would be good to start documenting how we want to feed the
> database.
> 
> Looking at
> http://w3c-test.org/framework/test/nav-timing-default/test_document_open/

My suggestion for test metadata is to adopt (or adapt) the metadata elements used by the CSS test suite:
http://wiki.csswg.org/test/css2.1/format

While the test framework doesn't read this metadata directly from the test files, we have other tools that do and generate the import manifests for the test framework as well as test suite index pages. We also have a test suite management system under development that uses (and will be able to edit) that metadata (something like a Bugzilla tool but with tight mercurial integration).

For cases where the metadata can't be stored within the test, it could be in the same format in a parallel file (i.e.: test_name.meta)

I'm happy to adjust the CSS test format (to some extent) to bring it into line with a generic system suitable for other groups.

Peter

> 
> I see a number of issues:
> 
> - this is a script test and should be ran automatically by default
> - link to reference page is not necessary
> - number of tests reported: the test itself is problematic since it
> doesn't report how many tests it contains and are failing. It contains
> 21 tests in fact. At the minimum, we need to tweak the test.
> - we need to start documenting (in the public wiki) how to write tests
> for the framework, starting with script tests and reftests.
> 
> Philippe
> 
> 
>

Received on Friday, 22 July 2011 18:04:38 UTC