Re: Documentation update

On Jan 11, 2012, at 17:38 , Linss, Peter wrote:
> Specifically, the title should be more than the file name

Right, but a lot of the data I'm seeing at this point uses the file name. I've called this out as bad practice.

> the assertion should not be the name of the test, but rather a list of the assertions in the specification tested.
> Also, a planned augmentation is to allow the spec links to point to any anchor in the spec to allow narrower targeting of testable assertions. this will go hand in hand with additional markup in the spec to identify those assertions in an automated way.

Yes, I was thinking about markup as used in the test meth document. I'm guessing that instead of assertions there could be assertion-links (or we could overload and guess).

> The test suite build code is at: 
> http://hg.csswg.org/dev/w3ctestlib/
> 
> There isn't a comprehensive list of work for it yet (except a collection of notes on my machine and a bunch in my head), but I did set up a dev tracker for it (and the other test framework projects) at:
> http://dev.csswg.org/
> 
> It's not really populated yet, but I've started entering a few items here and there.
> 
> The build code it also tightly coupled to the Shepherd project which uses it to extract (and eventually manipulate) the test metadata. I don't think it's at a point where it makes sense to hand off much of the work yet, I think we'd be stepping on each others toes too much. But hopefully soon.

Indeed, I've tried to get the above working but I've so far given up at the point where I've had to patch the already patched specific version of html5lib in order to sacrifice the right goats that make it not completely hate Unicode strings :)

I don't need much to make WebApps's test suites importable so I'll just start with a simple throwaway script that generates a manifest from the metadata you describe. We'll replace that when the above is more more easily sharable.

> A good first step to be able to use the build system on other test suites is more likely to adapt the format of the tests to that expected by the build system (documented in the link above).

Right, that was my thinking. I'll document the generic version. I don't expect the differences to be huge (mostly documentation-related in some cases), but there ought to be some.

> The fun part is going to be extracting metadata from script tests (which will require executing the script in the build process).

A lot of the script tests I've seen to date depend on a companion HTML file which serves as the entry point and so the metadata would presumably be there. Supporting pure JS tests does not necessarily require executing them — we could simply extract metadata from a special comment (e.g. starting with /** for instance).

> A planned change to the build system is to be able to build multiple test suites at once and use the specification links in the meta data to automatically place tests (and their related support files) into the proper test suite(s).

Is this useful outside of CSS? With the WebApps and DAP TSs in mind, I would expect them to mostly if not always have simple 1-1 mappings between suite and spec.

> Yes, we do. My current plan is to back port the work you an Mike have done to my codebase, then back port some of the work from the Shepherd project into the harness (the use the same core library), mostly the login code (I'm also going to integrate the LDPA support in a cleaner fashion) and UI cleanups. then you can re-sync your codebase from mine. After that I plan on improving the auto-submit code for scripted tests.

Ok, please keep us posted here since some of that is also stuff that I will be working on (assuming you don't get there first), notably autosubmit and UI.

Thanks!

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Monday, 16 January 2012 15:29:06 UTC