Re: Documentation update

On Jan 16, 2012, at 7:28 AM, Robin Berjon wrote:

> On Jan 11, 2012, at 17:38 , Linss, Peter wrote:
>> Specifically, the title should be more than the file name
> 
> Right, but a lot of the data I'm seeing at this point uses the file name. I've called this out as bad practice.
> 
>> the assertion should not be the name of the test, but rather a list of the assertions in the specification tested.
>> Also, a planned augmentation is to allow the spec links to point to any anchor in the spec to allow narrower targeting of testable assertions. this will go hand in hand with additional markup in the spec to identify those assertions in an automated way.
> 
> Yes, I was thinking about markup as used in the test meth document. I'm guessing that instead of assertions there could be assertion-links (or we could overload and guess).

Fantasai and Alan Stearns have had a long discussion about how best to markup specs in a manner that testable assertions can be found by automated tools. I'm not sure if what they came up with has been documented anywhere yet.

> 
>> The test suite build code is at: 
>> http://hg.csswg.org/dev/w3ctestlib/
>> 
>> There isn't a comprehensive list of work for it yet (except a collection of notes on my machine and a bunch in my head), but I did set up a dev tracker for it (and the other test framework projects) at:
>> http://dev.csswg.org/
>> 
>> It's not really populated yet, but I've started entering a few items here and there.
>> 
>> The build code it also tightly coupled to the Shepherd project which uses it to extract (and eventually manipulate) the test metadata. I don't think it's at a point where it makes sense to hand off much of the work yet, I think we'd be stepping on each others toes too much. But hopefully soon.
> 
> Indeed, I've tried to get the above working but I've so far given up at the point where I've had to patch the already patched specific version of html5lib in order to sacrifice the right goats that make it not completely hate Unicode strings :)

Yeah… that part of (actually most of) the build system code predated my involvement so I'm not entirely sure of all the issues with html5lib and the required patches. IIRC there was some effort to get those changes folded back into the trunk but I don't think that ended well...

> 
> I don't need much to make WebApps's test suites importable so I'll just start with a simple throwaway script that generates a manifest from the metadata you describe. We'll replace that when the above is more more easily sharable.
> 
>> A good first step to be able to use the build system on other test suites is more likely to adapt the format of the tests to that expected by the build system (documented in the link above).
> 
> Right, that was my thinking. I'll document the generic version. I don't expect the differences to be huge (mostly documentation-related in some cases), but there ought to be some.
> 
>> The fun part is going to be extracting metadata from script tests (which will require executing the script in the build process).
> 
> A lot of the script tests I've seen to date depend on a companion HTML file which serves as the entry point and so the metadata would presumably be there. Supporting pure JS tests does not necessarily require executing them — we could simply extract metadata from a special comment (e.g. starting with /** for instance).

The problem is that for many scripted tests the tests themselves are generated by code rather than simply listed in any file. So there's no complete listing of tests available to be parsed out in any form except running JavaScript code. We had a few meetings about this at TPAC and the best approach we could come up with so far is to allow code to execute at least to the point that all the tests are constructed. Listing the tests would be best from my perspective, but this apparently puts too much of a burden on test authors (and also allows the metadata to easily get out of sync with the tests a run, which would be quite bad).

> 
>> A planned change to the build system is to be able to build multiple test suites at once and use the specification links in the meta data to automatically place tests (and their related support files) into the proper test suite(s).
> 
> Is this useful outside of CSS? With the WebApps and DAP TSs in mind, I would expect them to mostly if not always have simple 1-1 mappings between suite and spec.

Most test suites will likely have a simple 1:1 mapping, but we really need this for CSS suites, so I'm going to do it anyway. It will be useful for any WG that modularizes their specs, or for tests that test features from multiple specs.

> 
>> Yes, we do. My current plan is to back port the work you an Mike have done to my codebase, then back port some of the work from the Shepherd project into the harness (the use the same core library), mostly the login code (I'm also going to integrate the LDPA support in a cleaner fashion) and UI cleanups. then you can re-sync your codebase from mine. After that I plan on improving the auto-submit code for scripted tests.
> 
> Ok, please keep us posted here since some of that is also stuff that I will be working on (assuming you don't get there first), notably autosubmit and UI.

Will do.

Received on Monday, 16 January 2012 19:25:18 UTC