Design of the Window test suite (was Re: Window 1.0 test suite)

Hi Ian,

On Mar 20, 2006, at 5:44 AM, Ian Davis wrote:

> Maciej,
>
> The test suite is a great piece of work.  I have question and a  
> suggestion: in the test suite you have the following directory  
> structure:

[... snip ...]

> Some of the code in ecmascript/browsing-contexts is repeated  
> underthe publish/shared/browsing-contexts directory. Which is the  
> canonical form? I'm guessing it's the one in ecmascript. The  
> publish/shared directory contains some test harness code too which  
> is cool.

Let me try to explain the design a bit. The only files that are truly  
source files are the ecmascript/ directory. The files in scripts/ are  
used to generate the published copies in publish/. And a handful of  
shared helper files are in publish/shared/.

When you write a new test, all you have to do is add a .js file to  
ecmascript that includes a mixture of JS code and test assertions.  
Then you run the scripts/generate-tests script. For those following  
along at home, here's an example of what one of the test files looks  
like:

--------------

description(
'This test checks testable assertions related to the <code>window</ 
code> attribute of the <code>Window</code> interface'
);

// Each view of a Document presented in a browsing context MUST be
// represented by an object that implements the Window interface.

// Window objects MUST implement the following attributes:
// ... window

shouldNotThrow("window");
shouldBeFalse("typeof window == 'undefined'");
shouldBeFalse("typeof window == 'null'");

// The window attribute MUST be a reference to the same Window object  
that
// has the attribute (a self-reference).

shouldBeTrue("window == this");
shouldBe("window", "this");
shouldBeTrue("window.window == window");
shouldBe("window.window", "window");

--------------

Here's an example of how to mix code with assertions, a hypothetical  
test for the ability to shadow the global window property with a var  
declaration:

---------

var window = 3;
shouldBe("window", "3");

---------

I'll write up some docs for all the test assertion functions available.


This script generates the published versions of the test as follows:

- For each source .js file, copies it to an appropriate subdirectory  
of publish/shared, appending the line "var successfullyParsed =  
true;" at the end. This extra statement is so that the harness can  
ensure tests don't accidentally appear to pass due to a parse error  
throwing an exception midway.

- For each source .js file, generates a wrapper based on each of the  
template files. Presently there are TEMPLATE.html, TEMPLATE.xhtml,  
and TEMPLATE.svg. Given a test ecmascript/category/testname.js and a  
template file named TEMPLATE.fooml, the generator makes a wrapper  
called publish/fooml/category/testname.fooml.

> The html, svg and xhtml directories play the role of test harnesses  
> for various ecmascript hosts. Presumably we'd accept contributions  
> from people who want to write harnesses for other bindings and  
> hosts. Is there a better structure to help with this?

I think the structure is already pretty good for this. Someone  
wanting to provide a harness for, say, VoiceXML, would only need to  
implement scripts/TEMPLATE.voicexml and (probably) publish/shared/js- 
test-pre-voicexml.js.

Things will start to get a bit more complicated once we start testing  
compound documents (by reference), in that case we'll need more  
templates to test the various kinds of embedding allowed. I imagine  
we'll want html-only, xhtml-only and svg-only versions where they  
compound only with themselves, but probably also all the other  
pairwise combinations of one language embedding another by reference.  
For these three languages, that makes 6 combos (since we want to test  
embedding both ways, so XHTML referencing SVG will be a separate test  
from SVG referencing XHTML).

> Perhaps the first subdirectory of the test suite should be the  
> binding with subdirectories for the harnesses and actual tests.  
> Maybe something like:
>
> /WindowTestSuite
>   ecmascript
>     harnesses
>       shared
>       html
>       svg
>       xhtml
>     tests
>       browsing-contexts
>
> The test assertion framework should come out of this tree into a  
> separate area for reuse by other components e.g. XHR. I've already  
> rewritten my XHR method tests to use it so we're consistent. I'd  
> like to be consistent on the directory structure too.

Sure, I'm happy to break out the generator scripts and shared helper  
files into a separate module. Suggested names? Maybe ReTest? :-)

Here's the directory structure I'd propose:

ReTest
     bin
         retest
     templates
         TEMPLATE.*
     resources
         # all the JS and CSS files that will be referenced by templates

WindowTestSuite
     source
         ecmascript
             browsing-contexts
             # etc
     publish
         shared
             browsing-contexts
             # etc
         html
             browsing-contexts
             # etc
         svg
             browsing-contexts
             # etc
         xhtml
             browsing-contexts
             # etc

The test generator script would end up copying some of its resources  
into the publish directory, but I think that's ok. It means we'll  
have just one place to update these for possibly multiple test  
suites, which can then be regenerated and remain self-contained.

For this test framework to be effective for testing XMLHttpRequest,  
though, I think it will need a few additional features. In  
particular, you'll want the ability to defer the final processing  
that adds the TEST COMPLETED message until a later time.

I already have a design for how to do this, since tests for timers  
and embedding will require this as well. The short version is that  
there will be a waitForTestCompleted() function that makes the test  
wait until you call testCompleted() to finish.

Regards,
Maciej