W3C home > Mailing lists > Public > public-html-testsuite@w3.org > February 2011

Re: Automated Test Runner

From: L. David Baron <dbaron@dbaron.org>
Date: Thu, 17 Feb 2011 15:23:33 -0800
To: James Graham <jgraham@opera.com>
Cc: Kris Krueger <krisk@microsoft.com>, Anne van Kesteren <annevk@opera.com>, "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>, "Jonas Sicking (jonas@sicking.cc)" <jonas@sicking.cc>
Message-ID: <20110217232333.GA25092@pickering.dbaron.org>
On Tuesday 2010-11-16 11:21 +0100, James Graham wrote:
> FWIW I have some plans in this area. I even have a little code, but
> it doesn't do anything useful yet :) (I also note that Ms2ger does
> have some code that does do something useful).
> 
> As part of my plan, I would like to add per-directory metadata to
> the test system. I think this has the advantage over global metadata
> that it is closer to the tests and so more likely to be kept up to
> date when tests change. In particular I would expect it to be owned
> by the test owner rather than someone coordinating the testsuite as
> a whole. It has the advantage over per-file metadata that it doesn't
> affect the test itself. In particular I propose having a json
> manifest file with a well-known name like "manifest.json" in each
> directory containing tests. The file would have a structure like
> (missing some syntax for ease of reading):
> 
> {tests:{"001.html":{type:"javascript",
>                     flags:["SVG"],
>                     expected_results:10,
>                     top_level_browsing_context:false
>                    }
>        },
>  subdirs: ["more_tests"]
> }
> 
> type is "javascript", "reftest" or "manual"
> flags indicates specific optional features required by the test or
> other unusual dependencies
> expected_results (missing default: 1) indicates the number of tests
> in that file
> top_level_browsing_context (missing default: false) indicates that
> the test needs to run in a top level browsing context (e.g. for
> testing window.top)
> 
> subdirs is a list of subdirectories in the current directory that
> should be checked for tests.
> 
> Does this sound reasonable? Did I miss anything obvious?

[ following up on a somewhat old thread here ]

I don't see why this is needed, and it's a extra work to maintain,
especially if people are contributing tests written elsewhere.

The number of tests isn't important (and is not a good measure of
testing coverage); what matters is whether any of them failed.

-David

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/
Received on Thursday, 17 February 2011 23:24:20 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 17 February 2011 23:24:20 GMT