- From: L. David Baron <dbaron@dbaron.org>
- Date: Thu, 17 Feb 2011 15:23:33 -0800
- To: James Graham <jgraham@opera.com>
- Cc: Kris Krueger <krisk@microsoft.com>, Anne van Kesteren <annevk@opera.com>, "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>, "Jonas Sicking (jonas@sicking.cc)" <jonas@sicking.cc>
On Tuesday 2010-11-16 11:21 +0100, James Graham wrote:
> FWIW I have some plans in this area. I even have a little code, but
> it doesn't do anything useful yet :) (I also note that Ms2ger does
> have some code that does do something useful).
>
> As part of my plan, I would like to add per-directory metadata to
> the test system. I think this has the advantage over global metadata
> that it is closer to the tests and so more likely to be kept up to
> date when tests change. In particular I would expect it to be owned
> by the test owner rather than someone coordinating the testsuite as
> a whole. It has the advantage over per-file metadata that it doesn't
> affect the test itself. In particular I propose having a json
> manifest file with a well-known name like "manifest.json" in each
> directory containing tests. The file would have a structure like
> (missing some syntax for ease of reading):
>
> {tests:{"001.html":{type:"javascript",
> flags:["SVG"],
> expected_results:10,
> top_level_browsing_context:false
> }
> },
> subdirs: ["more_tests"]
> }
>
> type is "javascript", "reftest" or "manual"
> flags indicates specific optional features required by the test or
> other unusual dependencies
> expected_results (missing default: 1) indicates the number of tests
> in that file
> top_level_browsing_context (missing default: false) indicates that
> the test needs to run in a top level browsing context (e.g. for
> testing window.top)
>
> subdirs is a list of subdirectories in the current directory that
> should be checked for tests.
>
> Does this sound reasonable? Did I miss anything obvious?
[ following up on a somewhat old thread here ]
I don't see why this is needed, and it's a extra work to maintain,
especially if people are contributing tests written elsewhere.
The number of tests isn't important (and is not a good measure of
testing coverage); what matters is whether any of them failed.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Received on Thursday, 17 February 2011 23:24:20 UTC