[Bug 15062] Test results appearing in other test suites

https://www.w3.org/Bugs/Public/show_bug.cgi?id=15062

Peter Linss <peter.linss@hp.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |
         AssignedTo|mike@w3.org                 |peter.linss@hp.com

--- Comment #8 from Peter Linss <peter.linss@hp.com> 2011-12-05 23:35:09 UTC ---
(In reply to comment #7)
> (In reply to comment #4)
> 
> While it is common to run CSS tests in the CSS test suite in more than one
> format, those tests are usually checking an assertion about the same part of
> the *CSS* spec.
> 
> That's not what we're talking about here. 
> 
> I'm talking about a test in html5 format that is checking something against the
> html5 spec, and another test, that happens to be testing the same assertion,
> but that is testing a point in a completely different spec, such as HTML4. 
> 
> If I'm interested in reviewing the results of tests related to the html5 spec,
> do i really want to see the results of tests for all the other formats that
> happen to test the same thing? Probably not, most of the time. Certainly not,
> if I'm trying to ascertain whether we have the two passes that are needed for
> CR entry criteria for the html5 spec.

Ok, then the tests should still be linking to the relevant sections of the
specs they're testing.

So you have one test 'foo' in HTML5 format, linked to a section of the HTML5
spec. Then you can have another test, also called 'foo' in HTML4 format, linked
to potentially a different section of the HTML4 spec.

In both cases, the tests are linked to the specs that they are testing, and the
results will show up in the relevant test suites.

So I do now see that there's an issue in the result reporting where tests in 
formats not specified by the test suite may show up in results (and I'm taking
the bug to fix that), however you're still not setting up your test suites
properly.

First, you need to remove your section data, this is only for sections of
specs, not of test suites. 

Second, you need Mike to set up the proper format data in the framework, there
is currently only one HTML format: 'html', you need separate formats listed for
each of your test formats, ie: HTML4, HTML5, etc. 

Third, ensure the proper spec section data is loaded for the specs you are
testing: ie: HTML4, HTML5 etc. 

Then make sure your test suites are set up properly, ie: the i18n-HTML4 test
suite lists only the tests in HTML4 format, linked to the relevant sections of
the HTML4 spec and using the HTML4 format identifier; the i18n-HTML5 test suite
lists only the tests in HTML5 format, linked to the HTML5 spec, using the HTML5
format identifier, etc.

The test suites you are creating are really sub-sections of the HTML4, HTML5,
etc test suites that deal with i18n issues. The tests in those suites will be
listed by the sections of the specs they are testing, and any results gathered
will be shared with other test suites that use the same tests (like a full
HTML5 test suite).

-- 
Configure bugmail: https://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

Received on Monday, 5 December 2011 23:35:15 UTC