Re: Requirements for (level >=3) tests

On Feb 16, 2012, at 10:10 AM, Aryeh Gregor wrote:

> On Thu, Feb 16, 2012 at 11:29 AM, Øyvind Stenhaug <oyvinds@opera.com> wrote:
>> I have assumed that the guidelines at
>> http://wiki.csswg.org/test/css2.1/contribute etc also apply to non-CSS2.1
>> tests, even though the URLs, breadcrumbs and content tend to mention 2.1
>> specifically. Whether this is true or not, I am wondering which parts are
>> absolutely *required* for submitted tests. Maybe that could be stated more
>> clearly.
>> 
>> For instance, is the XHTML format mandatory for other testsuites? (It seems
>> not all of them use it.) Are #help <link>s always necessary for building?
>> 
>> Of course, if I write a new testsuite I will try to keep all of these
>> guidelines in mind. However, for existing and more ad-hoc tests, and tests
>> written by others, the higher the bar the harder it will be to find the
>> required time to do all the work for converting and submitting.
> 
> To date, I've been ignoring the CSS 2.1 guidelines.  Just as a
> motivating example, I currently have a ref file at
> contributors/aryehgregor/incoming/ref-3d/green-rect.html, which
> consists of

Aryeh, simply ignoring the guidelines is not acceptable. They exist for a reason and have been developed over years of experience developing CSS test suites by many people.

If you have an issue with the guidelines, or feel they can be improved, speak up, and let's discuss it. By developing tests that don't follow the guidelines, you're developing tests that won't work with our tools, or will place an undue burden on those who will use the test suite.

Yes, the wiki page refers only to 2.1, it was developed during the 2.1 test suite development. By and large, those requirements stand for post 2.1 test suite development. If the wiki needs to be updated, well, it's a wiki, update it.


> 
> <!doctype html>
> <div style="width: 100px; height: 100px; background: lime"></div>
> 
> If I read the guidelines correctly, this should really be
> 
> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
> <html xmlns="http://www.w3.org/1999/xhtml">
> <head>
>  <title>CSS Reftest Reference</title>
>  <link rel="author" title="Aryeh Gregor" href="mailto:ayg@aryeh.name"/>
>  <style type="text/css"><![CDATA[
>   div { width: 100px; height: 100px; background: lime }
>  ]]></style>
> </head>
> <body>
>  <div></div>
> </body>
> </html>
> 
> I think I'm not being unreasonable when I suggest this requirement is
> onerous, given that the two files render identically in all browsers
> and contain essentially the same information.  Requiring XHTML 1.0
> Strict may have made sense in the early days of the CSS 2.1 test
> suite, but it doesn't make sense now.

The requirement for XHTML was to ease processing of the tests with our automated tools. The toolchain does now include an HTML5 parser and should be able to process tests in HTML5.

Also note however, that built test suites include HTML, XHTML and XHTML-Print versions of the tests. These versions are generated by the build system from the single source input. This is true for both tests and reference files (at least reference files in XHTML or HTML).

While using a minimal HTML test source should work, the test that will be part of the official test suite will be the output of the build process, and will be a serialization of the DOM.

Furthermore, while you can get away with minimal markup in HTML source files, most of the other metadata components listed in the requirements are not optional (the only thing really optional is the assertion, and that's helpful to have anyway). For example the spec links are absolutely mandatory, the build system will be using those to map tests to test suites. They're also used by Shepherd to help searching for tests and issues. As I told you previously, once you actually submit your tests, Shepherd will point out the issues with the format.

Another issue with submitting tests in a minimal markup form is that it has the potential to introduce unnecessary diffs. Shepherd will at some point allow editing of test metadata via the web interface, when this happens, the test will be re-serialized from its DOM and will contain significantly more diffs than those required by the edit, which will be bad. Others may use other tools to edit the files and have similar results.

Received on Friday, 17 February 2012 01:51:21 UTC