Re: UAs passing tests if they don't implement a feature

On Sun, Jun 24, 2012 at 7:06 PM, Linss, Peter <peter.linss@hp.com> wrote:
> Understood, note that the documentation is in a wiki… feel free to update it yourself as well where you feel that improvements are warranted (but it's not your responsibility to do that).

I would, but I don't want to make changes that others might not agree
with -- particularly since it seems like others tend to disagree with
me here fairly often.

> I'm thinking of some improvements to Shepherd as well here to help avoid misunderstandings in the future.
>
> I accept that the status of "Needs Work" certainly gives the impression that "this _must_ be changed". Perhaps an additional status level which means "this is acceptable, but it'd be nice if…", suggestions of what to call that are welcome.

The Linux kernel has a directory called staging/ where drivers are put
that are known to be useful but need code-quality work.  (They have a
big issue where vendors will write large drivers behind closed doors,
not to kernel coding standards, and then try dumping them so they can
claim Linux support.)

How about "Accepted For Further Work"?  "Accepted (Cleanup Requested)"?

On Sun, Jun 24, 2012 at 7:15 PM, "Gérard Talbot"
<css21testsuite@gtalbot.org> wrote:
>
> Le Dim 24 juin 2012 11:33, Aryeh Gregor a écrit :
>> I appreciate the clarification, and accept that this was just a
>> misunderstanding.  Please keep in mind that this is my first
>> involvement with CSSWG testing; as such, things that might seem
>> obvious to you or fantasai are not necessarily obvious to me.
>> Up-to-date documentation is essential.
>
> Documentation on test format and guidelines is updated as far as I can
> see. Always has been.

Current documentation seems to suggest that, e.g., using style
attributes is not allowed:

"Do not use the style attribute (inline styles) unless specifically
testing that attribute"
http://wiki.csswg.org/test/format#body-content

It's not clear on whether tests can be accepted if they don't obey the
guideline.

> Has there been one single test where a test was rejected or put into the
> NeedsWork status because of what you refer to as stylistic reasons?
> Aryeh, please be specific here.

Yes -- for instance, here:

http://test.csswg.org/shepherd/testcase/transform-background-001#comment-776e0f90dc6c

See also here:

http://test.csswg.org/shepherd/testcase/transform-background-001#comment-b8400648b91e

Similar comments were posted on a bunch of my submitted tests, and it
was clear they applied to all of them.

In the first case, I saw that the feedback seemed to match the
guidelines, and discussion on public-css-testsuite seemed to indicate
that there was no support for changing the guidelines to allow the
tests I submitted.  In retrospect, although I haven't reread the
threads to be sure, perhaps it was unclear to participants in the
discussion that I was mostly interested in not having to rewrite my
already-written tests, rather than allowing variation for tests
written a priori for the test suite.  In any event, it certainly
seemed to me like my test was not going to be approved unless I
rewrote it.

In the second case, I didn't think the feedback was justified by the
written guidelines, so after replying, I set the status back to
Awaiting Review.

> Several questions still remain. What are the guidelines, requirements
> for new tests? And why people do not follow these?
>
> There are long-term and short-term benefits to following guidelines and
> requirements for new tests.
>
> Aryeh Gregor, are you and were you using (pasting) the template when
> creating new tests?

First of all, a lot of my tests were from Mozilla's internal reftest
suite.  They weren't written to CSSWG style guidelines because they
weren't originally written for the CSSWG at all.

Second of all, at the time I wrote my tests, I looked briefly at the
guidelines but saw that they were clearly written several years ago
and never updated.  For instance, parts were written assuming that the
tests were for CSS 2.1 rather than CSS 3, and XHTML was the only
allowed input format (which made sense in 2006 but doesn't in 2012).
I concluded that I would write the tests first and deal with style
later.  This was apparently a mistake.

Received on Monday, 25 June 2012 12:19:25 UTC