W3C home > Mailing lists > Public > public-css-testsuite@w3.org > June 2012

Re: UAs passing tests if they don't implement a feature

From: Aryeh Gregor <ayg@aryeh.name>
Date: Sun, 24 Jun 2012 18:33:55 +0300
Message-ID: <CAKA+AxndQhgBUemce=Y8rbzOKUPEUAEvk8ZD224=mM3ZUsgzJw@mail.gmail.com>
To: "Linss, Peter" <peter.linss@hp.com>
Cc: fantasai <fantasai.lists@inkedblade.net>, "L. David Baron" <dbaron@dbaron.org>, CSS-testsuite <public-css-testsuite@w3.org>
On Sun, Jun 24, 2012 at 5:52 PM, Linss, Peter <peter.linss@hp.com> wrote:
> Ok, I think I'm finally seeing the underlying disconnect here. Please correct me if I'm wrong.
>
> Aryeh, please look here:
> http://test.csswg.org/shepherd/search/reviser/AryehGregor/status/rejected/
>
> This is the complete list of all the tests you touched that have been rejected. Note that it's an empty list.
>
> Also check here:
> http://test.csswg.org/suites/css3-transforms/nightly-unstable/
>
> This is the current version of the Transforms test suite as created by our nightly builds, note that every one of your submitted tests is present. While this is a nightly build, I can assure you that at no time was any of your tests rejected from the build (if one was missing it's due to a bug in the build code).
>
> None of your tests have _ever_ been rejected for stylistic reasons. Looking through Shepherd's logs, you did receive feedback asking for improvements. I see some feedback that's technical, file name conflicts and the like, and some stylistic feedback. Note that I didn't go though all of your tests so if I missed something significant, please point it out to me.
>
> The stylistic feedback you received was mostly from Dirk, and I read and interpret the bulk of those as him expressing his personal opinion (about that tests should be guide how to write good documents) and a polite request "I would like to ask that…".
>
> Please not that Dirk's opinion here (while not invalid in itself), is _not_ an official requirement of the WG and in no way were your tests rejected for stylistic reasons.
>
> While there's value in your having modified your tests in response to Dirk's feedback, a perfectly valid response to that would have been "Ok, noted for future tests that I write from scratch" and you could have left it at that (at least as far as I'm concerned).

I appreciate the clarification, and accept that this was just a
misunderstanding.  Please keep in mind that this is my first
involvement with CSSWG testing; as such, things that might seem
obvious to you or fantasai are not necessarily obvious to me.
Up-to-date documentation is essential.  In this case, the guidelines
make it sound like certain stylistic conventions are mandatory for all
tests.

I should also point out that not approving a test isn't much better
than rejecting it, or at least looks that way to me.  So when I said
"reject" I really meant "not approve".  If reviewers are allowed to
not approve my tests for stylistic reasons, it means that I have to
rewrite them if I want them approved, even if they're not technically
rejected.  To avoid this, I think the guidelines should clearly
specify what are valid grounds for withholding approval, and
specifically list some things that are *not* valid grounds for
withholding approval (like style, or "you should test this too").

> I agree with you completely here (except for the perception that your tests were rejected due to this).
>
> Purely stylistic issues in tests are in the "nice to have" category. There are expectations for people writing tests from scratch to use best stylistic practices, but for adapting existing tests to our format, our concern is really only getting the required metadata and making sure the test is well formed (and functionally correct, of course).

I'm glad to hear this, and I think this should be reflected in the
guidelines -- both for people who might come after me, and for me to
point people to when they ask me to make certain types of changes
before they'll give review.
Received on Sunday, 24 June 2012 15:34:45 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 24 June 2012 15:34:51 GMT