Re: UAs passing tests if they don't implement a feature

On Jun 24, 2012, at 8:33 AM, Aryeh Gregor wrote:

> On Sun, Jun 24, 2012 at 5:52 PM, Linss, Peter <peter.linss@hp.com> wrote:
>> Ok, I think I'm finally seeing the underlying disconnect here. Please correct me if I'm wrong.
>> 
>> Aryeh, please look here:
>> http://test.csswg.org/shepherd/search/reviser/AryehGregor/status/rejected/
>> 
>> This is the complete list of all the tests you touched that have been rejected. Note that it's an empty list.
>> 
>> Also check here:
>> http://test.csswg.org/suites/css3-transforms/nightly-unstable/
>> 
>> This is the current version of the Transforms test suite as created by our nightly builds, note that every one of your submitted tests is present. While this is a nightly build, I can assure you that at no time was any of your tests rejected from the build (if one was missing it's due to a bug in the build code).
>> 
>> None of your tests have _ever_ been rejected for stylistic reasons. Looking through Shepherd's logs, you did receive feedback asking for improvements. I see some feedback that's technical, file name conflicts and the like, and some stylistic feedback. Note that I didn't go though all of your tests so if I missed something significant, please point it out to me.
>> 
>> The stylistic feedback you received was mostly from Dirk, and I read and interpret the bulk of those as him expressing his personal opinion (about that tests should be guide how to write good documents) and a polite request "I would like to ask that…".
>> 
>> Please not that Dirk's opinion here (while not invalid in itself), is _not_ an official requirement of the WG and in no way were your tests rejected for stylistic reasons.
>> 
>> While there's value in your having modified your tests in response to Dirk's feedback, a perfectly valid response to that would have been "Ok, noted for future tests that I write from scratch" and you could have left it at that (at least as far as I'm concerned).
> 
> I appreciate the clarification, and accept that this was just a
> misunderstanding.  Please keep in mind that this is my first
> involvement with CSSWG testing; as such, things that might seem
> obvious to you or fantasai are not necessarily obvious to me.
> Up-to-date documentation is essential.  In this case, the guidelines
> make it sound like certain stylistic conventions are mandatory for all
> tests.

Understood, note that the documentation is in a wiki… feel free to update it yourself as well where you feel that improvements are warranted (but it's not your responsibility to do that).

> 
> I should also point out that not approving a test isn't much better
> than rejecting it, or at least looks that way to me.  So when I said
> "reject" I really meant "not approve".  If reviewers are allowed to
> not approve my tests for stylistic reasons, it means that I have to
> rewrite them if I want them approved, even if they're not technically
> rejected.  To avoid this, I think the guidelines should clearly
> specify what are valid grounds for withholding approval, and
> specifically list some things that are *not* valid grounds for
> withholding approval (like style, or "you should test this too").

I'm thinking of some improvements to Shepherd as well here to help avoid misunderstandings in the future.

I accept that the status of "Needs Work" certainly gives the impression that "this _must_ be changed". Perhaps an additional status level which means "this is acceptable, but it'd be nice if…", suggestions of what to call that are welcome.

> 
>> I agree with you completely here (except for the perception that your tests were rejected due to this).
>> 
>> Purely stylistic issues in tests are in the "nice to have" category. There are expectations for people writing tests from scratch to use best stylistic practices, but for adapting existing tests to our format, our concern is really only getting the required metadata and making sure the test is well formed (and functionally correct, of course).
> 
> I'm glad to hear this, and I think this should be reflected in the
> guidelines -- both for people who might come after me, and for me to
> point people to when they ask me to make certain types of changes
> before they'll give review.

Received on Sunday, 24 June 2012 16:07:24 UTC