Re: Conversion of MS CSS 2.1 tests to reftests

On 10/09/10 23:12, fantasai wrote:
> On 09/10/2010 09:15 AM, John Jansen wrote:
>>
>>>
>>> On 09/09/2010 12:46 PM, Geoffrey Sneddon wrote:
>>>> Hey,
>>>>
>>>> Attached is a diff to convert 830 tests to reftests (with a
>>>> mere four references!). The list of tests for each reference is
>>>> based upon tests that pass in Opera with the same screenshot,
>>>> so anything that we fail won't be included, but still, 830
>>>> tests is a nice start.
>>>>
>>>> It'd be nice if you could apply the diff (just adding five new
>>>> files), and adding it to the build system, sometime in the near
>>>> future.
>>>
>>> Arron, if this looks okay to you, I'm happy to check it in for
>>> you.
>>>
>>> ~fantasai
>>>
>> I would like to get some clarification on this ask of Arron before
>> he commits to this work.
>>
>> Based on my understanding of how test ownership works, you are
>> asking Microsoft to take ownership of these references moving
>> forward, and so any changes to the tests referenced here will
>> require that we update the references themselves. Correct?
>
> Only changes to the test that would also change the intended
> rendering would require updating the references. For the kinds of
> tests Geoffrey is creating references for, this doesn't seem
> particularly likely.
>
> Also, if the concern is about maintaining references, I suspect
> Geoffrey would be willing to handle maintenance of the references if
> Arron is willing to delegate that job to him. (Geoffrey can confirm
> or deny that.) The main ownership issue here is that Arron is
> managing those directories, and should be aware of anything that
> affects them.

Yeah, I'm happy to maintain the references (at the same time, I agree 
with fantasai that they're unlikely to change, though as more tests are 
automated that may become less true, but I'd hope the spec doesn't 
change in a way to break too many tests with the timeline Daniel was 
talking about at the F2F).

>> Is your expectation that the 830 tests here are just the beginning?
>>  The test suite should be locked down in 5 days and then
>> Implementation Reports should be complete within 30 days. If there
>> are more changes coming, that seems to go against 'locking down'.
>
> Adding references doesn't change the test, it just makes it easier to
>  run. (The tests can still be run manually, too.)
>
> I think we do want most of the CSS2.1 test suite to be automatable,
> although I don't think we'll get there within the REC time frame.

Nor do I, but I would be pleased to see a second edition of the test 
suite a number of months after the spec reaches REC (given the current 
timezone) which does reach that goal.

>> If there are no additional changes coming, it seems like reftesting
>>  ~8% of the suite isn't worth the cost/benefit analysis. Maybe I'm
>> missing something, but I also think that each Vendor will want to
>> run these tests manually anyway in order to verify your
>> references.
>
> I think the vendors with reftest infrastructure will check that the
> reference renders correctly, and then let the reftest infrastructure
> run the tests. And then fill in the rest of the implementation
> reports manually.

Indeed. (As it is now, I would expect both ourselves and Microsoft who 
already have the tests in our regression tracking systems to produce an 
implementation result from that; the only real difference is moving the 
verification of the screenshots to being automated).

>> That feels like not only a 0 sum gain, but in reality extra work
>> for everyone as we are approaching the end to CSS2.1 (which I what
>> I believe we all want - to be done and moving on with CSS3).
>>
>> Please verify, I may just be misunderstanding the expectations
>> here.
>
> If Geoffrey is putting in the time to create references for tests,
> that will reduce the amount of work in creating and updating
> implementation reports. It might make the difference between getting
> an implementation report from Mozilla and not getting one.

In our case, it will make the difference between us producing an 
implementation for a single shipping browser/platform combination (i.e., 
Opera Desktop on Windows XP) or for multiple platforms, which may become 
relevant for things such as font-weight which are not implemented per 
spec in the majority of browsers on Windows.

Finally, can I reiterate what was said at the F2F: we're willing to help 
automate some of the testsuite, but we can't commit the resources to 
automate it all ourselves. Any help from anyone else would be 
appreciated. David Singer said he'd try and nag some people into 
helping; this would be much appreciated if the nagging succeeded. :)

-- 
Geoffrey Sneddon — Opera Software
<http://gsnedders.com>
<http://opera.com>

Received on Tuesday, 14 September 2010 17:24:05 UTC