Re: Conversion of MS CSS 2.1 tests to reftests

On 09/10/2010 09:15 AM, John Jansen wrote:
>
>>
>> On 09/09/2010 12:46 PM, Geoffrey Sneddon wrote:
>>> Hey,
>>>
>>> Attached is a diff to convert 830 tests to reftests (with a mere four
>>> references!). The list of tests for each reference is based upon tests
>>> that pass in Opera with the same screenshot, so anything that we fail
>>> won't be included, but still, 830 tests is a nice start.
>>>
>>> It'd be nice if you could apply the diff (just adding five new files),
>>> and adding it to the build system, sometime in the near future.
>>
>> Arron, if this looks okay to you, I'm happy to check it in for you.
>>
>> ~fantasai
>>
> I would like to get some clarification on this ask of Arron before he commits to this work.
>
> Based on my understanding of how test ownership works, you are asking Microsoft to take
> ownership of these references moving forward, and so any changes to the tests referenced
> here will require that we update the references themselves. Correct?

Only changes to the test that would also change the intended rendering would require
updating the references. For the kinds of tests Geoffrey is creating references for,
this doesn't seem particularly likely.

Also, if the concern is about maintaining references, I suspect Geoffrey would be
willing to handle maintenance of the references if Arron is willing to delegate
that job to him. (Geoffrey can confirm or deny that.) The main ownership issue
here is that Arron is managing those directories, and should be aware of anything
that affects them.

> Is your expectation that the 830 tests here are just the beginning? The test suite should
> be locked down in 5 days and then Implementation Reports should be complete within 30 days.
> If there are more changes coming, that seems to go against 'locking down'.

Adding references doesn't change the test, it just makes it easier to run. (The
tests can still be run manually, too.)

I think we do want most of the CSS2.1 test suite to be automatable, although I
don't think we'll get there within the REC time frame.

> If there are no additional changes coming, it seems like reftesting ~8% of the suite isn't
> worth the cost/benefit analysis. Maybe I'm missing something, but I also think that each
> Vendor will want to run these tests manually anyway in order to verify your references.

I think the vendors with reftest infrastructure will check that the reference renders
correctly, and then let the reftest infrastructure run the tests. And then fill in
the rest of the implementation reports manually.

> That feels like not only a 0 sum gain, but in reality extra work for everyone as we are
> approaching the end to CSS2.1 (which I what I believe we all want - to be done and moving
> on with CSS3).
>
> Please verify, I may just be misunderstanding the expectations here.

If Geoffrey is putting in the time to create references for tests, that will reduce
the amount of work in creating and updating implementation reports. It might make
the difference between getting an implementation report from Mozilla and not getting
one.

~fantasai

Received on Friday, 10 September 2010 22:12:39 UTC