Re: Size for reftests

On Thu, Jan 23, 2014 at 11:25 AM, James Graham <james@hoppipolla.co.uk>wrote:

> On 23/01/14 17:38, Dirk Pranke wrote:
>
>  It would be unfortunate if we had to figure out a way to use two different
>> sized windows, and I would expect running tests designed for a 600x600
>> screen to occasionally fail on an 800x600 screen. I would also be
>> surprised
>> if 600x600 really made things much faster, but I am often surprised by
>> things ...
>>
>
> So, the main argument for 600x600 seems to be that if you are running on a
> mobile device with a 768x1024 screen and can't force it into landscape
> mode, a reftest designed for 600x600 will be fine, but an 800x600 one will
> break.
>
> I don't really know how important that particular form factor is.
> Obviously it is something that Mozilla care about right now, but for all I
> know in a few years everyone will have far more (CSS) pixels than that. Or
> people will be trying to run reftests on their watch with far fewer pixels.
>
> I should also say that, for us, the plan is to keep web-platform-tests
> isolated from our legacy tests so if we need to use a different viewport
> size in the two cases that is possible. But if I add a whole load of tests
> that we can't (eventually) run on mobile then it will probably make my
> colleagues grumpy…
>
>
>  One could argue that reftests should be mostly viewport-size-independent
>> (within reason), and that we should change or fix ones that aren't. It
>> would not surprise me that there are some tests that *have* to be
>> size-dependent, but I don't know of any offhand. I also would not be
>> surprised if guaranteeing size independence often introduced unnecessary
>> complexity into a test as well ...
>>
>
> Yes, I think we need to pick a size and go with it here.


It's hard for me to say anything further about this without some recourse
to numbers ... How many of the existing web-platform-test and csswg
reftests require a particular size or range of sizes, and would break given
a 600x600 viewport or a 800x600 viewport? I don't actually know.

Also, while I certainly understand the desire to have our tests actually
render without being cropped (or overflowing or scrolling) on real devices,
I'm not very sure where to draw the line on resolution sizes. It seems
likely that some reasonable percentage of mobile devices will be smaller
than 600x600 for at least the next few years. I'm not actually sure that
I'm reading http://developer.android.com/about/dashboards/index.htmlproperly,
but if I am, that suggests at least 15-20% of devices currently
in use are smaller than that. Note that relatively recent and very popular
devices likes the Samsung Galaxy S II and the iPhone 3GS are as well.

(Of course, even more will be smaller than 800x800 (which you would need if
you wanted to support both portrait and landscape); *no* currently shipping
iPhone has more than 640 pixels in width, and the Galaxy S III only has
720, so perhaps that is an argument that 800px is really too wide ..).

-- Dirk