On Monday 2016-11-28 15:48 -0800, Daniel Holbert wrote:
> Mozilla solves this using "fuzzy" annotations for reftests:
> https://hg.mozilla.org/mozilla-central/annotate/8387a4ada9a5c4cab059d8fafe0f8c933e83c149/layout/tools/reftest/README.txt#l118-122
>
> They look like this in our reftest manifest:
> fuzzy(3,640) == bug917595-exif-rotated.jpg bug917595-pixel-rotated.jpg
>
> This particular annotation means the harness will allow 640 pixels to
> differ by up to 3 in any given color channel. Put another way: if the
> testcase renders a particular pixel with rgb(0,0,255), and the reference
> renders rgb(3,0,255) for the same pixel, then this wouldn't trigger a
> test-failure (unless there were more than 640 such pixels).
That said, I don't particularly encourage this, for two reasons.
One, fuzzy is badly broken because of:
https://bugzilla.mozilla.org/show_bug.cgi?id=1252361
and doesn't produce unexpected passes for many cases where it ought
to.
Second, there are many cases where this can be avoided by
structuring the tests so they go through the same codepaths, so the
test and reference only differ by the thing that is being tested.
-David
--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)
Received on Tuesday, 29 November 2016 00:36:53 UTC