Hello public-test-infra,
As Erika said previously [1], Microsoft is working on adding support to IE to wptrunner and contributing back as much as we can. While we created our first internal prototype one of the problems we found were the ref tests. Some of them were failing just because the antialias on a curve was different depending on the browser. I don't think those tests should fail.
To mitigate the number of false negatives we tested different approaches and at the end we decided to use ImageMagick, its compare tool and a fuzz factor [2]. Basically we compare how different the two images are and if we get a factor equal or less than 0.015 then we pass the test. These value is experimental and it is the best we got after trying different algorithms and factors. I've attached a few images for you to better see how even if the images are not exactly equal, the test should pass (at least in this example).
Some concerns about this approach:
* It has a dependency on ImageMagick (we could implement the algorithm to remove this dependency if needed)
* There might be some tests where the factor should be tweaked or even disabled. This number could even change depending on the browser we are testing
So what does public-test-infra think of this?
[1] http://lists.w3.org/Archives/Public/public-test-infra/2014AprJun/0036.html
[2] http://www.imagemagick.org/Usage/compare/
