Re: wptrunner and how to handle ref tests

On Tue, Jul 1, 2014 at 9:31 AM, James Graham <james@hoppipolla.co.uk> wrote:

> On 01/07/14 17:19, Dirk Pranke wrote:
>
> > I certainly don't, that's true. But, if I was being a standards purist,
> it
> > seems like defining fuzzy matching criteria would be a good idea, rather
> > than leaving it be implementation defined.
> >
> > That said, I'm not being a standards purist and I'd rather focus on
> > whatever gets people running more tests more often :).
>
> Sure. But there are pragmatic reasons for not depending on Imagemagick.
>
> The simplest is that it potentially requires everyone to use the same
> version of Imagemagick, with the same compile time switches, in order to
> get the same results. This leaves a high chance of things like OS
> upgrades on test infrastructure breaking results. We might be lucky and
> get away with it, but it is a risk.
>

I'm sorry, I thought you were arguing that Imagemagick was a good idea, and
now I think you're suggesting the opposite?

I actually don't have a strong leaning either way, as I don't have an
answer I really like to this problem, short of understanding where the
actual diffs come from and what can be done about them.

-- Dirk