Re: [WebIDL-tests] WebIDL tests (#271)

My personal experience with modifying idlharness is that it's a huge pain.  The code is scattered all over, there's lots of repetition, no particular rhyme or reason to where certain things are tested, etc.

Also, idlharness assertions are not that well factored into test() blocks, so it's easy to get no coverage of all sorts of stuff because you don't implement some one thing somewhere that has nothing to do with the other stuff idlharness is testing.

Fixing these problems is a pretty huge undertaking.  So I actually somewhat favor less monolithic, more maintainable tests to some extent.

In principle, I'm not sure anything prevents us from rolling these tests into idlharness and ensuring that idlharness runs on interfaces that exercise the relevant functionality.

In practice, let's consider the workflow around adding a test for a new IDL feature.  With small tests like this, you pick an interface that uses the feature, write a test against it.  With idlharness, you pick an interface that uses the feature.  You make sure idlharness is running against that interface.  You make sure the idlharness parser can parse the feature (want to test `record<>`? have to go add it; it's not supported there yet).  You make sure idlharness can auto-detect which bits use the feature.  You add some tests conditioned on that auto-detection.  This is a lot more pain...

And that's assuming you can do it at all.  If you have some function that exercises the feature you want, and it needs some specific arguments, auto-figuring-out how it gets those arguments is quite nontrivial.  Can they be constructed via constructor calls based on their IDL interface types?  What if those have no constructors but we have factory methods?  etc.

As a concrete example, how would you write a [Clamp] test in idlharness?  You need to observe the value that gets passed through to the underlying implementation, but you need to know something about the underlying implementation to manage to do that, right?

I guess the upshot of all that rambling is that I think we should definitely get targeted tests of specific features written, _outside_ of idlharness.  idlharness is then somewhat useful for checking that the IDL in browsers matches the spec IDL, effectively, for the things idharness can test...

View on GitHub: https://github.com/w3c/web-platform-tests/pull/271#issuecomment-276717119

Received on Wednesday, 1 February 2017 17:08:37 UTC