Re: WPT test dependencies

I'd like to thanks Soares for bringing the test coverage of the spec from
10% to close to 99%, in a few months, Bocoup for their invaluable help in
reviewing the tests and the process, and google for their support in the
process.

Even thought it s not perfect (hence the 99%) I think it's an important
effort, that was necessary so we end up in a good position. Starting with
passing tests, and only having a few failing tests happening makes it easy
to maintain, as opposed to the original situation where we had no
visibility, or another usual situations where most test fails and shadow
new mistake, providing few incentive to add tests or clean the house. Tests
where originally only written by two or three volunteers on their free
time, (harald, dom, myself), with not enough time to really catch up.

This in turn allows people interested, and willing to keep things working
by providing PRs, as well as those who love to belittle other people work,
nag, mock, and humiliate on public mailing lists or tweets, to quickly
identify the failing tests and act on it.

At the time the test were written, they were fully and provably in sync
with the specs. A new coverage system was developed in collaboration with
Bocoup to make sure of that, that is being extended to other specs.

Now, the fact that recent changes in the specs broke the sync' with the
tests was already identified as a problem by Google (harald, foolip), and
is being addressed. The only way to address this is of course to enforce
changing the tests when there is a change in the spec, automatically. The
process is being changed right now to allow that. Even though we do not
have the equivalent of block chain's smart contract, we can modify the
commit / review / merge process to enforce it.

Everything is open source, if you spot a mistake, you are one PR away from
fixing it. It's faster than writing e-mails usually. Don't be shy, "Be a
programmer" (TM). The community will be very happy to have the relevant
discussion in the review thread.


On Wed, Jan 31, 2018 at 5:47 AM, Harald Alvestrand <harald@alvestrand.no>
wrote:

> Den 31. jan. 2018 09:27, skrev Sergio Garcia Murillo:
> > When I first read the mail, I was quite in agreement with Philipp, but
> > the more I think about it, the less I am.. :)
> >
> > Regarding reducing dependencies with other parts of the spec, i think it
> > is a good goal, and we should do it if the alternative provides the same
> > validity of the result. If the alternative is parsing the SDP or not
> > testing it, then I would strongly oppose to that.
> >
> > I agree that most of the test in WPT test the js api layer, like
> > validating that if I set a value, then I get the same value back, and
> > not the effect of that setting, but in order to test that part we would
> > need some kind of external server that test IETF stuff compliance. I
> > found that piece missing and really important (for testing simulcast for
> > example), although I am not sure if that testing should be in the scope
> > of the WPT or on something different (KITE?).
>
>
> It is certainly in-scope for KITE. (That's my generic answer when the
> question involves interoperability or Net-facing behavior).
>
> >
> > Regarding changing new RTCPeerConnection() by new
> > RTCPeerConnection(null) just because some browser does "a silly thing"
> > and is not spec compliant, well.. We should not hack the test so a
> > certain browser can pass it, but viceversa.
> >
> > One last comment, I think we should be more careful when making changes
> > to the spec that potentially breaks WPTs and if not providing a change
> > to the WPT itself, at least open an issue so we can track those changes.
>
> Our theoretical policy (after the last TPAC) is that a spec change
> request needs to have a test change request along with it.
> So far, we've not been very good about policing that.
>
>
>


-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -