Re: Proposed policy change: reusability of tests by other browsers

On 10/11/12 5:20 AM, James Graham wrote:
> Hmm, so I'm not quite sure how that would work. Is the system that you
> are imagining one where the tests (and the harness) are all installed
> locally on the device being tested, or, at least, that there is a 1:1
> ratio between server instances and test machines?

Hmm.  That's certainly what you want for testing while developing, 
usually, since you typically need to modify both the code and the tests 
as you go (keep in mind that at Mozilla the developer of the code is 
also responsible for writing tests for it).  So yes, if we want Mozilla 
adopting the harness as part of the normal development/testing process 
that would be ideal.  You're right that for mobile devices it may be 
desirable to run the server on a PC host instead; our test automation 
may well do something like that too.  I haven't kept up with it that well.

In general, though, the server and tests just need to sort of go 
together.  They don't need to be on the device being tested, and clearly 
such a requirement would be untenable for purposes of W3C testing.

I'm not quite sure there's an ideal solution here, to be honest.  So far 
I'm still at the identifying pain points stage.  :(

> What exactly are the requirements that you have? Presumably it should be
> possible to invoke custom server behaviour for some tests? How should
> this work? How do you deal with things like https?

So what I know about Mozilla's current test harness is that it gives me, 
as a consumer, the following:

1)  An HTTP server.
2)  A bunch of hostnames that are all mapped to that HTTP server (by 
setting up a proxy in the profile that the test automation generates), 
so that cross-domain tests can be done.
3)  Ability to easily send content from the server with a given set of 
headers (by putting a foo^headers^ file alongside the foo file in the 
test directory).
4)  Ability to fully script HTTP response header and body generation 
based on information in the request or internal server state (e.g. to 
test caching behavior when requests for the same URI return different 
sorts of things, or to echo back stuff the browser sent so the browser 
can then check that it sent the thing the server expected).

I suspect that 1, 2, 4 is sort of a bare minimum for any test setup that 
actually expects to test what UAs are putting on the wire for testing 
things like CORS.  The only question is whether we can prevent every 
harness deployment having to reinvent those bits somehow.  And in 
particular, note that #4 means that parts of the server are in fact 
parts of the test...

HTTPS "just works" from a harness consumer point of view.  There is some 
infrastructure that pregenerates certs for various hostnames and imports 
the relevant root into the test profile, I believe.  There's some 
complexity there in terms of being able to flag which hosts should use 
which certs when, etc, that I think only comes into play when testing 
the actual SSL implementation in Firefox.

I don't know yet how to make this all work in some sort of shared 
harness.  :(

-Boris

Received on Thursday, 11 October 2012 16:00:30 UTC