Re: Proposed policy change: reusability of tests by other browsers

On 10/11/2012 05:59 PM, Boris Zbarsky wrote:
> On 10/11/12 5:20 AM, James Graham wrote:
>> Hmm, so I'm not quite sure how that would work. Is the system that you
>> are imagining one where the tests (and the harness) are all installed
>> locally on the device being tested, or, at least, that there is a 1:1
>> ratio between server instances and test machines?
>
> Hmm.  That's certainly what you want for testing while developing,
> usually, since you typically need to modify both the code and the tests
> as you go (keep in mind that at Mozilla the developer of the code is
> also responsible for writing tests for it).

Yes, our typical solution for that is to have people set up an 
apache+php install on their local box that sufficiently closely mimics 
the central setup that they are able to run tests locally when needed.

>> What exactly are the requirements that you have? Presumably it should be
>> possible to invoke custom server behaviour for some tests? How should
>> this work? How do you deal with things like https?
>
> So what I know about Mozilla's current test harness is that it gives me,
> as a consumer, the following:
>
> 1)  An HTTP server.
> 2)  A bunch of hostnames that are all mapped to that HTTP server (by
> setting up a proxy in the profile that the test automation generates),
> so that cross-domain tests can be done.
> 3)  Ability to easily send content from the server with a given set of
> headers (by putting a foo^headers^ file alongside the foo file in the
> test directory).
> 4)  Ability to fully script HTTP response header and body generation
> based on information in the request or internal server state (e.g. to
> test caching behavior when requests for the same URI return different
> sorts of things, or to echo back stuff the browser sent so the browser
> can then check that it sent the thing the server expected).
>
> I suspect that 1, 2, 4 is sort of a bare minimum for any test setup that
> actually expects to test what UAs are putting on the wire for testing
> things like CORS.  The only question is whether we can prevent every
> harness deployment having to reinvent those bits somehow.  And in
> particular, note that #4 means that parts of the server are in fact
> parts of the test...

Indeed. So our setup is basically apache + some of the standard modules 
+ PHP + a known set of domains and subdomains that tests can use (that 
mostly map back to the same central server, but with some unusual 
requirements like FTP being handled by a second server that I know much 
less about).

I can understand why you wouldn't want to run a full apache + php 
install on every test client; it seems like it has the potential to be a 
maintenance headache, especially if it is also a requirement to be able 
to automatically install and run it in an ad-hoc way on developer's 
machines.

Received on Thursday, 11 October 2012 16:34:03 UTC