Re: Language requirements for server side code in the test framework (was: Review of tests upstreamed by implementors)

On Thu, 21 Mar 2013, Tobie Langel wrote:

> I was planning to look at what the requirements for that were based on 
> the existing tests. Should I expect this to be much more challenging 
> than a twist on an echo server?

I guess using something like that could work for a subset of cases, but it 
seems particularly bad for readbility if it encourages people to make a 
substantial amount of the test URL-encoded. However there are a large 
number of things that it doesn't work for.

> Any pointers as to which tests have specific requirements in that area?
>> and also to store state across requests.
>
> Hadn't thought about that one. What are the use cases?

Progress events. Tests that require reloading documents. Tests that need 
to count the requests made, or check that requests reach the server in the 
right order. Tests that depend on authentication (e.g. authenticate in one 
step and check that credentials are required, then check that credentials 
are sent automatically in following steps). Tests for the HTTP cache or 
appcache.

>> For
>> comparison, see the documentation of Mozilla's javascript based solution
>> [1] (Opera have had more or less all the reequirements described in that
>> document, but have historically implemented them on top of Apache + PHP
>> instead of a custom solution).
>
> Lifting Mozilla's server might be a possible alternative, here.

I believe it is heavily dependent on XPCOM, so it isn't at all portable. 
Using the same idea but in python could work. However note that we also 
need a solution that we can run on w3c-test.org where there could be 
multiple simultaneous clients accessing the server. That means that e.g. 
state can't be global. It also means that it shouldn't fall over under 
load. Not an easy problem :(

>> I don't think the most significant property of the system is that it is
>> simple to use (although of course making easy things easy is a good
>> thing); I think the most significant property is that hard things are
>> possible.
>
> Well, you can always fallback to writing custom code for this, can't you?

Custom code where? It isn't clear from what you said that you were 
thinking of a server that can run custom code. But yes, I think my point 
is that the server has to be able to run custom code on a per-test basis 
to deal with the edge cases.

Received on Thursday, 21 March 2013 20:19:49 UTC