Re: Language requirements for server side code in the test framework (was: Review of tests upstreamed by implementors)

On Thursday, March 21, 2013 at 5:32 PM, James Graham wrote:
> On Thu, 21 Mar 2013, Tobie Langel wrote:
> > On Thursday, March 21, 2013 at 2:11 PM, James Graham wrote:
> > > We also have the problem that many of the tests simply won't run in
> > > vendor's systems. Tests that require an extra server to be set up (e.g.
> > > websockets tests) are a particular problem, but they are rare. More
> > > problematic is that many people can't run tests that depend on Apache+PHP
> > > (because they run all the servers on the individual test node and don't
> > > have Apache+PHP in that environment). Unless everyone is happy to deploy
> > > something as heavyweight as Apache+PHP, we may need to standardise on a
> > > diffferent solution for tests that require custom server-side logic. Based
> > > on previous discussions, this would likely be a custom Python-based
> > > server, with special features for testing (I believe Chrome/WebKit already
> > > have something like this?).
> > 
> > So I hop[ing to be able to get rid of Apache/PHP in favor of a much more 
> > lightweight system, combined with a convention (a manifest?) to set HTTP 
> > headers where necessary.
> 
> FWIW "being able to set headers where necessary" makes it sound like your 
> imagined system doesn't have anything like the level of control that is 
> needed to write many kinds of tests. As well as being able to set headers, 
> one needs to be able to produce more or less arbitary responses, with 
> request-based variation,

I was planning to look at what the requirements for that were based on the existing tests. Should I expect this to be much more challenging than a twist on an echo server?

Any pointers as to which tests have specific requirements in that area?
> and also to store state across requests.

Hadn't thought about that one. What are the use cases? 
> For 
> comparison, see the documentation of Mozilla's javascript based solution 
> [1] (Opera have had more or less all the reequirements described in that 
> document, but have historically implemented them on top of Apache + PHP 
> instead of a custom solution).

Lifting Mozilla's server might be a possible alternative, here.
> I don't think the most significant property of the system is that it is 
> simple to use (although of course making easy things easy is a good 
> thing); I think the most significant property is that hard things are 
> possible.

Well, you can always fallback to writing custom code for this, can't you? 
> > I was tempted with a node.js based solution, because you know, 
> > JavaScript is the language of the Web. But if picking something else 
> > guarantees these tests are going to be run by implementors, I'm willing 
> > to learn COBOL (just kidding).
> 
> Past conversations suggest that Python is a blessed language in many of 
> the relevant institutions. All of Opera, Mozilla and Google/Chromium 
> already have Python-based components of their test system. I think 
> internally Microsoft use .NET things, which it seems highly unlikely that 
> other people will adopt :)

Thanks.

--tobie

Received on Thursday, 21 March 2013 16:48:53 UTC