RE: getting your tests into browser test framework

I'm ready to work on obsoleting 3896 and replacing it with a URL standard based on current implementation status.

I think we need to separate the "http" implementation from the "URL" implementation, though, and test other protocol handlers too.

Every incompatible change from 3896/3897 should be justified by test cases which are passed by current, widely deployed implementations.



> -----Original Message-----
> From: Simon Pieters [mailto:simonp@opera.com]
> Sent: Thursday, November 01, 2012 12:22 AM
> To: Chris Weber
> Cc: Larry Masinter; julian.reschke@gmx.de; jet@junglecode.com;
> mike@w3.org; 'Peter Saint-Andre'
> Subject: Re: getting your tests into browser test framework
> 
> On Wed, 31 Oct 2012 18:49:11 +0200, Chris Weber <chris@lookout.net> wrote:
> 
> > On 10/30/2012 3:03 PM, Simon Pieters wrote:
> >> On Tue, 30 Oct 2012 22:38:14 +0200, Chris Weber <chris@lookout.net>
> >> wrote:
> >>> That sounds good, how does it relate to
> >>> http://w3c-test.org/framework/app/suite?  It seems like the test suite
> >>> here would be an ideal place to store the tests and the results.
> >>
> >> That seems to be a repo similar to the html and webapps ones:
> >> http://dvcs.w3.org/hg/testframework/

> >>
> >
> > Ultimately we'd want these tests and data to be in Robin Berjon's
> > http://w3c-test.org/framework/ correct?
> 
> I don't have experience with this runner, but I guess they would be
> runnable in that.
> 
> >  I'm not sure how all this
> > relates... but storing all the test results seems most valuable.
> 
> I think making browser vendors to run the tests as part of their
> regression testing would be valuable.
> 
> >>
> >> Is such a setup high enough fidelity to be confident that edge cases
> >> went through untouched from what the browser sent over the wire? (It
> >> might well be; I don't know.)
> >
> > Good question.  In my lab integrity was guaranteed : -)  But there were
> > no meddling devices, just the client direct to the server.  If there
> > were a WAF or other device modifying requests before they reached the
> > server, wouldn't that be a problem for the test framework in general?
> 
> Yes, it would, but I guess the vast majority of test cases in general are
> unaffected. If we want to test weird conditions in the network layer,
> maybe we need something else for such tests. We have not figured out yet
> what the requirements are for Mozilla and WebKit when it comes to the
> server-side part of tests in general. PHP works for Opera and, I'm told,
> IE, but Mozilla and WebKit currently don't use an Apache+PHP server for
> tests.
> 
> >> Would it not be more useful if the "expected" is an absolute URL, since
> >> an absolute URL is what you get by resolving ref to base? Also, I think
> >> it would be useful to have expected results according to Anne's URL spec
> >> -- for instance the space in path (but not in the query) is expected to
> >> turn into %20 after resolving. (file: isn't done in his spec, yet, but
> >> it's being worked on.)
> >
> > This will need to be addressed based on the test "group".  You'll notice
> > there are currently 22 groups of tests, each might have a different
> > schema from the others.  I can document what that schema is so it's easy
> > to build the testing functions and expected results.  I'm open to making
> > changes based on any of your suggestions about the current naming
> > conventions and schema, if this format is something you're interested in!
> >
> > All of these tests and expected conditions come directly from Webkit
> > with only minor modification. The most significant mod was that I had to
> > convert all \xNN notation to %NN since \x is not supported in JSON.
> > Luckily there were only a handful of cases for this.
> 
> So %NN, if tested as three seperate characters, is different to using a
> raw character. JSON supports \uNNNN. I don't know if \xNN was supposed to
> represent a byte or a character, however.
> 
> > Regarding expected results - these are Webkits, so they expect what they
> > want to see in Webkit.  In other words, many of these will differ in
> > Opera, IE, and FF.  I think this list will need to be curated more
> > perhaps on a case by case basis.
> 
> OK, that's good to know.
> 
> > Now I imagine myself emailing the uri and public-iri lists for each of
> > the 549 test cases asking for feedback on the expected result.  Would
> > that be reasonable to do? : -)
> 
> I guess one email for all tests would be appropriate. :-)
> 
> >>
> >> Which forum?
> >
> > both uri and public-iri
> >
> > -Chris
> 
> 
> --
> Simon Pieters
> Opera Software

Received on Thursday, 1 November 2012 04:57:11 UTC