Re: CR exit criteria and features at risk for HTML5

On 08/16/2012 09:21 AM, Boris Zbarsky wrote:
> On 8/16/12 3:04 AM, James Graham wrote:
>> In this case I'm not that worried because the W3C has the WHATWG to keep
>> it in line. If the W3C spec can't be trusted on
>> interopeability-affecting matters then the WHATWG version will become
>> the de-facto reference for implementors. So as long as bugs in the spec
>> are found at all there will be strong pressure on W3C to fix their
>> branch or risk becoming irrelevant.
>
> Even if we posit this, it seems like a failure mode best avoided by
> planning, not by UA implementors basically abandoning the W3C HTML spec
> as a useful reference...
>
>> I would happily give up any mention of testsuites in the
>> Process in exchange for solving *that* problem.
>
> Well, a good start would be having a simple, advertized way to
> contribute tests.  I don't even know how I'd go about contributing a
> test for the HTML5 test suite, and my experience with contributing tests
> for other W3C working groups has been less than pleasant.  :(
>
> Of course contributing test suites to browsers that are not Mozilla is
> not that easy for me either.  But for Mozilla, it takes me very little
> time to add a test to the Mozilla test suite.  If we can get
> contributing tests for the official test suite down to that level, I bet
> developers will be more likely to contribute.

Well the actual "contribute" part is basically "commit to a hg repo". 
The full instructions are at [1].

> Another alternative is what the CSS working group is doing: making it
> easier to sync tests back and forth between their test suite and the
> browser-developer test suites (e.g. by adopting a test format that the
> browser developers can use easily inside their own test suites, setting
> up procedures for mirroring tests, etc).  Then I could just check in my
> tests to the Mozilla test suite as I normally do and they'd
> automatically end up in the spec test suite.  ;)

Well we are using reftests for rendering tests and testharness.js tests 
for javascript-driven tests. Thanks to Ms2ger Mozilla has a harness for 
running testharness.js tests inside mochitest. I know this isn't quite 
as easy for you as "just write a mochitest", but reusing that directly 
would have made things harder for others. Inevitably what we have is 
something of a compromise, but it mostly seems to work OK. Suggestions 
for improvements are of course welcome.

I'm not sure what you need us to do to help you mirror tests, but I am 
happy to help with anything that will make it easier for you. At Opera 
we have a slightly different setup to Mozilla/WebKit where we can keep 
tests in a seperate repository. Therefore, for W3C tests we are moving 
to a git-backed setup where the master branch is always in sync with 
upstream and any local patches we need to get the tests working with our 
infrastructure are on an "opera" branch. Then pulling in updates is a 
"git fetch && git rebase origin/master" away. Similarly, pushing local 
changes is a matter of getting them onto the master

I don't see any reason that whatever infrastructure on your end works 
for contributing to CSS can't be trivially ported to work with HTML.

>> I'm not sure what you consider "the most risky" parts of the spec to be
>
> The navigation and event queue bits.  I did say that...

Oh wow, I missed that whole paragraph. Sorry.

>> but in my experience the parts least likely to match reality in their
>> details are the parts documenting the most fundamental behaviour of the
>> web.
>
> Indeed.  Like navigation.
>
>> For example the specification of session history is -- I think --
>> considerably more buggy than the specification of the 2D context or many
>> other new features.
>
> Yes.  Session history is definitely part of the whole navigation mess.
>
>> nor are people likely to be happy stalling waiting for
>> browsers to rewrite such fundamental code to get to Rec.
>
> Why not?  Why bother going to Rec with a spec we _know_ is bogus?

Well I guess it depends what you think the value of a Rec. is. If you 
think that it's basically IPR related then there is value in getting 
there even if the details of the spec aren't all correct.

It seems to me that people who take that view are also the most likely 
to stop contributing tests at all if there is no Rec. incentive; people 
who think that the value of Rec. is in reaching a level of 
interoperability can still aim to improve interoperability by writing 
and publishing tests even when it isn't coupled to a Process transition.

>> So there will be strong pressure to just drop those tests from the
>> testsuite "for this
>> version".
>
> In which case we've failed and might as well go home.  Which _is_ being
> pretty tempting, for sure.

If your claim is that the W3C Process doesn't work, then I can't say I 
disagree. I would much prefer a process where the spec — much like 
browsers — underwent continual development and there were time based 
snapshots for lawyers. Figuring out a motivation beyond "improving the 
web as a platform" for people to write and contribute tests in this 
scenario is of course more difficult.

>> So I don't really forsee a testsuite written for the purpose
>> of proving interoperability actually helping interoperability as much as
>> a testsuite written for the purpose or finding bugs.
>
> There seems to be 0 incentive to write the latter, unfortunately.  :(

You don't think that you benefit as browser interoperability improves?

[1] http://www.w3.org/html/wg/wiki/Testing/Submission/

Received on Thursday, 16 August 2012 08:32:55 UTC