Re: Consolidating css-wg and web-platform-tests repositories (Was: test suite meta data)

On Aug 6, 2013, at 3:38 PM, James Graham wrote:

> On 2013-08-06 14:39, Linss, Peter wrote:
>> On Aug 6, 2013, at 12:40 PM, James Graham wrote:
>>> On 2013-08-06 11:39, Peter Linss wrote:
>>>> On Aug 6, 2013, at 11:14 AM, James Graham wrote:
>>>>> Where does the requirement to have the full suite in multiple formats come from? It seems unlikely that the CSS layer in browsers would depend on the parser that was originally used. Do you have examples of tests that found bugs when run in XML but not in HTML?
>>>> It's not so much to test a browser's behavior in both input formats,
>>>> but to make the suite available for clients that don't support one
>>>> format or the other. Clients which we needed to exit CR for CSS2.1 and
>>>> will likely need again for other specs. For example, some of the
>>>> implementations are offline XHTML to PDF converters or XHTML-Print
>>>> renderers embedded in printers. This is particularly true for
>>>> paged-media CSS features that are generally poorly supported in
>>>> browsers (but I'd be more than happy if we could rely on browsers to
>>>> pass those tests).
>>> So essentially all this complexity is to support non-web use cases? That seems unfortunate.
>> Sorry, but web technologies are used in places outside browsers, that
>> doesn't make it "non-web".
> 
> Well that's a matter of definitions, but I would consider things like consumer electronics device UIs to be "non-web", even if they happen to be implemented in a web browser.
> 
>> It's also not unfortunate that this
>> happens, it's a good thing. Pretending those use cases don't exist is
>> unfortunate.
> 
> I'm not pretending that those use cases don't exist. But I don't think that we should optimise for the non-web cases at the expense of the simplicity for the web cases, or even edge cases of the web at expense of the more common cases. For example I don't think we should require that DOM tests also work for Java implementations of the DOM, and I don't think that we should require all tests work in monochrome just so that eink-based ebooks can be tested.

No disagreement here, but that's not exactly the situation either.

> 
>> But that's all beside the point, we're not actually going out of our
>> way to support "non-web" use cases, we're supporting getting our specs
>> to REC.
> 
> It's unfortunate that you still see the testing primarily as a tool to get specs to move along the Rec. track. I think we should see testing as a way of improving interop. and strengthening the platform. After all the Process is a means to an end and non an end in itself.

No, I see that as phase one of a spec's testing effort. Once the spec exits CR then the focus of the test suite shifts to conformance testing, I've always said so. But as co-chair of a WG, phase one has my personal priority.

My fundamental point here is that if the test suite can't get a spec out of CR, then it has little utility to the WG developing the spec, who, at the end of the day, needs the specs to advance if they want their charter to get renewed or be able to work on "the next cool thing". Building an entirely new testing infrastructure that can't get a spec out of CR is, IMO, a big waste of time, and not something I'm signing on to help with. Don't get me wrong, I see the value of testing regardless, but if getting specs past CR isn't a primary focus of *this* effort, then we have a serious problem. 

> 
>> The fact that it takes implementations other than browsers to
>> do that _is_ unfortunate, but it's what we have had to do, and will
>> continue to need to do for some time yet. If you can guarantee
>> implementations of all of CSS in two browsers, I'd be happy to drop
>> these extra requirements. The day we can do that will be a good day.
> 
> Perhaps the build system could be dropped for modules that are actually implemented in browsers then? I guess that will be the vast majority.

It's possible that the format conversion can be dropped for some test suites, but, as I've said before, the build process does more than that, and the benefits are worth the cost. If they weren't I wouldn't be spending my time on it, there are other things I'd rather be building.

I fully accept that some test suites will live just fine without a build step. I am asserting, however, that some will not. As long as one test suite needs it, a build step has to be accounted for in the process. And as long as we have one, we might as well get as many benefits from it as we can (like auto-generating manifests, rather than trying to keep separate files in sync, generating .htaccess files from .header files, etc). I don't want a build step for the sake of having one, believe me, but it serves a purpose and we can't simply throw it away (at least not without a suitable replacement for the things it provides).

I've also believe that once other WGs see what the build code can produce for them (once the re-work is complete), they'll opt-in and start using it too. For example, as more specs become modular, and start having multiple versions, the build code can generate new test suites by intelligently (and mostly automatically) incorporating tests from other existing test suites. But I'm not trying, by any means, to force it on anyone who doesn't need it.

Received on Wednesday, 7 August 2013 00:19:46 UTC