Re: Testing CSS tokenization

On Mon, Jun 17, 2013 at 1:40 PM, Simon Sapin <simon.sapin@exyr.org> wrote:
> Hi,
>
> So, I’ve been implementing the new CSS Syntax draft:
>
> http://dev.w3.org/csswg/css-syntax/
> https://github.com/SimonSapin/tinycss2
>
> And of course, I’m writing tests. This time around, I’m writing them in JSON
> in the hope to share them in multiple implementations. (I’m working on at
> least two of them myself.)
>
> https://github.com/SimonSapin/tinycss2/tree/master/tinycss2/tests
> (This is very much "in progress", but I hope you get the idea.)
>
>
> In order to test tokenization, I really want to look at the output of the
> tokenizer. (Actually the output of "Parse a list of component values", which
> is close enough.) Unfortunately, the tokenizer is not exposed on the web. I
> can not take these tests and run them in any browsers; this would require
> writing custom code against the browser’s source code. I understand this is
> not acceptable for a W3C spec’s test suite, but anything else has to be much
> higher level and I don’t find it very satisfying.
>
> For example, many corner cases of the tokenizer are not tested AFAICT by the
> 2.1:
>
> http://hg.csswg.org/test/file/63d1b51beff2/approved/css2.1/src/syntax
>
>
> What do you think? What is the way to test a spec such as CSS Syntax whose
> product is not directly exposed to the web platform?

Indirectly!  There's no general-purpose way to test it, but there are
plenty of properties which accept different kinds of tokens, such that
you should be able to find something that can test all or nearly all
pairs of tokens.

I'd thought about exposing a tokenizing interface directly, as it
could be useful for web-based CSS parsers, but I don't think it's a
good idea.  Precise tokenization is rarely important for web content,
but if we expose it directly it's almost certain that scripts will end
up depending on it accidentally, making it difficult to change later.

~TJ

Received on Tuesday, 18 June 2013 06:54:15 UTC