Re: HTML5 RDFa Test Suite (was Re: Request to publish HTML+RDFa (draft 3) as FPWD)

Maciej Stachowiak wrote:
>> This test suite was upgraded this past weekend to cover HTML5 and
>> contains 127 unit tests specifically for HTML5. There are also at least
>> three implementations that are close to 100% conformant with HTML+RDFa
>> (the PyRDFa processor, the MarkLogic processor, and the librdfa
>> processor).
> 
> I don't think this demonstrates that the spec is unambiguous. 

I wasn't trying to demonstrate that the spec is ambiguous. I was trying
to demonstrate that it is possible to create a conforming RDFa processor
by following the spec and running a processor through the test suite.

> Can you relate every test case back to normative conformance 
> requirements in the spec?

If I understand what you're asking, we could make it more explicit by
adding the section of the RDFa spec that applies to each test case into
the test case manifest for each language.

Is this going to be done for the HTML5 spec as well?

> Can you relate every implementation conformance requirement back
> to a test in the test suite?

We could add this to the test case manifest as well.

The purpose of the test suite was to get good coverage of all major
features of RDFa as well as catching potential implementation bugs.

>> There are also 7 XHTML+RDFa processors implemented in non-browser
>> technologies and 2 XHTML+RDFa processors implemented in Javascript in
>> the browser.
> 
> Given the serious errors in the specification for prefix mapping in the
> XHTML+RDFa spec, it's pretty clear that in that case, no one was coding
> to just the language of the spec. They either used out of band
> information about what it's really supposed to mean, or coded to the
> test suite instead of to the spec. Others (e.g. Phillip Taylor) have
> pointed out errors in the test suite itself where tests don't match what
> the spec requires, but many implementations pass the tests. This would
> seem to indicate coding to the test suite instead of to the spec. Or
> conversely, it may indicate that the test suite was written to match
> what implementations do, instead of testing what the spec requires.

The test suite was written to test what the spec requires. Additional
tests were added by implementers as they implemented their RDFa
processor and debugged their implementations. We have no control over
whether or not developers coded to the test suite or to the spec... but
if you pass the test suite, more than likely you're doing something right.

> Then let's make the spec accurately and precisely state the requirements
> tested by the test suite. 

Agreed. We've been doing that for over two years, since the inception of
the test suite, and we will keep doing it as we move forward with HTML+RDFa.

> And when people raise confusing edge cases,
> let's turn them into tests and make sure the spec accurately reflects
> the desired behavior. 

I whole-heartedly agree with you, as I'm sure the rest of the RDFa
Community does. Let me remind everyone that the RDFa test suite is
completely open source, is git-based, and allows anybody to contribute
test cases for RDFa (via git format-patch, or github merges):

http://github.com/msporny/rdfa-test-suite

If there is a feature that you feel stumps the RDFa processing rules,
please write the XHTML up and submit it. I can write up the SPARQL query
if you don't know SPARQL. Don't wait for us to get around to doing it -
you can do it yourself.

Philip - my plate is full right now. If you could write up your tests
and submit them, that would be great. If not, it'll be a couple of weeks
before I will be able to get to them (since I do all of this in my free
time).

> For example, how do implementations behave for
> Jonas's <a> inside a <table> example? How do browser-hosted
> implementations behave when xmlns: attributes in the DOM have been
> modified after parsing? Can that behavior be derived from the spec?

If the behavior can't be derived from the spec now, it should be and we
should update the spec to reflect those clarifications.

I believe that we cover Jonas's <a> inside a <table> example with
Section 2, but if we don't, we should add more text to be more precise.

The browser-hosted dueling xmlns: attributes in the DOM issue is
something that we should clarify in the spec.

xmlns: DOM modifications are not covered in the HTML+RDFa spec, but we
can add language to clarify what to do if an xmlns: value changes in the
DOM.

> It would be much
> more convincing to add the requested precision, or if it's really truly
> there, to walk through examples like Jonas's and explain what normative
> statements describe how to process it.

This discussion has clarified a number of things, for all involved. So,
it was a useful discussion to have.

If you think that anyone in the RDFa Task Force or community is taking a
hard-line stance against making the HTML+RDFa spec easier to understand,
then that could not be further from the truth. We have to go through
these conversations to learn the issues with the spec text. Clearly,
Jonas learned several things during the conversation as did we, thus the
conversation was fruitful.

Jonas clearly had no idea that there were already a number of Javascript
implementations when he first raised the DOM concern. Many didn't know
that the RDFa test suite existed or what it contained or what it tested.
It takes time to walk through the examples and point out the normative
statements. I believe that Shane did that, but that doesn't mean that
Jonas or others took the time to read the entire XHTML+RDFa or HTML+RDFa
spec and fully understand every sentence in there.

We must first understand the requested precision that is desired and how
we can make it clarify the specification without duplicating content or
creating an implementation issue or breaking backwards compatibility.
These things take time and discussions like this.

In the next e-mail, I'll try and wrap this up into a way forward that
will address everyone's concerns.

-- manu

-- 
Manu Sporny (skype: msporny, twitter: manusporny)
President/CEO - Digital Bazaar, Inc.
blog: The Pirate Bay and Building an Equitable Culture
http://blog.digitalbazaar.com/2009/08/30/equitable-culture/

Received on Wednesday, 23 September 2009 04:58:20 UTC