Re: progress on schemas for model testing

On Sat, 2016-07-30 at 22:13 +0000, Cole, Timothy W wrote:
> 
[...]
> Hmm... I think we need to decide what a "feature" is for CR
> purposes.  For example, textDirection does not feel like a feature to
> me.  It is just an attribute of the annotation.  There are lots of
> attributes, right?  I think that for CR the feature granularity
> should be at a higher level.  Do annotation clients support the use
> of bodyValue?  of body? In body, which body types are used?  That's
> enough.

Usually the W3C Director (or delegated staff ember) is looking for fine
enough granularity that we have a good sense that the spec can be
implemented interoperably by groups outside W3C in the future.

If it turns out that textDirection simply doesn't work for some reason,
we want to know that before publishing it in a Recommendation.

I think for XQuery we had around 20,000 test cases. Some other specs
have had as few as a couple of dozen test cases. There isn't a single
answer.

> If I understand correctly, we
> generally do not need to test whether SHOULD or MAY keys have been
> implemented by anyone, at least not for the purposes of exiting CR.

If we say someone SHOULD do something and it turns out to be impossible
to make a conforming implementation that behaves in that way, we'd look
pretty silly - and the SHOULD would need to be struck from the spec.

Optional features generally need only one implementation to pass all
their tests, but I'd for sure expect several hundred test cases at
least for Web annotations.

>  For what it's worth, I note that the textDirection key was added to
> the model late in response to feedback from Internationalization

This makes it more important to test it.

>  that there could be annotation use cases requiring this information
> explicitly.  I don't think we have anyone to this point who's
> implemented such a use case,

Then it should be removed from the draft, or there should be test cases
for it and an implementation found (or made).


> Do need clarification on two points, however:
> 
> A. Given that we are not tracking if keys like textDirection have
> been implemented, do we even need to check (again, just for the
> purposes of documenting feature implementation in order to exit CR)
> that if implemented, the value of textDirection if present is
> correct?

If it's not correct, what behaviour are you expecting exactly? Can you
test to see if an implementation behaves in that way?

>  it's actually not practical to use JSON schema to check validity
> (both MUST and SHOULD) of many key values (e.g., not practical to
> verify that all values of the language key conform to bcp47).

Convert them to XML and use XSD instead??

>  So if in order to satisfy CR exit requirements, we don't need to
> include any test relating to textDirection (or its value) when
> implemented, so much the better.
Remember of course that it's actual implementations that are being
tested for CR purposes, not the test cases... and what we're testing is
really that everything in the spec can be implemented.


Hope this helps.

Liam

Received on Sunday, 31 July 2016 00:02:15 UTC