Re: progress on schemas for model testing

Liam,

Some notes inline:

On Sat, Jul 30, 2016 at 7:02 PM, Liam R. E. Quin <liam@w3.org> wrote:

> On Sat, 2016-07-30 at 22:13 +0000, Cole, Timothy W wrote:
> >
> [...]
> > Hmm... I think we need to decide what a "feature" is for CR
> > purposes.  For example, textDirection does not feel like a feature to
> > me.  It is just an attribute of the annotation.  There are lots of
> > attributes, right?  I think that for CR the feature granularity
> > should be at a higher level.  Do annotation clients support the use
> > of bodyValue?  of body? In body, which body types are used?  That's
> > enough.
>
> Usually the W3C Director (or delegated staff ember) is looking for fine
> enough granularity that we have a good sense that the spec can be
> implemented interoperably by groups outside W3C in the future.
>
> If it turns out that textDirection simply doesn't work for some reason,
> we want to know that before publishing it in a Recommendation.
>

I don't think this is a case of doesn't work.  The specification says what
an annotation client SHOULD do when there is an annotation that has a
non-default text direction.  Clearly that is relatively easy to test
(provide an annotation where the body of the annotation uses a right to
left character set).  No idea if any clients actually indicate the text
direction of the annotation.   But I don't think it actually matters for
interoperability...


> I think for XQuery we had around 20,000 test cases. Some other specs
> have had as few as a couple of dozen test cases. There isn't a single
> answer.
>

Good point.


>
> > If I understand correctly, we
> > generally do not need to test whether SHOULD or MAY keys have been
> > implemented by anyone, at least not for the purposes of exiting CR.
>
> If we say someone SHOULD do something and it turns out to be impossible
> to make a conforming implementation that behaves in that way, we'd look
> pretty silly - and the SHOULD would need to be struck from the spec.
>

Yes.  The SHOULDs in these specs are not things that would be challenging
to implement.  Rather they are optional parts of the data model and the
spec is proscribing the way in which the data MUST be provided if the
feature is present.  So an implementation that included a textDirection key
with an undefined value would be non-conforming.  But an implementation
that provided no textDirection value at all would be just fine.


>
> Optional features generally need only one implementation to pass all
> their tests, but I'd for sure expect several hundred test cases at
> least for Web annotations.
>

Interesting.  I expect on the order of 20 actual test cases with many
sub-tests per test case.  The Annotation Model has almost no actual
conformance requirements.  In terms of features, the discrete ones seem to
be about the various types of things that might be annotated, and the ways
in which those things might be referenced.  I expect there will be tests
for each of these so that we can determine if those features are actually
implemented anywhere.


> >  For what it's worth, I note that the textDirection key was added to
> > the model late in response to feedback from Internationalization
>
> This makes it more important to test it.
>

I disagree.  You are thinking of this data model as if it were some sort of
an API that needs to be implemented and exercised.  It isn't that at all.
It's just data structures.  And those structures have A LOT of optional
aspects.


>
> >  that there could be annotation use cases requiring this information
> > explicitly.  I don't think we have anyone to this point who's
> > implemented such a use case,
>
> Then it should be removed from the draft, or there should be test cases
> for it and an implementation found (or made).
>

Hmm.  I appreciate what you are saying here, but why?  What's wrong with
defining a feature in advance of implementation when it is optional?  The
conformance tests still exist and, should someone implement it, when they
test their implementation it will not pass if it is broken .  And they will
fix it.  The alternative is that people see a need, implement this same
thing, do it differently, and then we need to find some way to compromise.


>
>
> > Do need clarification on two points, however:
> >
> > A. Given that we are not tracking if keys like textDirection have
> > been implemented, do we even need to check (again, just for the
> > purposes of documenting feature implementation in order to exit CR)
> > that if implemented, the value of textDirection if present is
> > correct?
>
> If it's not correct, what behaviour are you expecting exactly? Can you
> test to see if an implementation behaves in that way?
>

We can and will test to see if it is implemented correctly.


>
> >  it's actually not practical to use JSON schema to check validity
> > (both MUST and SHOULD) of many key values (e.g., not practical to
> > verify that all values of the language key conform to bcp47).
>
> Convert them to XML and use XSD instead??
>

Umm... no.  And it is easy to test that the values of a language key
conform to bcp47.  I am surprised that Tim thinks otherwise.  It is just a
list of strings in an enum that the schema validator will test against.
Simple.


>
> >  So if in order to satisfy CR exit requirements, we don't need to
> > include any test relating to textDirection (or its value) when
> > implemented, so much the better.
> Remember of course that it's actual implementations that are being
> tested for CR purposes, not the test cases... and what we're testing is
> really that everything in the spec can be implemented.
>

That everything MANDATORY can be implemented.  At least, that's my belief.
If there is a MAY in the spec, for example, then I don't care if it is
implemented.  I won't test for it.  If there is a SHOULD I care that it is
implemented correctly when it is implemented, but if it were not
implemented anywhere that I looked.... I still wouldn't care.


>
>
> Hope this helps.
>
> Liam
>
>


-- 
Shane McCarron
Projects Manager, Spec-Ops