RE: Making the annotation tests more informative

Shane-

The test runner doesn't seem to be picking up assertionType:should or assertionType:may. See:

http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationOptionals-manual.html

All of the assertion files referenced by this test are one of these 2 assertionTypes...

Please note I still need to correct a few minor assertion bugs. Will do so in the morning, and then begin adding the body and target assertions and tests.

-Tim Cole


________________________________
From: Cole, Timothy W
Sent: Wednesday, August 17, 2016 12:03
To: 'Shane McCarron'; 'W3C Public Annotation List'
Subject: RE: Making the annotation tests more informative

Is this now in line with what you are recommending?

http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationMusts-manual.html

(Note, a couple of the assertion schema remain to be debugged.)

-Tim Cole

From: Shane McCarron [mailto:shane@spec-ops.io]
Sent: Friday, August 12, 2016 11:54 AM
To: W3C Public Annotation List <public-annotation@w3.org>
Subject: Making the annotation tests more informative

As a result of today's meeting, I have made a couple of changes to the test framework.  These are available now on the testdev sandbox and will be in WPT as soon as they are merged:

1.      errorMessage was not being included in the message output when an assertion failed.  It is now.
2.      assertionType was not being used.  This optional notation on an assertion was put in the grammar as a way of helping us inform the test framework what to do, but never got fully developed. We are now using the information to augment the test documentation and output:

"must" means an assertion is "[MANDATORY]" and a failure is an ERROR
"should" means an assertion is "[RECOMMENDED]" and a failure is a WARNING
"may" means an assertion is "[OPTIONAL]" and a failure is INFORMATIONAL
In order to take advantage of these changes and make the tests more usable, I recommend changing the assertion titles so that they do not include the words "Check that" or "Check for".  Make the titles just be clean active voice, present tense summary of what is required for the test to pass.  For example:
·         'body' is a String
·         '@context' contains http://<UrlBlockedError.aspx>...
·         'textDirection' is used once with a value in ['ltr', 'rtl']
Also, note that it is okay to embed HTML into .test file 'description' fields.  The content of these fields is dropped into the template for each manual test using the following structure:

<p>Fill the textarea below with JSON output from your annotation client
implementation that supports the following criteria:</p>
<div id="testDescription"></div>
<p>Specifically, the following assertions will be evaluated:</p>
<div id="assertion"></div>

So, if we have a more friendly description field in each test, and consistently structured assertion titles, when we generate the instructions for people running manual tests it will look like:

Fill the textarea below with JSON output from your annotation client implementation that supports the following criteria:

 An annotation that points to an external web resource for the body of the annotation.

Specifically, the following assertions will be evaluated:

1.      [RECOMMENDED] 'body' is a String
2.      [MANDATORY] '@context' contains http://<UrlBlockedError.aspx>...
3.      [OPTIONAL] 'textDirection' is used once with with a value in ['ltr', 'rtl']
We can obviously tune this but I think it is a good start toward making the manual tests more legible.

--
Shane McCarron
Projects Manager, Spec-Ops