RE: Making the annotation tests more informative

Is this now in line with what you are recommending?
 
http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationMusts-manual.html 
 
(Note, a couple of the assertion schema remain to be debugged.)
 
-Tim Cole
 
From: Shane McCarron [mailto:shane@spec-ops.io] 
Sent: Friday, August 12, 2016 11:54 AM
To: W3C Public Annotation List <public-annotation@w3.org>
Subject: Making the annotation tests more informative
 
As a result of today's meeting, I have made a couple of changes to the test framework.  These are available now on the testdev sandbox and will be in WPT as soon as they are merged:
 
1.      errorMessage was not being included in the message output when an assertion failed.  It is now.
2.      assertionType was not being used.  This optional notation on an assertion was put in the grammar as a way of helping us inform the test framework what to do, but never got fully developed. We are now using the information to augment the test documentation and output:

"must" means an assertion is "[MANDATORY]" and a failure is an ERROR
"should" means an assertion is "[RECOMMENDED]" and a failure is a WARNING
"may" means an assertion is "[OPTIONAL]" and a failure is INFORMATIONAL
In order to take advantage of these changes and make the tests more usable, I recommend changing the assertion titles so that they do not include the words "Check that" or "Check for".  Make the titles just be clean active voice, present tense summary of what is required for the test to pass.  For example:
*         'body' is a String
*         '@context' contains http://...
*         'textDirection' is used once with a value in ['ltr', 'rtl']
Also, note that it is okay to embed HTML into .test file 'description' fields.  The content of these fields is dropped into the template for each manual test using the following structure:
 
<p>Fill the textarea below with JSON output from your annotation client
implementation that supports the following criteria:</p>
<div id="testDescription"></div>
<p>Specifically, the following assertions will be evaluated:</p>
<div id="assertion"></div>
 
So, if we have a more friendly description field in each test, and consistently structured assertion titles, when we generate the instructions for people running manual tests it will look like:
 
Fill the textarea below with JSON output from your annotation client implementation that supports the following criteria: 
 
 An annotation that points to an external web resource for the body of the annotation. 
 
Specifically, the following assertions will be evaluated: 
 
1.      [RECOMMENDED] 'body' is a String
2.      [MANDATORY] '@context' contains http://...
3.      [OPTIONAL] 'textDirection' is used once with with a value in ['ltr', 'rtl']
We can obviously tune this but I think it is a good start toward making the manual tests more legible.
 
-- 
Shane McCarron
Projects Manager, Spec-Ops