Making the annotation tests more informative

As a result of today's meeting, I have made a couple of changes to the test
framework.  These are available now on the testdev sandbox and will be in
WPT as soon as they are merged:


   1. errorMessage was not being included in the message output when an
   assertion failed.  It is now.
   2. assertionType was not being used.  This optional notation on an
   assertion was put in the grammar as a way of helping us inform the test
   framework what to do, but never got fully developed. We are now using the
   information to augment the test documentation and output:

   "must" means an assertion is "*[MANDATORY]*" and a failure is an ERROR
   "should" means an assertion is "*[RECOMMENDED]*" and a failure is a
   WARNING
   "may" means an assertion is "*[OPTIONAL]*" and a failure is INFORMATIONAL

In order to take advantage of these changes and make the tests more usable,
I recommend changing the assertion titles so that they do not include the
words "Check that" or "Check for".  Make the titles just be clean active
voice, present tense summary of what is required for the test to pass.  For
example:

   - 'body' is a String
   - '@context' contains http://...
   - 'textDirection' is used once with a value in ['ltr', 'rtl']

Also, note that it is okay to embed HTML into .test file 'description'
fields.  The content of these fields is dropped into the template for each
manual test using the following structure:

<p>Fill the textarea below with JSON output from your annotation client
implementation that supports the following criteria:</p>
<div id="testDescription"></div>
<p>Specifically, the following assertions will be evaluated:</p>
<div id="assertion"></div>

So, if we have a more friendly description field in each test, and
consistently structured assertion titles, when we generate the instructions
for people running manual tests it will look like:

Fill the textarea below with JSON output from your annotation client
> implementation that supports the following criteria:



>  An annotation that points to an external web resource for the body of the
> annotation.



> Specifically, the following assertions will be evaluated:



>
>    1. *[RECOMMENDED] *'body' is a String
>    2. *[MANDATORY] *'@context' contains http://...
>    3. *[OPTIONAL]* 'textDirection' is used once with with a value in
>    ['ltr', 'rtl']
>
> We can obviously tune this but I think it is a good start toward making
the manual tests more legible.

-- 
Shane McCarron
Projects Manager, Spec-Ops

Received on Friday, 12 August 2016 16:54:48 UTC