Re: Making the annotation tests more informative

I will look into it now.

On Wed, Aug 17, 2016 at 9:49 PM, Cole, Timothy W <t-cole3@illinois.edu>
wrote:

> Shane-
>
> The test runner doesn't seem to be picking up assertionType:should or
> assertionType:may. See:
>
> http://testdev.spec-ops.io:8000/annotation-model/annotations/
> annotationOptionals-manual.html
>
> All of the assertion files referenced by this test are one of these 2
> assertionTypes...
>
> Please note I still need to correct a few minor assertion bugs. Will do so
> in the morning, and then begin adding the body and target assertions and
> tests.
>
> -Tim Cole
>
>
> ------------------------------
> *From:* Cole, Timothy W
> *Sent:* Wednesday, August 17, 2016 12:03
> *To:* 'Shane McCarron'; 'W3C Public Annotation List'
> *Subject:* RE: Making the annotation tests more informative
>
> Is this now in line with what you are recommending?
>
>
>
> http://testdev.spec-ops.io:8000/annotation-model/
> annotations/annotationMusts-manual.html
>
>
>
> (Note, a couple of the assertion schema remain to be debugged.)
>
>
>
> -Tim Cole
>
>
>
> *From:* Shane McCarron [mailto:shane@spec-ops.io]
> *Sent:* Friday, August 12, 2016 11:54 AM
> *To:* W3C Public Annotation List <public-annotation@w3.org>
> *Subject:* Making the annotation tests more informative
>
>
>
> As a result of today's meeting, I have made a couple of changes to the
> test framework.  These are available now on the testdev sandbox and will be
> in WPT as soon as they are merged:
>
>
>
> 1.      errorMessage was not being included in the message output when an
> assertion failed.  It is now.
>
> 2.      assertionType was not being used.  This optional notation on an
> assertion was put in the grammar as a way of helping us inform the test
> framework what to do, but never got fully developed. We are now using the
> information to augment the test documentation and output:
>
> "must" means an assertion is "*[MANDATORY]*" and a failure is an ERROR
> "should" means an assertion is "*[RECOMMENDED]*" and a failure is a
> WARNING
> "may" means an assertion is "*[OPTIONAL]*" and a failure is INFORMATIONAL
>
> In order to take advantage of these changes and make the tests more
> usable, I recommend changing the assertion titles so that they do not
> include the words "Check that" or "Check for".  Make the titles just be
> clean active voice, present tense summary of what is required for the test
> to pass.  For example:
>
> ·         'body' is a String
>
> ·         '@context' contains http:// <http://UrlBlockedError.aspx>...
>
> ·         'textDirection' is used once with a value in ['ltr', 'rtl']
>
> Also, note that it is okay to embed HTML into .test file 'description'
> fields.  The content of these fields is dropped into the template for each
> manual test using the following structure:
>
>
>
> <p>Fill the textarea below with JSON output from your annotation client
>
> implementation that supports the following criteria:</p>
>
> <div id="testDescription"></div>
>
> <p>Specifically, the following assertions will be evaluated:</p>
>
> <div id="assertion"></div>
>
>
>
> So, if we have a more friendly description field in each test, and
> consistently structured assertion titles, when we generate the instructions
> for people running manual tests it will look like:
>
>
>
> Fill the textarea below with JSON output from your annotation client
> implementation that supports the following criteria:
>
>
>
>  An annotation that points to an external web resource for the body of the
> annotation.
>
>
>
> Specifically, the following assertions will be evaluated:
>
>
>
> 1.      *[RECOMMENDED] *'body' is a String
>
> 2.      *[MANDATORY] *'@context' contains http://
> <http://UrlBlockedError.aspx>...
>
> 3.      *[OPTIONAL]* 'textDirection' is used once with with a value in
> ['ltr', 'rtl']
>
> We can obviously tune this but I think it is a good start toward making
> the manual tests more legible.
>
>
>
> --
>
> Shane McCarron
>
> Projects Manager, Spec-Ops
>



-- 
Shane McCarron
Projects Manager, Spec-Ops