Re: Making the annotation tests more informative

Actually, it turns out it was a bug.  I have pushed in a PR for the fix and
dropped the updated version into the tree on the server so you can use it
Tim.  Sorry!

On Thu, Aug 18, 2016 at 10:16 AM, Shane McCarron <shane@spec-ops.io> wrote:

> Tim et. al.,
>
> I am going to rebuild the testdev server.  I am not quite sure what is up,
> but there are changes to the tree that are not reflect there.  Some
> conflict with the auto-updating deployments from github stuff I expect.
>
> On Thu, Aug 18, 2016 at 9:49 AM, Shane McCarron <shane@spec-ops.io> wrote:
>
>> I will look into it now.
>>
>> On Wed, Aug 17, 2016 at 9:49 PM, Cole, Timothy W <t-cole3@illinois.edu>
>> wrote:
>>
>>> Shane-
>>>
>>> The test runner doesn't seem to be picking up assertionType:should or
>>> assertionType:may. See:
>>>
>>> http://testdev.spec-ops.io:8000/annotation-model/annotations
>>> /annotationOptionals-manual.html
>>>
>>> All of the assertion files referenced by this test are one of these 2
>>> assertionTypes...
>>>
>>> Please note I still need to correct a few minor assertion bugs. Will do
>>> so in the morning, and then begin adding the body and target assertions and
>>> tests.
>>>
>>> -Tim Cole
>>>
>>>
>>> ------------------------------
>>> *From:* Cole, Timothy W
>>> *Sent:* Wednesday, August 17, 2016 12:03
>>> *To:* 'Shane McCarron'; 'W3C Public Annotation List'
>>> *Subject:* RE: Making the annotation tests more informative
>>>
>>> Is this now in line with what you are recommending?
>>>
>>>
>>>
>>> http://testdev.spec-ops.io:8000/annotation-model/annotations
>>> /annotationMusts-manual.html
>>>
>>>
>>>
>>> (Note, a couple of the assertion schema remain to be debugged.)
>>>
>>>
>>>
>>> -Tim Cole
>>>
>>>
>>>
>>> *From:* Shane McCarron [mailto:shane@spec-ops.io]
>>> *Sent:* Friday, August 12, 2016 11:54 AM
>>> *To:* W3C Public Annotation List <public-annotation@w3.org>
>>> *Subject:* Making the annotation tests more informative
>>>
>>>
>>>
>>> As a result of today's meeting, I have made a couple of changes to the
>>> test framework.  These are available now on the testdev sandbox and will be
>>> in WPT as soon as they are merged:
>>>
>>>
>>>
>>> 1.      errorMessage was not being included in the message output when
>>> an assertion failed.  It is now.
>>>
>>> 2.      assertionType was not being used.  This optional notation on an
>>> assertion was put in the grammar as a way of helping us inform the test
>>> framework what to do, but never got fully developed. We are now using the
>>> information to augment the test documentation and output:
>>>
>>> "must" means an assertion is "*[MANDATORY]*" and a failure is an ERROR
>>> "should" means an assertion is "*[RECOMMENDED]*" and a failure is a
>>> WARNING
>>> "may" means an assertion is "*[OPTIONAL]*" and a failure is
>>> INFORMATIONAL
>>>
>>> In order to take advantage of these changes and make the tests more
>>> usable, I recommend changing the assertion titles so that they do not
>>> include the words "Check that" or "Check for".  Make the titles just be
>>> clean active voice, present tense summary of what is required for the test
>>> to pass.  For example:
>>>
>>> ·         'body' is a String
>>>
>>> ·         '@context' contains http:// <http://UrlBlockedError.aspx>...
>>>
>>> ·         'textDirection' is used once with a value in ['ltr', 'rtl']
>>>
>>> Also, note that it is okay to embed HTML into .test file 'description'
>>> fields.  The content of these fields is dropped into the template for each
>>> manual test using the following structure:
>>>
>>>
>>>
>>> <p>Fill the textarea below with JSON output from your annotation client
>>>
>>> implementation that supports the following criteria:</p>
>>>
>>> <div id="testDescription"></div>
>>>
>>> <p>Specifically, the following assertions will be evaluated:</p>
>>>
>>> <div id="assertion"></div>
>>>
>>>
>>>
>>> So, if we have a more friendly description field in each test, and
>>> consistently structured assertion titles, when we generate the instructions
>>> for people running manual tests it will look like:
>>>
>>>
>>>
>>> Fill the textarea below with JSON output from your annotation client
>>> implementation that supports the following criteria:
>>>
>>>
>>>
>>>  An annotation that points to an external web resource for the body of
>>> the annotation.
>>>
>>>
>>>
>>> Specifically, the following assertions will be evaluated:
>>>
>>>
>>>
>>> 1.      *[RECOMMENDED] *'body' is a String
>>>
>>> 2.      *[MANDATORY] *'@context' contains http://
>>> <http://UrlBlockedError.aspx>...
>>>
>>> 3.      *[OPTIONAL]* 'textDirection' is used once with with a value in
>>> ['ltr', 'rtl']
>>>
>>> We can obviously tune this but I think it is a good start toward making
>>> the manual tests more legible.
>>>
>>>
>>>
>>> --
>>>
>>> Shane McCarron
>>>
>>> Projects Manager, Spec-Ops
>>>
>>
>>
>>
>> --
>> Shane McCarron
>> Projects Manager, Spec-Ops
>>
>
>
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>



-- 
Shane McCarron
Projects Manager, Spec-Ops