- From: Shane McCarron <shane@spec-ops.io>
- Date: Thu, 18 Aug 2016 10:16:51 -0500
- To: "Cole, Timothy W" <t-cole3@illinois.edu>
- Cc: W3C Public Annotation List <public-annotation@w3.org>
- Message-ID: <CAJdbnOD9nRKsvxp15OZSHM5HX+7iubgDY7TrbWVOyNY0PLqkSg@mail.gmail.com>
Tim et. al., I am going to rebuild the testdev server. I am not quite sure what is up, but there are changes to the tree that are not reflect there. Some conflict with the auto-updating deployments from github stuff I expect. On Thu, Aug 18, 2016 at 9:49 AM, Shane McCarron <shane@spec-ops.io> wrote: > I will look into it now. > > On Wed, Aug 17, 2016 at 9:49 PM, Cole, Timothy W <t-cole3@illinois.edu> > wrote: > >> Shane- >> >> The test runner doesn't seem to be picking up assertionType:should or >> assertionType:may. See: >> >> http://testdev.spec-ops.io:8000/annotation-model/annotations >> /annotationOptionals-manual.html >> >> All of the assertion files referenced by this test are one of these 2 >> assertionTypes... >> >> Please note I still need to correct a few minor assertion bugs. Will do >> so in the morning, and then begin adding the body and target assertions and >> tests. >> >> -Tim Cole >> >> >> ------------------------------ >> *From:* Cole, Timothy W >> *Sent:* Wednesday, August 17, 2016 12:03 >> *To:* 'Shane McCarron'; 'W3C Public Annotation List' >> *Subject:* RE: Making the annotation tests more informative >> >> Is this now in line with what you are recommending? >> >> >> >> http://testdev.spec-ops.io:8000/annotation-model/annotations >> /annotationMusts-manual.html >> >> >> >> (Note, a couple of the assertion schema remain to be debugged.) >> >> >> >> -Tim Cole >> >> >> >> *From:* Shane McCarron [mailto:shane@spec-ops.io] >> *Sent:* Friday, August 12, 2016 11:54 AM >> *To:* W3C Public Annotation List <public-annotation@w3.org> >> *Subject:* Making the annotation tests more informative >> >> >> >> As a result of today's meeting, I have made a couple of changes to the >> test framework. These are available now on the testdev sandbox and will be >> in WPT as soon as they are merged: >> >> >> >> 1. errorMessage was not being included in the message output when >> an assertion failed. It is now. >> >> 2. assertionType was not being used. This optional notation on an >> assertion was put in the grammar as a way of helping us inform the test >> framework what to do, but never got fully developed. We are now using the >> information to augment the test documentation and output: >> >> "must" means an assertion is "*[MANDATORY]*" and a failure is an ERROR >> "should" means an assertion is "*[RECOMMENDED]*" and a failure is a >> WARNING >> "may" means an assertion is "*[OPTIONAL]*" and a failure is INFORMATIONAL >> >> In order to take advantage of these changes and make the tests more >> usable, I recommend changing the assertion titles so that they do not >> include the words "Check that" or "Check for". Make the titles just be >> clean active voice, present tense summary of what is required for the test >> to pass. For example: >> >> · 'body' is a String >> >> · '@context' contains http:// <http://UrlBlockedError.aspx>... >> >> · 'textDirection' is used once with a value in ['ltr', 'rtl'] >> >> Also, note that it is okay to embed HTML into .test file 'description' >> fields. The content of these fields is dropped into the template for each >> manual test using the following structure: >> >> >> >> <p>Fill the textarea below with JSON output from your annotation client >> >> implementation that supports the following criteria:</p> >> >> <div id="testDescription"></div> >> >> <p>Specifically, the following assertions will be evaluated:</p> >> >> <div id="assertion"></div> >> >> >> >> So, if we have a more friendly description field in each test, and >> consistently structured assertion titles, when we generate the instructions >> for people running manual tests it will look like: >> >> >> >> Fill the textarea below with JSON output from your annotation client >> implementation that supports the following criteria: >> >> >> >> An annotation that points to an external web resource for the body of >> the annotation. >> >> >> >> Specifically, the following assertions will be evaluated: >> >> >> >> 1. *[RECOMMENDED] *'body' is a String >> >> 2. *[MANDATORY] *'@context' contains http:// >> <http://UrlBlockedError.aspx>... >> >> 3. *[OPTIONAL]* 'textDirection' is used once with with a value in >> ['ltr', 'rtl'] >> >> We can obviously tune this but I think it is a good start toward making >> the manual tests more legible. >> >> >> >> -- >> >> Shane McCarron >> >> Projects Manager, Spec-Ops >> > > > > -- > Shane McCarron > Projects Manager, Spec-Ops > -- Shane McCarron Projects Manager, Spec-Ops
Received on Thursday, 18 August 2016 15:17:49 UTC