RE: Model testing - body and targets and suggestions for testing UI

 
From: Shane McCarron [mailto:shane@spec-ops.io] 
Sent: Thursday, September 01, 2016 11:40 AM
To: Cole, Timothy W <t-cole3@illinois.edu>
Cc: W3C Public Annotation List <public-annotation@w3.org>
Subject: Re: Model testing - body and targets and suggestions for testing UI
 
A is trivial and I will do it right away.  note that there is always a delay though while the WPT keepers evaluate things and integrate them.
 
Thanks.
 
B...  In which summary do you mean? 
 
The summary that appears at the bottom of the pages in which an implementer pastes their annotations right after they click Check JSON, e.g.: 
http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationOptionals-manual.html 
If using the runner.js, these pages disappear as soon as each test completes, but you can also go to these pages directly (linked from the results displayed by runner) to see what's happening. You also see all of this in the bottom of the runner page when you check dump JSON.
 
 
C... I don't hate the idea but I dont think there is a good way to copy that data over from test to test (that would be really handy so you don't need to click the checkboxes on each form... I will look at it.
Thanks.
 
D... We don't have control of that interface, unfortunately.
Okay.
 
 
 
On Thu, Sep 1, 2016 at 11:30 AM, Cole, Timothy W <t-cole3@illinois.edu <mailto:t-cole3@illinois.edu> > wrote:
Okay, though if A and/or B at least were easy, it would make a better first impression as we recruit implementers... 
 
Regarding D, when you do get a chance to look at it, one option might be to add checkboxes to simply suppress reporting of failed assertions that are should or may (as opposed to must), and to suppress test harness and AJV error messages. The report would then tell us that the annotation passed all MUST assertions and implemented features associated with the should and may assertions. Checkboxes could be unchecked for those who want to see...
 
-Tim Cole

  _____  

From: Shane McCarron [shane@spec-ops.io <mailto:shane@spec-ops.io> ]
Sent: Thursday, September 01, 2016 11:16
To: Cole, Timothy W
Cc: W3C Public Annotation List
Subject: Re: Model testing - body and targets and suggestions for testing UI
I will look into making these improvements.  However, they don't impact the initial integration.  We should pull in the updated tests ASAP.
 
On Thu, Sep 1, 2016 at 9:56 AM, Timothy Cole <t-cole3@illinois.edu <mailto:t-cole3@illinois.edu> > wrote:
Shane-
 
New data model tests (i.e., checking body and target constraints) are have been added to the annotation-level tests already available on the testdev server. These are ready to be migrated up to production, but as mentioned last night we need to get rid of old, deprecated tests and json files. I also have some feedback on the test script UI that would be nice to address if not too difficult (see below). Where we are with the data model tests:
 
Mandatory tests – all annotations should pass all assertions checked by these 3 tests:
1. Annotation-level: http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationMusts-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=cDNzcnRXV70fcCdoV2TYPgN2x79rtDxaqn3t_miXK9A&e=>   (14 assertions) 
2. Body-level:  http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/bodyMusts-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=mxafD2YyfEUK5mir6sFVJ-60VKn8J52vLFU6swrpxN4&e=>   (16 assertions)
3. Target-level: http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/targetMusts-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_targetMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=wrst7qqnG5_0Ux88Hkgh1O2SgZVjzu12H70C8Zz7jxI&e=>   (15 assertions) 
 
Recommended / Optional tests – most individual annotations will "fail" a majority of these assertions:
1. Annotation-level: http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationOptionals-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=uDCca66_BslPSrDB8BXuRTExJpdcc_dcMzZ09ZQ9rdQ&e=>   (15 assertions)
2. Annotation-level Agents: http://testdev.spec-ops.io:8000/annotation-model/annotations/annotationAgentOptionals-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationAgentOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=YZwq1AwUyMzU5yFMGnVARgidjZa6gpvLLohmTX-1aRY&e=>   (16 assertions)
3. Body-level: http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/bodyOptionals-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=Eb3jDKbmiBHtZnb9D5RCFIEMywOSIMuDv4VwdA-EwEA&e=>    (28 Assertions)
4. Target-level: http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/targetOptionals-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_targetOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=VZ3Ms8SRjhtF23jRVNeqHrncyNT4kaNfrxaaZ6eB9Ps&e=>   (25 assertions)
5. Body and Target-level Agents: http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/bodyTargAgentOptionals-manual.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyTargAgentOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=z2L5TUdEnZrccQwYRanqUlpnVeSsJre5YLsvYOF8mbA&e=>   (16 assertions)
 
Suggestions:
 
A. The text box for inputting JSON-LD annotation and the Check JSON button need to appear on the form before the list of assertions being checked.  The list of assertions is of variable length and can be quite long. Better when you have to paste in your annotation 8 times that the input box be higher on the form and always in the same spot.
 
B. Per discussion, the title of each assertion contains mark-down. This shows up formatted on the input form, but in the summary of results, the mark-down is not being processed. Any way to fix?
 
C. When a test 'fails' the error message from the assertion is displayed first (good), but then the AJV error message and test script trace is concatenated, not useful for the person submitting the annotations for testing. Is there any way to suppress (e.g., in hidden HTML) the AJV and test harness trace error messages, or to format to be less prominent?   
 
D. Most annotations will fail most recommended / optional assertions. These are intended to identify what features have been implemented in any given annotation and do not really go to validation per se, and relatively few individual annotations implemented more than a handful of optional features. Example 44 from our model only pass 3 out of 25 target 'tests'.  Is there any option on should and may assertions to tone down the red small caps FAIL error message? Something in yellow and maybe a different word than FAIL?  Alternatively, we may need to discuss granularity further – separate post forthcoming.
 
Any changes to help ameliorate any of these issues that are easy to fix before moving latest into production would be appreciated.
 
Thanks,
 
Tim Cole
 
 
 
 
 



 
-- 
Shane McCarron 
Projects Manager, Spec-Ops



 
-- 
Shane McCarron
Projects Manager, Spec-Ops