Re: Model testing - body and targets and suggestions for testing UI

Okay - I understand what you mean...  I am working on a PR to integrate MD
processing into the runner page.  so it should effect that too.  Won't be
immediate though.

On Thu, Sep 1, 2016 at 12:47 PM, Timothy Cole <t-cole3@illinois.edu> wrote:

>
>
> *From:* Shane McCarron [mailto:shane@spec-ops.io]
> *Sent:* Thursday, September 01, 2016 11:40 AM
> *To:* Cole, Timothy W <t-cole3@illinois.edu>
> *Cc:* W3C Public Annotation List <public-annotation@w3.org>
> *Subject:* Re: Model testing - body and targets and suggestions for
> testing UI
>
>
>
> A is trivial and I will do it right away.  note that there is always a
> delay though while the WPT keepers evaluate things and integrate them.
>
>
>
> Thanks.
>
>
>
> B...  In which summary do you mean?
>
>
>
> The summary that appears at the bottom of the pages in which an
> implementer pastes their annotations right after they click Check JSON,
> e.g.:
>
> http://testdev.spec-ops.io:8000/annotation-model/annotations/
> annotationOptionals-manual.html
>
> If using the runner.js, these pages disappear as soon as each test
> completes, but you can also go to these pages directly (linked from the
> results displayed by runner) to see what's happening. You also see all of
> this in the bottom of the runner page when you check dump JSON.
>
>
>
>
>
> C... I don't hate the idea but I dont think there is a good way to copy
> that data over from test to test (that would be really handy so you don't
> need to click the checkboxes on each form... I will look at it.
>
> Thanks.
>
>
>
> D... We don't have control of that interface, unfortunately.
>
> Okay.
>
>
>
>
>
>
>
> On Thu, Sep 1, 2016 at 11:30 AM, Cole, Timothy W <t-cole3@illinois.edu>
> wrote:
>
> Okay, though if A and/or B at least were easy, it would make a better
> first impression as we recruit implementers...
>
>
>
> Regarding D, when you do get a chance to look at it, one option might be
> to add checkboxes to simply suppress reporting of failed assertions that
> are should or may (as opposed to must), and to suppress test harness and
> AJV error messages. The report would then tell us that the annotation
> passed all MUST assertions and implemented features associated with the
> should and may assertions. Checkboxes could be unchecked for those who want
> to see...
>
>
>
> -Tim Cole
> ------------------------------
>
> *From:* Shane McCarron [shane@spec-ops.io]
> *Sent:* Thursday, September 01, 2016 11:16
> *To:* Cole, Timothy W
> *Cc:* W3C Public Annotation List
> *Subject:* Re: Model testing - body and targets and suggestions for
> testing UI
>
> I will look into making these improvements.  However, they don't impact
> the initial integration.  We should pull in the updated tests ASAP.
>
>
>
> On Thu, Sep 1, 2016 at 9:56 AM, Timothy Cole <t-cole3@illinois.edu> wrote:
>
> Shane-
>
>
>
> New data model tests (i.e., checking body and target constraints) are have
> been added to the annotation-level tests already available on the testdev
> server. These are ready to be migrated up to production, but as mentioned
> last night we need to get rid of old, deprecated tests and json files. I
> also have some feedback on the test script UI that would be nice to address
> if not too difficult (see below). Where we are with the data model tests:
>
>
>
> *Mandatory tests* – all annotations should pass all assertions checked by
> these 3 tests:
>
> 1. Annotation-level: http://testdev.spec-ops.io:8000/annotation-model/
> annotations/annotationMusts-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=cDNzcnRXV70fcCdoV2TYPgN2x79rtDxaqn3t_miXK9A&e=>
>  (14 assertions)
>
> 2. Body-level:  http://testdev.spec-ops.io:8000/annotation-model/
> bodiesTargets/bodyMusts-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=mxafD2YyfEUK5mir6sFVJ-60VKn8J52vLFU6swrpxN4&e=>
> (16 assertions)
>
> 3. Target-level: http://testdev.spec-ops.io:8000/annotation-model/
> bodiesTargets/targetMusts-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_targetMusts-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=wrst7qqnG5_0Ux88Hkgh1O2SgZVjzu12H70C8Zz7jxI&e=>
>  (15 assertions)
>
>
>
> *Recommended / Optional tests* – most individual annotations will "fail"
> a majority of these assertions:
>
> 1. Annotation-level: http://testdev.spec-ops.io:8000/annotation-model/
> annotations/annotationOptionals-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=uDCca66_BslPSrDB8BXuRTExJpdcc_dcMzZ09ZQ9rdQ&e=>
>  (15 assertions)
>
> 2. Annotation-level Agents: http://testdev.spec-ops.io:
> 8000/annotation-model/annotations/annotationAgentOptionals-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_annotationAgentOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=YZwq1AwUyMzU5yFMGnVARgidjZa6gpvLLohmTX-1aRY&e=>
>  (16 assertions)
>
> 3. Body-level: http://testdev.spec-ops.io:8000/annotation-model/
> bodiesTargets/bodyOptionals-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=Eb3jDKbmiBHtZnb9D5RCFIEMywOSIMuDv4VwdA-EwEA&e=>
>   (28 Assertions)
>
> 4. Target-level: http://testdev.spec-ops.io:8000/annotation-model/
> bodiesTargets/targetOptionals-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_targetOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=VZ3Ms8SRjhtF23jRVNeqHrncyNT4kaNfrxaaZ6eB9Ps&e=>
>  (25 assertions)
>
> 5. Body and Target-level Agents: http://testdev.spec-ops.io:
> 8000/annotation-model/bodiesTargets/bodyTargAgentOptionals-manual.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_bodiesTargets_bodyTargAgentOptionals-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=uWLhL21I8ttNHIjO9n-8d6jmuSDCbDvDTnmQomQehys&s=z2L5TUdEnZrccQwYRanqUlpnVeSsJre5YLsvYOF8mbA&e=>
>  (16 assertions)
>
>
>
> *Suggestions*:
>
>
>
> A. The text box for inputting JSON-LD annotation and the Check JSON button
> need to appear on the form before the list of assertions being checked.
> The list of assertions is of variable length and can be quite long. Better
> when you have to paste in your annotation 8 times that the input box be
> higher on the form and always in the same spot.
>
>
>
> B. Per discussion, the title of each assertion contains mark-down. This
> shows up formatted on the input form, but in the summary of results, the
> mark-down is not being processed. Any way to fix?
>
>
>
> C. When a test 'fails' the error message from the assertion is displayed
> first (good), but then the AJV error message and test script trace is
> concatenated, not useful for the person submitting the annotations for
> testing. Is there any way to suppress (e.g., in hidden HTML) the AJV and
> test harness trace error messages, or to format to be less prominent?
>
>
>
> D. Most annotations will fail most recommended / optional assertions.
> These are intended to identify what features have been implemented in any
> given annotation and do not really go to validation per se, and relatively
> few individual annotations implemented more than a handful of optional
> features. Example 44 from our model only pass 3 out of 25 target 'tests'.
> Is there any option on should and may assertions to tone down the red small
> caps FAIL error message? Something in yellow and maybe a different word
> than FAIL?  Alternatively, we may need to discuss granularity further –
> separate post forthcoming.
>
>
>
> Any changes to help ameliorate any of these issues that are easy to fix
> before moving latest into production would be appreciated.
>
>
>
> Thanks,
>
>
>
> Tim Cole
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Shane McCarron
>
> Projects Manager, Spec-Ops
>
>
>
>
>
> --
>
> Shane McCarron
>
> Projects Manager, Spec-Ops
>



-- 
Shane McCarron
Projects Manager, Spec-Ops