Re: FW: [w3c/web-platform-tests] Annotation model first real tests (#3509)

Yes.  Or rather, probably.

You can see an example of what the generated output of a report will look
like at http://testdev.spec-ops.io/test-results/annotation-model/all.html

Basically the process is pretty manual right now.  Drop your json into a
clone of the test-results repo, run wpt-report and it will generate the
report.  git add your result file, git commit the directory, git push and
submit the PR.

Now.... arguably there should be an automated way to do this.  I am working
that with W3C now.  For our purposes, maybe I can set up some place where
people can drop their results and magic will occur.

On Wed, Aug 24, 2016 at 11:03 AM, Timothy Cole <t-cole3@illinois.edu> wrote:

> Shane-
>
>
>
> Now that your PR for the first set of data model tests has been merged
> into WPT (hurray!), can you help me understand (well enough for when we
> email them) the process implementers should follow to save test results?  Here's
> my understanding of the test running process for implementers so far:
>
>
>
> 1. Go to http://w3c-test.org/tools/runner/index.html and in the 'Run
> tests under path' textbox enter:
>
>
>
>     /annotation-model/annotations
>
>
>
> Then click Start. [question – does it matter if they check either of the 2
> checkboxes on the form?]
>
>
>
> 2. Three tests will be run, each testing a set of assertions. After you
> click Start, a separate window will appear and the test forms will be
> displayed sequentially. You will need to paste the JSON-LD of your
> annotation to test into the text box at the bottom of each test form and
> then click the check JSON button. Only one annotation can be submitted at a
> time. Once the final of the 3 tests completes the test form window will
> close and you will see a summary of results. On completion of the 3 tests,
> please click the Download JSON results and save to your disk.
>
>
>
> NOTE: your annotation should pass all of the assertions of the 2nd test, annotationMusts-manual.html
> test. These assertions are derived from the MUST statements in the Web
> Annotation model.  Most annotations will pass all of the assertions
> checked by the other 2 tests – and that is okay, since these assertions are
> derived from SHOULD and MAY statements.   We use these tests primarily to
> keep track of what optional features of the model are being implemented.
>
>
>
> 3. Then ???
>
>
>
> There's a bit of placeholder information here (
> https://github.com/w3c/test-results/tree/gh-pages/annotation-model) about
> creating a PR and then using a node.js package (
> https://www.npmjs.com/package/wptreport) to generate the report, but I'm
> not sure how much of this is for the implementer to do and how much we are
> supposed to do.
>
>
>
> Can you fill in the blanks?
>
>
>
> Thanks,
>
>
>
> Tim Cole
>
>
>
> *From:* Michael[tm] Smith [mailto:notifications@github.com]
> *Sent:* Tuesday, August 23, 2016 6:05 PM
> *To:* w3c/web-platform-tests <web-platform-tests@noreply.github.com>
> *Cc:* Tim Cole <t-cole3@illinois.edu>; Mention <mention@noreply.github.com
> >
> *Subject:* Re: [w3c/web-platform-tests] Annotation model first real tests
> (#3509)
>
>
>
> Merged #3509
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_web-2Dplatform-2Dtests_pull_3509&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=bGRyUAI0zlxfvCKYYmQPhd6fLQ4ZAO4gDvCR_dB6mSE&s=YNf2cDsCtb4UhLz4_kdZMI2xFXdScPklPHQl0l9O8YA&e=>
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_web-2Dplatform-2Dtests_pull_3509-23event-2D765250296&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=bGRyUAI0zlxfvCKYYmQPhd6fLQ4ZAO4gDvCR_dB6mSE&s=XD2Is2NKbkGRZlKZKEV1CX63UUD1uZgZ3b6w8qgNVk8&e=>,
> or mute the thread
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHRRQnX8llnbtPimrcJqCwdyrtBEVq8oks5qi3yXgaJpZM4Jo4uF&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=bGRyUAI0zlxfvCKYYmQPhd6fLQ4ZAO4gDvCR_dB6mSE&s=F25YvAm3bZoE3Qnvmr3WMwENNwqtEgNr_oxIAAyfIEE&e=>
> .[image: Image removed by sender.]
>



-- 
Shane McCarron
Projects Manager, Spec-Ops