W3C home > Mailing lists > Public > public-wot-ig@w3.org > December 2018

[TestFest] minutes - 12 December 2018

From: Kazuyuki Ashimura <ashimura@w3.org>
Date: Thu, 13 Dec 2018 00:46:44 +0900
Message-ID: <CAJ8iq9U-DYOYmrG0-081ZK=p=JUfwHH1cVzFGJiFx9YvTpA4Vg@mail.gmail.com>
To: Public Web of Things IG <public-wot-ig@w3.org>, public-wot-wg@w3.org
available at:
  https://www.w3.org/2018/12/12-wot-pf-minutes.html

also as text below.

Thanks,

Kazuyuki

---

   [1]W3C

      [1] http://www.w3.org/

                               - DRAFT -

                              WoT TestFest

12 Dec 2018

Attendees

   Present
          Kaz_Ashimura, Michael_McCool, Michael_Lagally,
          Tomoaki_Mizushima, Toru_Kawaguchi, Taki_Kamiya,
          Michael_Koster, Sebastian_Kaebisch, Daniel_Peintner,
          Kunihiko_Toumura, Ege_Korkan

   Regrets

   Chair
          McCool

   Scribe
          kaz

Contents

     * [2]Topics
         1. [3]First hour
         2. [4]Seconf hour
     * [5]Summary of Action Items
     * [6]Summary of Resolutions
     __________________________________________________________

   <scribe> scribenick: kaz

First hour

   McCool: my OCF device is still down
   ... but want to talk about speech device over google hangout
   ... would work on OCF devices as well
   ... merged Oracle's update
   ... now 6 vendors

   [7]draft implementation report (local file on McCool's PC is
   newer than this online version)

      [7] https://cdn.staticaly.com/gh/mmccool/wot-thing-description/updated-test-results/testing/report.html

   McCool: consistent terminology
   ... generator or producer
   ... this (Node-WoT with Oracle Binding) is interesting

   <McCool> mlagally, are you on mute? we can't hear you

   <McCool>
   [8]https://github.com/mmccool/wot-thing-description/blob/update
   d-test-results/testing/report.html

      [8] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/report.html

   McCool: wanted characterize all the implementations
   ... consumer/producer
   ... this (Device Model Converter) is both consumer/producer

   Lagally: yes
   ... Oracle Digital Twin Simulator exposes TDs

   McCool: goes through Oracle's implementations
   ... btw, this "Oracle Asset Monitoring" doesn't produce or
   consume TDs, do it?
   ... a couple of things to restructure the interoperability
   table

   Lagally: it's not a simulator
   ... integration via node-wot
   ... connecting TDs to devices

   McCool: we should be clear about whether double counting or not

   Lagally: count based on code-base?

   McCool: we count systems with mostly independent code bases

   Lagally: to be clear, code base of the asset management is
   significantly bigger
   ... so would say it's a separate implementation

   McCool: also we should be clear which each implementation is,
   producer or consumer
   ... moving on

   Lagally: McCool, please create a PR with the proposed changes
   to the Oracle input

   McCool: unfortunately, Ege is not here
   ... but used my time for playground as well

   <McCool>
   [9]https://github.com/w3c/wot-thing-description/issues/321

      [9] https://github.com/w3c/wot-thing-description/issues/321

   McCool: issue 321
   ... problem with assertion stated
   ... TD arrays
   ... required and enum
   ... created an issue for TD
   ... also
   ... another problem
   ... issue 322

   <McCool>
   [10]https://github.com/w3c/wot-thing-description/issues/322

     [10] https://github.com/w3c/wot-thing-description/issues/322

   McCool: readOnly and writeOnly
   ... are optional
   ... but for JSON Schema, they're required
   ... if you look at the testfest-1 TD version
   ... writeOnly is mandatory
   ... this version has writable and writeOnly
   ... so there are mistakes
   ... we have a decision to make
   ... modify the schema to be up-to-date with the testfest-1
   tagged version?

   Lagally: would advocate fixing the schema

   McCool: would agree
   ... encourage people to update the schema
   ... would take an action to fix it

   <McCool>
   [11]https://github.com/mmccool/thingweb-playground/blob/dec-201
   8-testfest/WebContent/td-schema.json

     [11] https://github.com/mmccool/thingweb-playground/blob/dec-2018-testfest/WebContent/td-schema.json

   McCool: remove writable and make readOnly/writeOnly not
   required

   Lagally: one more thing
   ... on the assertions
   ... version in Ege's assertion is mandatory

   McCool: there are bunch of versions
   ... optional in the tagged version TD
   ... that's good
   ... the Assertions
   ... we have a problem
   ... 3 outcomes
   ... pass/fail/not-implemented
   ... but how to handle those 3 options?
   ... version is good example
   ... property object
   ... there is another properties tag here
   ... type of the members properties and items MUST be serialized
   as a JSON bject
   ... exist and has to have property
   ... the problem is
   ... in my test cases
   ... problem with camera
   ... bunch of properties
   ... simple integer type
   ... for brightness
   ... actually has sub property but not implemented

   <McCool>
   [12]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
   ine/TDs/Intel/SimpleWebCamera.jsonld

     [12] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/SimpleWebCamera.jsonld

   McCool: one of my properties
   ... updated schema

   <McCool>
   [13]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
   ine/TDs/Intel/BetterWebCamera.jsonld

     [13] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/BetterWebCamera.jsonld

   McCool: we now have security definition
   ... list of values
   ... and writable is wrong
   ... to be fixed
   ... did in contrast
   ... mixture of properties
   ... some of them have types
   ... JSON rule says property expressed as object is required
   ... only some of the properties are expressed
   ... properties sub field
   ... this TD should be a PASS
   ... but it turns to be fail or not-impl
   ... was scratching my head...
   ... work around is
   ... (dec-2018... branch)
   ... TD validator for JSON Schema
   ... special option called $comment
   ... callback function with this value
   ... particular rule match
   ... for schema
   ... I can write a rule
   ... if it matches and valid, it's pass
   ... if it matches but invalid, it's fail
   ... good news is can tell how to fix
   ... bad news is bunch of things to do

   <McCool> [14]https://github.com/epoberezkin/ajv

     [14] https://github.com/epoberezkin/ajv

   <McCool> [15]https://github.com/epoberezkin/ajv#options

     [15] https://github.com/epoberezkin/ajv#options

   Kaz: sorry but have started to wonder if it would make sense to
   try manual test first, and then this automatic testing
   mechanism as the second stage

   McCool: kind of agree
   ... but
   ... we should have some automatic tooling mechanism for the
   next testfest

   Kaz: yeah

   McCool: technically each implementer can provide implementation
   report
   ... manually creating the CSV input is one possible option
   ... do we have any gaps?

   Kaz: you've already found one possible solution, so we can try
   that at some point
   ... but maybe we should start manual check first

   McCool: yeah
   ... can try to solve the problem by Friday
   ... worth while trying both approaches
   ... manually filling the CSV files and tooling

   <McCool>
   [16]https://github.com/mmccool/wot-thing-description/blob/updat
   ed-test-results/testing/inputs/results/template.csv

     [16] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/results/template.csv

   McCool: basically I need a CSV file for each implementation
   ... that's kind of homework for people

   Lagally: wondering about a couple of TDs by others
   ... do we expect the other companies' TDs as well?

   McCool: Siemens is expected
   ... Hitachi does TD consumer only
   ... don't know about Dave's implementation
   ... my main concern is
   ... we may not have gaps
   ... do we have 2 implementations for security features?
   ... might show up down the vocabulary
   ... may see 0 here

   Lagally: what is the implication to have only one
   implementation?

   Kaz: if we can get only one implementation for some features,
   we need to drop the features
   ... and we have to republish a Candidate Rec after dropping
   them
   ... but we can identify some of the difficult features as
   "features at risk" when we transtion to CR
   ... and we can drop those "features at risk" and transition to
   PR without going back to CR

   Lagally: how are optional features treated? What happens if
   they are not implemented at all?

   Kaz: the trend is "even optional features should have 2 or more
   implementations"

   McCool: also wondering about default values

   Kaz: btw, the 2nd assertion "property, action, event MUST
   have..." should be split into 3 separate assertions, e.g.,
   "property MUST have...", "action MUST have..." and "event MUST
   have..."

   McCool: also thinking about that point
   ... using child assertions
   ... can create some extra assertions
   ... and consolidate all the child assertions

   Lagally: why don't we define the combined assertions separately
   in the spec and generate the tables

   McCool: also can do so

   Lagally: that might be easier

   Kaz: that said, the more important at the moment is checking
   the coverage of features
   ... so that we can tell which features are possibly at risk

   McCool: right
   ... at the moment, people should manually generate the CSV
   reports

   Lagally: the CSV format would work even after the TD spec is
   updated?

   McCool: as long as the feature ID doesn't change
   ... we need example and counter example as well
   ... these assertions should not change much
   ... btw, I went through the test spec at the appendix as well
   ... need to go through all the details and see counter examples
   ... parent vs sub assertions
   ... the parent assertion is true if all the child assertions
   are true
   ... need a bit more work
   ... a few assertions are still vague
   ... confusing assertions to be away

   Kaz: we should ask the others, Koster, Sebastian, Toumura-san,
   etc., to provide their TDs and reports

   McCool: recap would be better

   Koster: would like to recap

   Sebastian: yeah
   ... also would like to talk about the f2f in Princeton

   McCool: date of f2f and IG Charter as well
   ... Kaz and I had discussion

   <McCool>
   [17]https://w3c.github.io/wot/charters/wot-ig-2019.html

     [17] https://w3c.github.io/wot/charters/wot-ig-2019.html

   McCool: we should send out a Call for Consensus message to the
   whole IG for the updated draft IG Charter
   ... we should wrap up the discussion by Friday
   ... regarding the f2f dates
   [18]https://doodle.com/poll/5d4gbht3wm9v66dy
   ... result here
   ... 2 weeks
   ... 2 opposing conflicts
   ... Dave's conflict is he'll miss Thursday
   ... not sure about Taki
   ... or Sebastian

     [18] https://doodle.com/poll/5d4gbht3wm9v66dy

   Sebastian: definitely OK
   ... just in favor with later meeting
   ... prefer latter one

   McCool: yeah
   ... maybe Matthias also has difficulty with the earlier option
   ... more preparation would be better
   ... Taki, what is your trouble?

   Taki: north carolina for IIC

   McCool: Taki can't make the latter option at all
   ... Dave just has conflict with Thursday

   Lagally: what about Matthias?

   McCool: in the worst case, he can't make either of the options
   ... given Taki's situation, my reincarnation is the first
   option

   Sebastian: let's make the decision by the end of this week

   <McCool> 28 Jan - 2 Feb 2019

   McCool: ok
   ... we'll finalize the date by the end of this week
   ... should send a message for CfC out
   ... should nail down by Friday

Second hour

   McCool: recap

   Koster: most of the test cases are validating TD

   <McCool>
   [19]https://github.com/w3c/wot/blob/master/testing/criteria.md

     [19] https://github.com/w3c/wot/blob/master/testing/criteria.md

   McCool: either TD producer or a TD consumer
   ... you have a device on the network and have a TD which
   describes the device
   ... automatically generated TD or manually generated TD
   ... proxies take a TD and possibly change it
   ... somewhere here (in the report.html)
   ... we have to count distinct implementations
   ... and what is "distinct implementations"?

   <McCool>
   [20]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results/testing

     [20] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing

   McCool: e.g., even though there are three implementation by
   Intel for security features, they're based on the same
   code-base (node-wot), so should be counted as one
   implementation
   ... we have 6 organizations at the implementations/
   subdirectory
   ... would have the other implementers as well
   ... (and then shows optional interop/ area)
   ... arcs within the connection diagram should be described here
   ... producer/consumer pairs

   Kaz: yeah, so at some point, we should split that interop part
   ... and create a separate WG Note

   McCool: yeah, maybe an interoperability Note
   ... (and then shows the implementation result CSV files)

   [21]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results/testing/inputs/results

     [21] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results

   Kaz summrizes what we should do:
   1. submit TDs at
   [22]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine/TDs
   2. submit CSV-based reports:
   [23]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results/testing/inputs/results
   3. implementation descriptin:
   [24]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results/testing/inputs/implementations

     [22] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs
     [23] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
     [24] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations

   McCool: that is kind of described within the manual at:
   [25]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine
   ... but the automatic tool is not available yet

     [25] https://github.com/w3c/wot/tree/master/testfest/2018-12-online

   Koster: ok
   ... the CSV file is the result of assertions

   McCool: (updates the testfest manual)

   [26]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine

     [26] https://github.com/w3c/wot/tree/master/testfest/2018-12-online

   McCool: not-impl means no example
   ... null will be ignored

   Koster: question about Ege's tool
   ... using npm
   ... AssertionTester

   McCool: there are 2 tools
   ... playground validator
   ... td-schema.json is almost updated for the testfest-1 TD
   version

   <McCool>
   [27]https://github.com/mmccool/thingweb-playground/tree/dec-201
   8-testfest/WebContent

     [27] https://github.com/mmccool/thingweb-playground/tree/dec-2018-testfest/WebContent

   McCool: we need to get it updated
   ... have a PR
   ... Ege also has a PR
   ... waiting for those PRs to be merged
   ... playground has web interface and command interface
   ... and then AssertionTester
   ... ajv schema validator
   ... some of my assertions here
   ... (McCool revisits his assertions)
   ... (and explains the issue with handling pass/fail/not-impl)

   Koster: TD schema with ajv hooks

   Sebastian: need to leave
   ... wondering about the goal of the testfest this week

   Kaz: thinks the goal of this first testfest effort is simply
   looking the implementability of each feature, i.e., can get one
   or more implementation for each feature (rather than 2 or more)

   McCool: would add some more extra assertions
   ... if you would like to split some of them, we can do so
   ... this is a snapshop as of today
   ... if node-wot is to be updated, that's fine

   Sebastian: another question about TD slot on Friday
   ... will we have a regular TD meeting?

   McCool: other than a few issues, you can use the rest for the
   TD discussion

   Sebastian: we have to make decision about some of the TD issues

   Kaz: using the first 30m for testfest wrap-up
   ... and using the remaining 1.5h for TD discussion
   ... is that ok?

   Sebastian: ok

   Toru: going back to the interop test
   ... pass/fail/secure, etc.
   ... is it possible to handle partial result?

   McCool: (visits interop/ area)
   ... right now we have producer, consumer, security, status
   ... you can mention two rows and merge them
   ... hundred passes and one failure would be failure
   ... purpose of the summary table
   ... maybe more information to be captured
   ... you can add more columns
   ... my tool will ignore the information
   ... e.g., you can add columns like cause, comment, etc.
   ... some failure may be caused by "authentication invalid" with
   "expired certificate"
   ... we can keep track those columns

   Toru: maybe just one additional comment column would work

   McCool: ok
   ... probably will become an interoperability document

   Lagally: one suggestion
   ... overview document
   ... private repos
   ... very confusing
   ... and Ege not on the call

   (Ege just joined)

   McCool: Ege, have you checked my PR?
   ... Ege's PR is the main one
   ... and I also have a PR

   Ege: now I'm free to work on that

   McCool: discovered bunch of issues
   ... fundamental issue

   Kaz: would suggest we resolve that issue via the PR/Issue since
   we've already discussed during this call twice :)

   Lagally: which of the repos to have the latest tools, schemas,
   etc.?

   McCool: here is the suggestion
   ... nothing is in a good shape at the moment...
   ... we need some more time to stabilize the situation
   ... (summarize the issue with schema)

   Ege: updated the schema Monday evening

   McCool: Ege will update the JSON Schema
   ... will also add much bashing
   ... within the next two days

   Lagally: thingweb resources also on ege's private repo

   Ege: will merge it but still need to check

   [28]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine

     [28] https://github.com/w3c/wot/tree/master/testfest/2018-12-online

   Kaz: so we'd like to use only the wot official repo for our
   testfest (if possible)
   ... note that TD files and CSV reports are already to be copied
   under w3c/wot repo
   ... and so I was wondering about the implementation description
   HTML

   McCool: that can be also copied under the official w3c repo

   <McCool>
   [29]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
   ine/implementations/README.md

     [29] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/implementations/README.md

   McCool: people can install their implementation description
   HTML as well to the above official area
   ... and please submit your generated CSV report under:
   [30]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine/results
   ... each organization should create a subdirectory there
   ... and copy your generated CSV files to that subdirectory

     [30] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results

   Kaz: is it OK to start with this manual approach on the w3c/wot
   repo?

   Lagally: ok
   ... so we're encouraged to copy the implementation description
   HTML as well to the w3c/wot repo

   McCool: yes
   ... can copy a template for your help
   ... and I'll copy all the resources to my mmccool repo to
   regenerate the report.html

   Lagally: ok

   Taki: particular assertion, e.g., property, action and event
   ... not sure we really should split it

   McCool: yeah
   ... can have one combined assertion as a parent with several
   child assertions
   ... can just generate them based on the current TD

   Taki: can also have some discussion on Friday

   McCool: yeah

   Ege: working on the schema fix

   McCool: taki, maybe you can merge Ege's PR

   <McCool>
   [31]https://github.com/w3c/wot-thing-description/pull/323

     [31] https://github.com/w3c/wot-thing-description/pull/323

   Taki: issue on the JSON Schema

   McCool: yeah
   ... writable
   ... ah, even older which just include writable
   ... another PR

   [32]https://github.com/w3c/wot-thing-description/pull/324

     [32] https://github.com/w3c/wot-thing-description/pull/324

   Taki: can merge it

   McCool: anything else for today?

   (none)

   McCool: please work on your CSV files
   ... will continue to work on the automation
   ... Ege, we need further collaboration

   [the official testfest call adjourned]

Summary of Action Items

Summary of Resolutions

   [End of minutes]
     __________________________________________________________


    Minutes manually created (not a transcript), formatted by
    David Booth's [33]scribe.perl version 1.154 ([34]CVS log)
    $Date: 2018/12/12 15:45:50 $

     [33] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
     [34] http://dev.w3.org/cvsweb/2002/scribe/
Received on Wednesday, 12 December 2018 15:47:49 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 12 December 2018 15:47:50 UTC