- From: Kazuyuki Ashimura <ashimura@w3.org>
- Date: Thu, 13 Dec 2018 00:46:44 +0900
- To: Public Web of Things IG <public-wot-ig@w3.org>, public-wot-wg@w3.org
available at:
https://www.w3.org/2018/12/12-wot-pf-minutes.html
also as text below.
Thanks,
Kazuyuki
---
[1]W3C
[1] http://www.w3.org/
- DRAFT -
WoT TestFest
12 Dec 2018
Attendees
Present
Kaz_Ashimura, Michael_McCool, Michael_Lagally,
Tomoaki_Mizushima, Toru_Kawaguchi, Taki_Kamiya,
Michael_Koster, Sebastian_Kaebisch, Daniel_Peintner,
Kunihiko_Toumura, Ege_Korkan
Regrets
Chair
McCool
Scribe
kaz
Contents
* [2]Topics
1. [3]First hour
2. [4]Seconf hour
* [5]Summary of Action Items
* [6]Summary of Resolutions
__________________________________________________________
<scribe> scribenick: kaz
First hour
McCool: my OCF device is still down
... but want to talk about speech device over google hangout
... would work on OCF devices as well
... merged Oracle's update
... now 6 vendors
[7]draft implementation report (local file on McCool's PC is
newer than this online version)
[7] https://cdn.staticaly.com/gh/mmccool/wot-thing-description/updated-test-results/testing/report.html
McCool: consistent terminology
... generator or producer
... this (Node-WoT with Oracle Binding) is interesting
<McCool> mlagally, are you on mute? we can't hear you
<McCool>
[8]https://github.com/mmccool/wot-thing-description/blob/update
d-test-results/testing/report.html
[8] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/report.html
McCool: wanted characterize all the implementations
... consumer/producer
... this (Device Model Converter) is both consumer/producer
Lagally: yes
... Oracle Digital Twin Simulator exposes TDs
McCool: goes through Oracle's implementations
... btw, this "Oracle Asset Monitoring" doesn't produce or
consume TDs, do it?
... a couple of things to restructure the interoperability
table
Lagally: it's not a simulator
... integration via node-wot
... connecting TDs to devices
McCool: we should be clear about whether double counting or not
Lagally: count based on code-base?
McCool: we count systems with mostly independent code bases
Lagally: to be clear, code base of the asset management is
significantly bigger
... so would say it's a separate implementation
McCool: also we should be clear which each implementation is,
producer or consumer
... moving on
Lagally: McCool, please create a PR with the proposed changes
to the Oracle input
McCool: unfortunately, Ege is not here
... but used my time for playground as well
<McCool>
[9]https://github.com/w3c/wot-thing-description/issues/321
[9] https://github.com/w3c/wot-thing-description/issues/321
McCool: issue 321
... problem with assertion stated
... TD arrays
... required and enum
... created an issue for TD
... also
... another problem
... issue 322
<McCool>
[10]https://github.com/w3c/wot-thing-description/issues/322
[10] https://github.com/w3c/wot-thing-description/issues/322
McCool: readOnly and writeOnly
... are optional
... but for JSON Schema, they're required
... if you look at the testfest-1 TD version
... writeOnly is mandatory
... this version has writable and writeOnly
... so there are mistakes
... we have a decision to make
... modify the schema to be up-to-date with the testfest-1
tagged version?
Lagally: would advocate fixing the schema
McCool: would agree
... encourage people to update the schema
... would take an action to fix it
<McCool>
[11]https://github.com/mmccool/thingweb-playground/blob/dec-201
8-testfest/WebContent/td-schema.json
[11] https://github.com/mmccool/thingweb-playground/blob/dec-2018-testfest/WebContent/td-schema.json
McCool: remove writable and make readOnly/writeOnly not
required
Lagally: one more thing
... on the assertions
... version in Ege's assertion is mandatory
McCool: there are bunch of versions
... optional in the tagged version TD
... that's good
... the Assertions
... we have a problem
... 3 outcomes
... pass/fail/not-implemented
... but how to handle those 3 options?
... version is good example
... property object
... there is another properties tag here
... type of the members properties and items MUST be serialized
as a JSON bject
... exist and has to have property
... the problem is
... in my test cases
... problem with camera
... bunch of properties
... simple integer type
... for brightness
... actually has sub property but not implemented
<McCool>
[12]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
ine/TDs/Intel/SimpleWebCamera.jsonld
[12] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/SimpleWebCamera.jsonld
McCool: one of my properties
... updated schema
<McCool>
[13]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
ine/TDs/Intel/BetterWebCamera.jsonld
[13] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/BetterWebCamera.jsonld
McCool: we now have security definition
... list of values
... and writable is wrong
... to be fixed
... did in contrast
... mixture of properties
... some of them have types
... JSON rule says property expressed as object is required
... only some of the properties are expressed
... properties sub field
... this TD should be a PASS
... but it turns to be fail or not-impl
... was scratching my head...
... work around is
... (dec-2018... branch)
... TD validator for JSON Schema
... special option called $comment
... callback function with this value
... particular rule match
... for schema
... I can write a rule
... if it matches and valid, it's pass
... if it matches but invalid, it's fail
... good news is can tell how to fix
... bad news is bunch of things to do
<McCool> [14]https://github.com/epoberezkin/ajv
[14] https://github.com/epoberezkin/ajv
<McCool> [15]https://github.com/epoberezkin/ajv#options
[15] https://github.com/epoberezkin/ajv#options
Kaz: sorry but have started to wonder if it would make sense to
try manual test first, and then this automatic testing
mechanism as the second stage
McCool: kind of agree
... but
... we should have some automatic tooling mechanism for the
next testfest
Kaz: yeah
McCool: technically each implementer can provide implementation
report
... manually creating the CSV input is one possible option
... do we have any gaps?
Kaz: you've already found one possible solution, so we can try
that at some point
... but maybe we should start manual check first
McCool: yeah
... can try to solve the problem by Friday
... worth while trying both approaches
... manually filling the CSV files and tooling
<McCool>
[16]https://github.com/mmccool/wot-thing-description/blob/updat
ed-test-results/testing/inputs/results/template.csv
[16] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/results/template.csv
McCool: basically I need a CSV file for each implementation
... that's kind of homework for people
Lagally: wondering about a couple of TDs by others
... do we expect the other companies' TDs as well?
McCool: Siemens is expected
... Hitachi does TD consumer only
... don't know about Dave's implementation
... my main concern is
... we may not have gaps
... do we have 2 implementations for security features?
... might show up down the vocabulary
... may see 0 here
Lagally: what is the implication to have only one
implementation?
Kaz: if we can get only one implementation for some features,
we need to drop the features
... and we have to republish a Candidate Rec after dropping
them
... but we can identify some of the difficult features as
"features at risk" when we transtion to CR
... and we can drop those "features at risk" and transition to
PR without going back to CR
Lagally: how are optional features treated? What happens if
they are not implemented at all?
Kaz: the trend is "even optional features should have 2 or more
implementations"
McCool: also wondering about default values
Kaz: btw, the 2nd assertion "property, action, event MUST
have..." should be split into 3 separate assertions, e.g.,
"property MUST have...", "action MUST have..." and "event MUST
have..."
McCool: also thinking about that point
... using child assertions
... can create some extra assertions
... and consolidate all the child assertions
Lagally: why don't we define the combined assertions separately
in the spec and generate the tables
McCool: also can do so
Lagally: that might be easier
Kaz: that said, the more important at the moment is checking
the coverage of features
... so that we can tell which features are possibly at risk
McCool: right
... at the moment, people should manually generate the CSV
reports
Lagally: the CSV format would work even after the TD spec is
updated?
McCool: as long as the feature ID doesn't change
... we need example and counter example as well
... these assertions should not change much
... btw, I went through the test spec at the appendix as well
... need to go through all the details and see counter examples
... parent vs sub assertions
... the parent assertion is true if all the child assertions
are true
... need a bit more work
... a few assertions are still vague
... confusing assertions to be away
Kaz: we should ask the others, Koster, Sebastian, Toumura-san,
etc., to provide their TDs and reports
McCool: recap would be better
Koster: would like to recap
Sebastian: yeah
... also would like to talk about the f2f in Princeton
McCool: date of f2f and IG Charter as well
... Kaz and I had discussion
<McCool>
[17]https://w3c.github.io/wot/charters/wot-ig-2019.html
[17] https://w3c.github.io/wot/charters/wot-ig-2019.html
McCool: we should send out a Call for Consensus message to the
whole IG for the updated draft IG Charter
... we should wrap up the discussion by Friday
... regarding the f2f dates
[18]https://doodle.com/poll/5d4gbht3wm9v66dy
... result here
... 2 weeks
... 2 opposing conflicts
... Dave's conflict is he'll miss Thursday
... not sure about Taki
... or Sebastian
[18] https://doodle.com/poll/5d4gbht3wm9v66dy
Sebastian: definitely OK
... just in favor with later meeting
... prefer latter one
McCool: yeah
... maybe Matthias also has difficulty with the earlier option
... more preparation would be better
... Taki, what is your trouble?
Taki: north carolina for IIC
McCool: Taki can't make the latter option at all
... Dave just has conflict with Thursday
Lagally: what about Matthias?
McCool: in the worst case, he can't make either of the options
... given Taki's situation, my reincarnation is the first
option
Sebastian: let's make the decision by the end of this week
<McCool> 28 Jan - 2 Feb 2019
McCool: ok
... we'll finalize the date by the end of this week
... should send a message for CfC out
... should nail down by Friday
Second hour
McCool: recap
Koster: most of the test cases are validating TD
<McCool>
[19]https://github.com/w3c/wot/blob/master/testing/criteria.md
[19] https://github.com/w3c/wot/blob/master/testing/criteria.md
McCool: either TD producer or a TD consumer
... you have a device on the network and have a TD which
describes the device
... automatically generated TD or manually generated TD
... proxies take a TD and possibly change it
... somewhere here (in the report.html)
... we have to count distinct implementations
... and what is "distinct implementations"?
<McCool>
[20]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results/testing
[20] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing
McCool: e.g., even though there are three implementation by
Intel for security features, they're based on the same
code-base (node-wot), so should be counted as one
implementation
... we have 6 organizations at the implementations/
subdirectory
... would have the other implementers as well
... (and then shows optional interop/ area)
... arcs within the connection diagram should be described here
... producer/consumer pairs
Kaz: yeah, so at some point, we should split that interop part
... and create a separate WG Note
McCool: yeah, maybe an interoperability Note
... (and then shows the implementation result CSV files)
[21]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results/testing/inputs/results
[21] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
Kaz summrizes what we should do:
1. submit TDs at
[22]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine/TDs
2. submit CSV-based reports:
[23]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results/testing/inputs/results
3. implementation descriptin:
[24]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results/testing/inputs/implementations
[22] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs
[23] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
[24] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations
McCool: that is kind of described within the manual at:
[25]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine
... but the automatic tool is not available yet
[25] https://github.com/w3c/wot/tree/master/testfest/2018-12-online
Koster: ok
... the CSV file is the result of assertions
McCool: (updates the testfest manual)
[26]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine
[26] https://github.com/w3c/wot/tree/master/testfest/2018-12-online
McCool: not-impl means no example
... null will be ignored
Koster: question about Ege's tool
... using npm
... AssertionTester
McCool: there are 2 tools
... playground validator
... td-schema.json is almost updated for the testfest-1 TD
version
<McCool>
[27]https://github.com/mmccool/thingweb-playground/tree/dec-201
8-testfest/WebContent
[27] https://github.com/mmccool/thingweb-playground/tree/dec-2018-testfest/WebContent
McCool: we need to get it updated
... have a PR
... Ege also has a PR
... waiting for those PRs to be merged
... playground has web interface and command interface
... and then AssertionTester
... ajv schema validator
... some of my assertions here
... (McCool revisits his assertions)
... (and explains the issue with handling pass/fail/not-impl)
Koster: TD schema with ajv hooks
Sebastian: need to leave
... wondering about the goal of the testfest this week
Kaz: thinks the goal of this first testfest effort is simply
looking the implementability of each feature, i.e., can get one
or more implementation for each feature (rather than 2 or more)
McCool: would add some more extra assertions
... if you would like to split some of them, we can do so
... this is a snapshop as of today
... if node-wot is to be updated, that's fine
Sebastian: another question about TD slot on Friday
... will we have a regular TD meeting?
McCool: other than a few issues, you can use the rest for the
TD discussion
Sebastian: we have to make decision about some of the TD issues
Kaz: using the first 30m for testfest wrap-up
... and using the remaining 1.5h for TD discussion
... is that ok?
Sebastian: ok
Toru: going back to the interop test
... pass/fail/secure, etc.
... is it possible to handle partial result?
McCool: (visits interop/ area)
... right now we have producer, consumer, security, status
... you can mention two rows and merge them
... hundred passes and one failure would be failure
... purpose of the summary table
... maybe more information to be captured
... you can add more columns
... my tool will ignore the information
... e.g., you can add columns like cause, comment, etc.
... some failure may be caused by "authentication invalid" with
"expired certificate"
... we can keep track those columns
Toru: maybe just one additional comment column would work
McCool: ok
... probably will become an interoperability document
Lagally: one suggestion
... overview document
... private repos
... very confusing
... and Ege not on the call
(Ege just joined)
McCool: Ege, have you checked my PR?
... Ege's PR is the main one
... and I also have a PR
Ege: now I'm free to work on that
McCool: discovered bunch of issues
... fundamental issue
Kaz: would suggest we resolve that issue via the PR/Issue since
we've already discussed during this call twice :)
Lagally: which of the repos to have the latest tools, schemas,
etc.?
McCool: here is the suggestion
... nothing is in a good shape at the moment...
... we need some more time to stabilize the situation
... (summarize the issue with schema)
Ege: updated the schema Monday evening
McCool: Ege will update the JSON Schema
... will also add much bashing
... within the next two days
Lagally: thingweb resources also on ege's private repo
Ege: will merge it but still need to check
[28]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine
[28] https://github.com/w3c/wot/tree/master/testfest/2018-12-online
Kaz: so we'd like to use only the wot official repo for our
testfest (if possible)
... note that TD files and CSV reports are already to be copied
under w3c/wot repo
... and so I was wondering about the implementation description
HTML
McCool: that can be also copied under the official w3c repo
<McCool>
[29]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
ine/implementations/README.md
[29] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/implementations/README.md
McCool: people can install their implementation description
HTML as well to the above official area
... and please submit your generated CSV report under:
[30]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine/results
... each organization should create a subdirectory there
... and copy your generated CSV files to that subdirectory
[30] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results
Kaz: is it OK to start with this manual approach on the w3c/wot
repo?
Lagally: ok
... so we're encouraged to copy the implementation description
HTML as well to the w3c/wot repo
McCool: yes
... can copy a template for your help
... and I'll copy all the resources to my mmccool repo to
regenerate the report.html
Lagally: ok
Taki: particular assertion, e.g., property, action and event
... not sure we really should split it
McCool: yeah
... can have one combined assertion as a parent with several
child assertions
... can just generate them based on the current TD
Taki: can also have some discussion on Friday
McCool: yeah
Ege: working on the schema fix
McCool: taki, maybe you can merge Ege's PR
<McCool>
[31]https://github.com/w3c/wot-thing-description/pull/323
[31] https://github.com/w3c/wot-thing-description/pull/323
Taki: issue on the JSON Schema
McCool: yeah
... writable
... ah, even older which just include writable
... another PR
[32]https://github.com/w3c/wot-thing-description/pull/324
[32] https://github.com/w3c/wot-thing-description/pull/324
Taki: can merge it
McCool: anything else for today?
(none)
McCool: please work on your CSV files
... will continue to work on the automation
... Ege, we need further collaboration
[the official testfest call adjourned]
Summary of Action Items
Summary of Resolutions
[End of minutes]
__________________________________________________________
Minutes manually created (not a transcript), formatted by
David Booth's [33]scribe.perl version 1.154 ([34]CVS log)
$Date: 2018/12/12 15:45:50 $
[33] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
[34] http://dev.w3.org/cvsweb/2002/scribe/
Received on Wednesday, 12 December 2018 15:47:49 UTC