- From: Kazuyuki Ashimura <ashimura@w3.org>
- Date: Tue, 11 Dec 2018 23:46:40 +0900
- To: Public Web of Things IG <public-wot-ig@w3.org>, public-wot-wg@w3.org
available at:
https://www.w3.org/2018/12/10-wot-pf-minutes.html
also as text below.
Thanks a lot for taking these minutes, Taki!
We'll continue our TestFest tomorrow on Wednesday, Dec. 12 at:
* 4am US Pacific
* 7am US Eastern
* noon UK
* 1pm Europe
* 8pm China
* 9pm Japan
I'll send a reminder of the WebEx coordinate to the Members lists.
Kazuyuki
---
[1]W3C
[1] http://www.w3.org/
- DRAFT -
WoT-TestFest
10 Dec 2018
[2]Agenda
[2] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12.2018_-_Test_Fest_Day_1
Attendees
Present
Kaz_Ashimura, Michael_McCool, Ege_Korkan,
Michael_Koster, Michael_Lagally, Taki_Kamiya,
Tomoaki_Mizushima, Toru_Kawaguchi, Sebastian_Kaebisch,
Kunihiko_Toumura, Zoltan_Kis
Regrets
Chair
McCool
Scribe
Taki
Contents
* [3]Topics
1. [4]TestFest logistics
2. [5]TestFest reports and TD files
* [6]Summary of Action Items
* [7]Summary of Resolutions
__________________________________________________________
<McCool>
[8]https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12
.2018_-_Test_Fest_Day_1
[8] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12.2018_-_Test_Fest_Day_1
<kaz> scribenick: taki
TestFest logistics
<kaz>
[9]https://github.com/w3c/wot/tree/master/testfest/2018-12-onli
ne
[9] https://github.com/w3c/wot/tree/master/testfest/2018-12-online
McCool explaining logistics...
McCool: [10]testfest-1 tagged version is used today.
... There is [11]diff between the published version (oct. 21)
and the testfest version (Nov. 29) also.
... Wednesday, we will use Editors call WebEx.
... Google hangout is ready. I and Panasonic are in there.
... Ege created tools using JSON Schema to validate
implementations.
[10] https://github.com/w3c/wot-thing-description/releases
[11] https://w3c.github.io/wot-thing-description/diff.html
McCool: Test involving validating TDs.
... I documented process.
... Three repos. Ege's playground repo. TD repo where test
tools are in, and WoT repo where we collect results.
<ege>
[12]https://github.com/egekorkan/thingweb-playground/tree/asser
tionTest is the repo and branch of the assertion tester
[12] https://github.com/egekorkan/thingweb-playground/tree/assertionTest
McCool: Run Ege's tool on TD, results goes into result
directory.
... Copy results into testfest directory.
... Copy results to csv file.
... Merge multiple results for a single implementation.
... Copy all results into WoT directory.
... Intel has several TDs today. README is available.
... Panasonic is on google hangout. Described in README.
... Intel has initial results. generated by Ege's tools.
... Ege's tool currently has three tests.
Lagally: There is also Oracle TD.
... instance has pump instance.
... also a few simulators.
... New server setup. New URL and TDs.
... same credentials, just on different machine.
McCool: Please fill out result directory, and use Ege's tool.
McCool is explaining how to use Ege's tool....
McCool: It can pass, fail, or not-implemented.
Kaz: Which result goes into which repository/directory?
McCool: Put the real result in
WoT/testfest/2018-12-online/results.
<kaz>
[13]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine/results
[13] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results
McCool: Raw results should be merged using scripts.
<kaz> everybody is encouraged to submit their reports to the
above master repo under their own sub directory (after creating
those sub directory, e.g., "Intel"
McCool: For example, Intel/intel-ocf.csv.
... For certain tests, we need to do manual test.
... testspec.html in TD repo lists assertions and how to test
them.
... We need to figure out which ones we can use JSON-Schema.
... We can generate new tests.
... ERCIM, Intel, Panasonic, Siemens implementation
descriptions are available.
Lagally: I will provide Oracle's implementation description.
McCool: Your entire simulator is one implementation, right?
... Fujitsu and Hitachi, please write them up.
McCool is showing report...
McCool: I am going to update the result in the report with real
result.
... Color red indicates feature at risk.
... As we finish test, we can convert them into green.
... Test Specification is in Appendix A.
Ege: we have version, action-objects and additional-context
tests.
McCool: Mark everything in red for ones we do not have test
yet.
... For security binding test, we need to do manual HTTP test.
Lagally: Can you show us one whole process for validation test?
Ege: I am going to show.
Ege is preparing for that...
McCool: In the meantime, I am going to update the report.
Ege is showing screen...
Ege: Playground repo. AssertionTester/Assertions folder has
JSON-Schemas.
... td-action-objects.json, and also for context and version.
There are three JSON-Schemas now.
... Run each with "node".
... It generates result.csv. Details are in result.json.
McCool: You can add lines in result.csv. Tools will ignore
them.
Ege: @context is tricky to test.
Ege showing td-action-objects test result...
Ege: if output in TD is removed, it causes an error.
McCool: mergeresults.js merges multiple result csv into one.
[14]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
ine/README.md
[14] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/README.md
Ege: I have result.csv and result2.csv.
McCool: It takes two or more test results as input, and writes
out to std output.
Ege is merging results...
Ege shows the merged results...
McCool: Multiple results for one implementation. If you have
multiple implementations, keep them as separate results.
Lagally: playground still uses old syntax and old schema.
Ege: It is in plan to update.
Ege showing thingweb-playground.
Ege: I will update.
McCool: Let's take a 10 minutes break.
... We can do maintenance work during that.
... We will resume in 10 minutes.
<kaz_> [break]
TestFest reports and TD files
[15]Updated report
[15] https://cdn.staticaly.com/gh/mmccool/wot-thing-description/updated-test-results/testing/report.html
McCool is showing report.
McCool: One assertion passed.
... Everything in pink, we need to test.
... For assertions that we do not have tests, I marked
everything in red.
... How do we generate JSON-Schemas.
Ege: Write schemas if we can test them with schemas.
McCool: atrisk.csv contains the list.
... We can divide the work among people.
... Would that accelerate the work?
... We can create more schemas before Wednesday.
<McCool>
[16]https://github.com/mmccool/wot-thing-description/blob/updat
ed-test-results/testing/inputs/atrisk.csv
[16] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv
McCool: I can split the work with Ege and work on those.
<McCool>
[17]https://raw.githubusercontent.com/mmccool/wot-thing-descrip
tion/updated-test-results/testing/report.html
[17] https://raw.githubusercontent.com/mmccool/wot-thing-description/updated-test-results/testing/report.html
McCool: We should have a more fleshed-out set of tests.
... We will run those tests on Wednesday.
Lagally: there is a bug in setup instruction. In the main
README in test fest.
... Section 2. Now I see it is fixed.
McCool: People should go off and try to work on this.
... For non-schema test, we will need to do network tests.
... security tests probably need network test.
Ege: We can also use testbench.
McCool: how can we check security assertions? e.g.
basic-authentication.
Ege: Penetration test is another.
McCool: Elena sent me some instructions.
... Burp sends stimulus and gets response. need to check if it
can do Authencation test.
... I will talk with Elena later.
... Adversarial testing is required by charter.
... This week, we will finish JSON-Schema based testing.
... at least for the current set of assertions.
... I can do more work before Wednesday.
Ege: I will update playground before Wednesday.
McCool: I will start from the the bottom of the list, Ege work
from the top.
Toumura: Where should I upload files?
<McCool>
[18]https://github.com/mmccool/wot-thing-description/blob/updat
ed-test-results/testing/inputs/atrisk.csv
[18] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv
<McCool>
[19]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results
[19] https://github.com/mmccool/wot-thing-description/tree/updated-test-results
McCool: Please make a PR against McCool's fork.
<McCool> please make sure to do it against the
updated-test-results branch
<McCool>
[20]https://github.com/mmccool/wot-thing-description/tree/updat
ed-test-results/testing/inputs/implementations
[20] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations
Toumura: I will upload files. Thank you.
<McCool>
[21]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
ine/TDs
[21] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs
McCool: Check TD in to the repo.
WoT/testfest/2018-12-online/TDs
... My TD, e.g. security part needs to be updated to the new TD
version.
... By Friday, we can fill in most of Result column in the
report.
Lagally: Is the JSON-Schema up-to-date?
McCool: Date is wrong.
... It has some of the changes. Not sure if it is all
up-to-date.
Lagally: Please make sure Ege's schema is consistent with the
spec JSON-Schema.
Ege: Playground has batch test.
... We can run with valid TDs and invalid TDs for JSON-Schema
debugging.
... It still does not response field.
McCool: missing response in JSON-schema. case-insensitive
mediatype and content type.
... We will create PR to fix the JSON-schema by Friday.
... Let's meet on Wednesday. Starting Editor's call time (4am
US Pacific, 7am US Eastern, noon UK, 1pm Europe, 8pm China, 9pm
Japan).
Lagally: 1pm in Germany.
McCool: We may not need three hours.
... anything else?
... We can continue to use hangout.
... let's adjourn. I will give time back. Please continue to
work.
Summary of Action Items
Summary of Resolutions
[End of minutes]
__________________________________________________________
Minutes manually created (not a transcript), formatted by
David Booth's [22]scribe.perl version 1.154 ([23]CVS log)
$Date: 2018/12/11 14:37:22 $
[22] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
[23] http://dev.w3.org/cvsweb/2002/scribe/
Received on Tuesday, 11 December 2018 14:47:46 UTC