W3C home > Mailing lists > Public > public-wot-ig@w3.org > December 2018

[TestFest] minutes - 10 December 2018

From: Kazuyuki Ashimura <ashimura@w3.org>
Date: Tue, 11 Dec 2018 23:46:40 +0900
Message-ID: <CAJ8iq9WdEAz-qX+k8sagPVU9RsjJY2dUkpOqKhOAS-PPK-JUCw@mail.gmail.com>
To: Public Web of Things IG <public-wot-ig@w3.org>, public-wot-wg@w3.org
available at:
  https://www.w3.org/2018/12/10-wot-pf-minutes.html

also as text below.

Thanks a lot for taking these minutes, Taki!

We'll continue our TestFest tomorrow on Wednesday, Dec. 12 at:
* 4am US Pacific
* 7am US Eastern
* noon UK
* 1pm Europe
* 8pm China
* 9pm Japan

I'll send a reminder of the WebEx coordinate to the Members lists.

Kazuyuki

---

   [1]W3C

      [1] http://www.w3.org/

                               - DRAFT -

                              WoT-TestFest

10 Dec 2018

   [2]Agenda

      [2] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12.2018_-_Test_Fest_Day_1

Attendees

   Present
          Kaz_Ashimura, Michael_McCool, Ege_Korkan,
          Michael_Koster, Michael_Lagally, Taki_Kamiya,
          Tomoaki_Mizushima, Toru_Kawaguchi, Sebastian_Kaebisch,
          Kunihiko_Toumura, Zoltan_Kis

   Regrets

   Chair
          McCool

   Scribe
          Taki

Contents

     * [3]Topics
         1. [4]TestFest logistics
         2. [5]TestFest reports and TD files
     * [6]Summary of Action Items
     * [7]Summary of Resolutions
     __________________________________________________________

   <McCool>
   [8]https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12
   .2018_-_Test_Fest_Day_1

      [8] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12.2018_-_Test_Fest_Day_1

   <kaz> scribenick: taki

TestFest logistics

   <kaz>
   [9]https://github.com/w3c/wot/tree/master/testfest/2018-12-onli
   ne

      [9] https://github.com/w3c/wot/tree/master/testfest/2018-12-online

   McCool explaining logistics...

   McCool: [10]testfest-1 tagged version is used today.
   ... There is [11]diff between the published version (oct. 21)
   and the testfest version (Nov. 29) also.
   ... Wednesday, we will use Editors call WebEx.
   ... Google hangout is ready. I and Panasonic are in there.
   ... Ege created tools using JSON Schema to validate
   implementations.

     [10] https://github.com/w3c/wot-thing-description/releases
     [11] https://w3c.github.io/wot-thing-description/diff.html

   McCool: Test involving validating TDs.
   ... I documented process.
   ... Three repos. Ege's playground repo. TD repo where test
   tools are in, and WoT repo where we collect results.

   <ege>
   [12]https://github.com/egekorkan/thingweb-playground/tree/asser
   tionTest is the repo and branch of the assertion tester

     [12] https://github.com/egekorkan/thingweb-playground/tree/assertionTest

   McCool: Run Ege's tool on TD, results goes into result
   directory.
   ... Copy results into testfest directory.
   ... Copy results to csv file.
   ... Merge multiple results for a single implementation.
   ... Copy all results into WoT directory.
   ... Intel has several TDs today. README is available.
   ... Panasonic is on google hangout. Described in README.
   ... Intel has initial results. generated by Ege's tools.
   ... Ege's tool currently has three tests.

   Lagally: There is also Oracle TD.
   ... instance has pump instance.
   ... also a few simulators.
   ... New server setup. New URL and TDs.
   ... same credentials, just on different machine.

   McCool: Please fill out result directory, and use Ege's tool.

   McCool is explaining how to use Ege's tool....

   McCool: It can pass, fail, or not-implemented.

   Kaz: Which result goes into which repository/directory?

   McCool: Put the real result in
   WoT/testfest/2018-12-online/results.

   <kaz>
   [13]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine/results

     [13] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results

   McCool: Raw results should be merged using scripts.

   <kaz> everybody is encouraged to submit their reports to the
   above master repo under their own sub directory (after creating
   those sub directory, e.g., "Intel"

   McCool: For example, Intel/intel-ocf.csv.
   ... For certain tests, we need to do manual test.
   ... testspec.html in TD repo lists assertions and how to test
   them.
   ... We need to figure out which ones we can use JSON-Schema.
   ... We can generate new tests.
   ... ERCIM, Intel, Panasonic, Siemens implementation
   descriptions are available.

   Lagally: I will provide Oracle's implementation description.

   McCool: Your entire simulator is one implementation, right?
   ... Fujitsu and Hitachi, please write them up.

   McCool is showing report...

   McCool: I am going to update the result in the report with real
   result.
   ... Color red indicates feature at risk.
   ... As we finish test, we can convert them into green.
   ... Test Specification is in Appendix A.

   Ege: we have version, action-objects and additional-context
   tests.

   McCool: Mark everything in red for ones we do not have test
   yet.
   ... For security binding test, we need to do manual HTTP test.

   Lagally: Can you show us one whole process for validation test?

   Ege: I am going to show.

   Ege is preparing for that...

   McCool: In the meantime, I am going to update the report.

   Ege is showing screen...

   Ege: Playground repo. AssertionTester/Assertions folder has
   JSON-Schemas.
   ... td-action-objects.json, and also for context and version.
   There are three JSON-Schemas now.
   ... Run each with "node".
   ... It generates result.csv. Details are in result.json.

   McCool: You can add lines in result.csv. Tools will ignore
   them.

   Ege: @context is tricky to test.

   Ege showing td-action-objects test result...

   Ege: if output in TD is removed, it causes an error.

   McCool: mergeresults.js merges multiple result csv into one.

   [14]https://github.com/w3c/wot/blob/master/testfest/2018-12-onl
   ine/README.md

     [14] https://github.com/w3c/wot/blob/master/testfest/2018-12-online/README.md

   Ege: I have result.csv and result2.csv.

   McCool: It takes two or more test results as input, and writes
   out to std output.

   Ege is merging results...

   Ege shows the merged results...

   McCool: Multiple results for one implementation. If you have
   multiple implementations, keep them as separate results.

   Lagally: playground still uses old syntax and old schema.

   Ege: It is in plan to update.

   Ege showing thingweb-playground.

   Ege: I will update.

   McCool: Let's take a 10 minutes break.
   ... We can do maintenance work during that.
   ... We will resume in 10 minutes.

   <kaz_> [break]

TestFest reports and TD files

   [15]Updated report

     [15] https://cdn.staticaly.com/gh/mmccool/wot-thing-description/updated-test-results/testing/report.html

   McCool is showing report.

   McCool: One assertion passed.
   ... Everything in pink, we need to test.
   ... For assertions that we do not have tests, I marked
   everything in red.
   ... How do we generate JSON-Schemas.

   Ege: Write schemas if we can test them with schemas.

   McCool: atrisk.csv contains the list.
   ... We can divide the work among people.
   ... Would that accelerate the work?
   ... We can create more schemas before Wednesday.

   <McCool>
   [16]https://github.com/mmccool/wot-thing-description/blob/updat
   ed-test-results/testing/inputs/atrisk.csv

     [16] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv

   McCool: I can split the work with Ege and work on those.

   <McCool>
   [17]https://raw.githubusercontent.com/mmccool/wot-thing-descrip
   tion/updated-test-results/testing/report.html

     [17] https://raw.githubusercontent.com/mmccool/wot-thing-description/updated-test-results/testing/report.html

   McCool: We should have a more fleshed-out set of tests.
   ... We will run those tests on Wednesday.

   Lagally: there is a bug in setup instruction. In the main
   README in test fest.
   ... Section 2. Now I see it is fixed.

   McCool: People should go off and try to work on this.
   ... For non-schema test, we will need to do network tests.
   ... security tests probably need network test.

   Ege: We can also use testbench.

   McCool: how can we check security assertions? e.g.
   basic-authentication.

   Ege: Penetration test is another.

   McCool: Elena sent me some instructions.
   ... Burp sends stimulus and gets response. need to check if it
   can do Authencation test.
   ... I will talk with Elena later.
   ... Adversarial testing is required by charter.
   ... This week, we will finish JSON-Schema based testing.
   ... at least for the current set of assertions.
   ... I can do more work before Wednesday.

   Ege: I will update playground before Wednesday.

   McCool: I will start from the the bottom of the list, Ege work
   from the top.

   Toumura: Where should I upload files?

   <McCool>
   [18]https://github.com/mmccool/wot-thing-description/blob/updat
   ed-test-results/testing/inputs/atrisk.csv

     [18] https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv

   <McCool>
   [19]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results

     [19] https://github.com/mmccool/wot-thing-description/tree/updated-test-results

   McCool: Please make a PR against McCool's fork.

   <McCool> please make sure to do it against the
   updated-test-results branch

   <McCool>
   [20]https://github.com/mmccool/wot-thing-description/tree/updat
   ed-test-results/testing/inputs/implementations

     [20] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations

   Toumura: I will upload files. Thank you.

   <McCool>
   [21]https://github.com/w3c/wot/tree/master/testfest/2018-12-onl
   ine/TDs

     [21] https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs

   McCool: Check TD in to the repo.
   WoT/testfest/2018-12-online/TDs
   ... My TD, e.g. security part needs to be updated to the new TD
   version.
   ... By Friday, we can fill in most of Result column in the
   report.

   Lagally: Is the JSON-Schema up-to-date?

   McCool: Date is wrong.
   ... It has some of the changes. Not sure if it is all
   up-to-date.

   Lagally: Please make sure Ege's schema is consistent with the
   spec JSON-Schema.

   Ege: Playground has batch test.
   ... We can run with valid TDs and invalid TDs for JSON-Schema
   debugging.
   ... It still does not response field.

   McCool: missing response in JSON-schema. case-insensitive
   mediatype and content type.
   ... We will create PR to fix the JSON-schema by Friday.
   ... Let's meet on Wednesday. Starting Editor's call time (4am
   US Pacific, 7am US Eastern, noon UK, 1pm Europe, 8pm China, 9pm
   Japan).

   Lagally: 1pm in Germany.

   McCool: We may not need three hours.
   ... anything else?
   ... We can continue to use hangout.
   ... let's adjourn. I will give time back. Please continue to
   work.

Summary of Action Items

Summary of Resolutions

   [End of minutes]
     __________________________________________________________


    Minutes manually created (not a transcript), formatted by
    David Booth's [22]scribe.perl version 1.154 ([23]CVS log)
    $Date: 2018/12/11 14:37:22 $

     [22] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
     [23] http://dev.w3.org/cvsweb/2002/scribe/
Received on Tuesday, 11 December 2018 14:47:46 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 11 December 2018 14:47:47 UTC