W3C home > Mailing lists > Public > public-wot-ig@w3.org > June 2018

[PlugFest/Test] 13 June 2018

From: Kazuyuki Ashimura <ashimura@w3.org>
Date: Fri, 22 Jun 2018 11:26:07 +0900
Message-ID: <CAJ8iq9WVw51BG7a5s4B_xhFofJp+UKHUM+hgfJdJW3BVxFQiTQ@mail.gmail.com>
To: Public Web of Things IG <public-wot-ig@w3.org>, public-wot-wg@w3.org
available at:
  https://www.w3.org/2018/06/13-wot-pf-minutes.html

also as text below.

Thanks a lot for taking these minutes, Michael McCool!

Kazuyuki

---

   [1]W3C

      [1] http://www.w3.org/

                               - DRAFT -

                              WoT PlugFest

13 Jun 2018

Attendees

   Present
          Kaz_Ashimura, Federico_Sismondi, Matthias_Kovatsch,
          Ryuichi_Matsukura, Toru_Kawaguchi, Takeshi_Sano,
          Tomoaki_Mizuhsima, Kunihiko_Toumura, Michael_McCool,
          Michael_Lagally, Sebastian_Kaebisch, Michael_Koster,
          Ege_Korkan, Dave_Raggett

   Regrets

   Chair
          Matsukura, Koster, McCool

   Scribe
          McCool

Contents

     * [2]Topics
         1. [3]Agenda
         2. [4]F-interop, by Federico
         3. [5]PlugFest prep
     * [6]Summary of Action Items
     * [7]Summary of Resolutions
     __________________________________________________________

   <kaz> scribenick: McCool

Agenda

   two topics: plugfest prep and testing

   McCool: I suggest we do Fredrico's pres on testing first, so he
   can leave if he has to

F-interop, by Federico

   Federico: presentation, then a little demonstration on what is
   being done for CoAP
   ... WoT may be different, but it will give you an idea
   ... F-Interop is an H2020 project
   ... almost done, Oct 2018, but tools will be maintained
   ... main objectives is to develop and provide online testing
   tools
   ... including interoperabilty testing
   ... members into INRIA, ETSI, etc.
   ... currently, SoA is F2F events ("plugfests")
   ... but F2F plugfests have issues...
   ... short, if you hit a bug, can wreck entire event
   ... so F-interop wants to provide online and remote process
   ... so can do continuous testing
   ... system description
   ... runs in cloud, can connect both through Web GUI and to IUT
   (Item under test)
   ... can also have simulated IUTs to connect to running in cloud
   ... various tools: communication, including tunneling (bypass
   NATs and Firewalls); coordinating tests based on test
   descriptions; traffic sniffing; dissecting messages; traffic
   analysis
   ... results -> issue PASS/FAIL/INCONCLUSIVE based on test
   description
   ... there is a tutorial available online that you can look at
   ... initially looked at a couple of IoT standards: CoAP and
   6TiSCH
   ... started with ETSI plugfest test descriptions
   ... now looking at extensions: oneM2M, OMA LwM2M, 6LoWPAN,
   and... hopefully WoT
   ... Ex: CoAP test case (test spec)
   ... did not make these up, used ETSI test specs
   ... gives objectives, configuration, pretest condition, then
   test sequence
   ... in this case, test sequence starts with a stimulus, then a
   set of checks on responses
   ... so what are WoT test cases?
   ... could not find anything similar to CoAP
   ... would be nice to have things more formal

   Ege: looks a bit like invoke simple action
   ... but autogenerated from TD
   ... can generate payload from description
   ... can send payload and check response
   ... for example, can check that output satisfies schema

   Federico: that makes sense...
   ... but not predefining coverage, looking at capabilities of
   device, making sure those work

   Ege: test description basically is given by the TD

   Federico: nice approach, but at the end of the day doing the
   same kind of tests that were in the BJ plugfest
   ... but a more advanced automation

   McCool: time check, only 5-10m more

   Federico: ok, I'd like to take the time to do a demo
   ... can access for free

   [8]https://go.f-interop.eu

      [8] https://go.f-interop.eu/

   Federico: can do both single user and user-to-user tests
   ... or can use a reference-based test
   ... pick device, and test case
   ... there is also a playground to launch requests without a
   predefined test
   ... start it, deploys instance in Docker containers and runs
   tests
   ... need to set up environment, communication, etc.
   ... communication can include a virtual network
   ... including support for UDP
   ... based on very formal set of test cases
   ... at the end get a report

   scribenick: kaz

   McCool: we should clarify which part is opensource and which is
   not

   scribenick: McCool

   McCool: also, I think a small team (Ege, Federico, Dave) should
   go off and put together a small proof-of-concept using just the
   OSS F-Interop tools and then report back

   scribenick: kaz

   Kaz: agree we should clarify which is open and which is not.
   ... also we should think about levels of testing again, and
   consider which level/part is this work (F-Interop) should be
   applied
   ... regarding the smaller call, would suggest at least all the
   co-Chairs and the staff contacts should be included

   scribenick: McCool

   McCool: I was thinking a working meeting with just the
   implementors, but let's discuss via email

   Koster: we should get in the habit of publishing an agenda
   ... so people who are just interested in the pf, for instance,
   know when to join

PlugFest prep

   Koster: did prepare one slide as a discussion point

   <ryuichi>
   [9]https://github.com/w3c/wot/blob/master/plugfest/2018-bundang
   /preparation.md

      [9] https://github.com/w3c/wot/blob/master/plugfest/2018-bundang/preparation.md

   Matsukura: want to first explain preparation.md structure
   ... sections: first is intro, info from last pf
   ... second section is new information
   ... 2.1 is a table for participants to fill out
   ... please insert your information in this table
   ... 2.2 is checking points
   ... these should also be discussed in the results.md for your
   report-out
   ... can follow this template and carry information forward
   ... then 2.3 are other issues
   ... if any suggestions to improve please make them
   ... section 4 are logistical requirements: network, power, etc.
   ... I also made an example, fujitsu-preparation.md
   ... information is the same as last plugfest

   Koster: table is useful, people can just add a row
   ... checkpoints are also useful for results.md
   ... but not everyone used the same template
   ... or maybe it was
   ... but edited
   ... section 2.3, other issues... my material
   ... kind of a mixture of different things; experimental
   features, validation, binding templates, services, etc.
   ... this section will go away/be integrated

   McCool: I think we should prioritize, e.g. security should be
   mandatory
   ... also, "other" maybe should be broken into "extras" and
   "experimental features". Latter are things that are candidates
   for inclusion in the standard but need so experimentation

   Sebastian: I think we need fewer "demos" and more "plugging"
   ... we really need to check how things can work together
   ... that would really help to improve the standard
   ... we need to actually focus on testing interoperability and
   the standard
   ... would like to see more "client" implementations
   ... a lot of devices, not enough systems trying to use other
   devices

   Koster: agree. we also need more formal test plans and better
   record keeping
   ... agree also that we need more applications
   ... we are building a system, we need to have all the
   components present
   ... need so see more interactions

   Sebastian: rather than bringing a lot of things, we should
   focus on the content and interactions
   ... we should concentrate on the functionality

   Federico: table of features that implementations should
   implement
   ... at the end of the day, pulling tables to see what features
   are being tested

   Koster: when you have a set of test cases, it can stand in for
   a standard application
   ... but we also need some ad-hoc applications to test
   integration
   ... and use cases
   ... for example, using semantic search to find a bunch of
   sensors and analyse them for some purpose
   ... in other words, system-level applications
   ... first cover functional, then cover system
   ... functional: list features you want to test, and then
   generate tests against them
   ... then, assuming functional tests done, how do we orchestrate
   them to solve a higher-level problem
   ... we don't really know what "interoperability" is yet

   McCool: I think the second part is more like "what is it good
   for", eg. use case analysis
   ... functional testing is just "does the implementation match
   the spec"
   ... system testing is "does the spec have the right things"

   scribenick: kaz

   Kaz: was on the queue and wanted to mention something like
   McCool :)
   ... probably we should have a combined demo scenario on the
   main preparation.md, and clarify what kind of use
   cases/features are included in that. And then clarify details
   in the company-preparation.md and test/report if it works after
   the plugfest

   scribenick: McCool

   Koster: so maybe we need to think about to structure this in
   the preparation.md file
   ... would still like to do system-level scenario exploration
   ... but should think about how to structure it better

   Matsukura: adding a "use case" section might be useful
   ... added, please insert your idea

   Koster: my presentation... more of the same
   ... we are building a system, e.g. an application that
   orchestrates multiple things
   ... but also have components and roles

   Koster: to test components need to have them in context, eg. in
   a system (application)
   ... several components are not servients...
   ... or are they?
   ... Servients use the scripting API, interacts with Thing
   Directory
   ... if it only exposes a TD, e.g. is an endpoint device, then
   it's not a servient
   ... system architecture includes both local and remote parts
   ... I myself am going to bring fewer things but focus on
   applications
   ... but applications have a bunch of common components and
   patterns, e.g. local/remote directories, proxies, etc.
   ... also some new concepts, like "Bridge Servient"; can also
   think of as a "role"
   ... also notable is that proxy-proxy connection doesn't matter
   ... but need to go through the proxy to get to local devices
   from the cloud
   ... other way does not necessarily need the proxy
   ... want to list the patterns separately
   ... but would like to take an application-layer approach
   ... register things locally
   ... tell the proxy I want to make available
   ... can use the information in the TD to figure out how to
   implement
   ... missing an arrow here, some extra orchestration

   McCool: noticed that the directory and the proxy are paired
   ... and also notable is that TD may have to be modified in
   various ways before being registered in the could TDir
   ... could be the other way around: could register in the TDir,
   then proxy can pick it up

   Koster: another point: we want to think about applications and
   how to deploy them
   ... I am thinking about using node-wot to deploy applications
   ... what I suggest we start thinking about
   ... we need goals that are bigger than just turning a light on
   ... what are the higher-level problems we are trying to solve,
   and what are their requirements?
   ... for example, what are the security requirements

   Sebastian: don't think we need to spend so much time on the
   application
   ... I am thinking more that we just do more 1:1 testing
   ... and try to use all the interaction patterns
   ... we could go further and do mash-ups
   ... even that simple approach exposes many issues

   Koster: I was thinking about how to accomplish what you say...
   ... can create a control panel in a client that lets you
   connect to all the

   McCool: could be a simple tool that generates a web dashboard
   given a TD

   Kaz: can clarify use cases, brief, in preparation.md
   ... so people can understand the "story"

   Sebastian: if you recall in BJ
   ... we tried to identified what worked well and what didn't
   ... I am a little worried about our timeline
   ... we only have a couple more plugfests to get experience with
   TD in practice
   ... then we are supposed to be done
   ... not convinced we are testing things completely enough
   ... we need to at least do all the basic component tests

   scribenick: kaz

   Kaz: would agree with your concern. however, each use case
   doesn't have to be very fancy but can be something simple, and
   the main preparation use case scenario simply refer to them and
   clarify concrete demo scenarios. sometimes in parallel and
   sometimes sequentially using dashboard, etc.

   scribenick: McCool

   Koster: this is definitely a different level
   ... we also need the basic testing at the plugfest
   ... but... the low-level testing is obviously a higher priority
   ... although we also need to move forward with system testing
   too

   Toru: do we really need scripting API?

   McCool: my opinion is that the minimum is that it "consumes" a
   TD
   ... if it only exposes a TD it's an endpoint device

   Koster: I do want to motivate using the TDir as well
   ... but, yeah...

   <Zakim> kaz, you wanted to ask about the availability of
   Koster's slides :)

   Matsukura: our implementation doesn't use scripting API, so...
   but we need to better focus on "applications"
   ... our implementation doesn't use scripting API, though

   Kaz: can you make slides available?

   [10]Koster's slides

     [10] https://github.com/w3c/wot/blob/master/plugfest/2018-bundang/Plugfest-System-Arch-20180613.pdf

   Lagally: suggestion
   ... suggest to add a legend to the architecture slides, to be
   able to understand the meaning of the colours of the boxes

   Koster: sure

   Matsukura-san: AOB?

   ok, adjourn

Summary of Action Items

Summary of Resolutions

   [End of minutes]
     __________________________________________________________


    Minutes formatted by David Booth's [11]scribe.perl version
    1.152 ([12]CVS log)
    $Date: 2018/06/18 08:28:28 $

     [11] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
     [12] http://dev.w3.org/cvsweb/2002/scribe/
Received on Friday, 22 June 2018 02:27:16 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:27:33 UTC