[wot-test] minutes - 2 May 2018

available at:
  https://www.w3.org/2018/05/02-wot-test-minutes.html

also as text below.

Thanks a lot for taking these minutes, Daniel!

Kazuyuki

---

   [1]W3C

      [1] http://www.w3.org/

                               - DRAFT -

                              WoT Testing

02 May 2018

   [2]Agenda

      [2] https://www.w3.org/WoT/IG/wiki/Test_WebConf#Agenda

Attendees

   Present
          Kaz_Ashimura, Michael_McCool, Daniel_Peintner,
          Tomoaki_Mizushima, Toru_Kawaguchi, Matthias_Kovatsch,
          Michael_Lagally, Michael_Koster

   Regrets

   Chair
          McCool

   Scribe
          dape

Contents

     * [3]Topics
         1. [4]Agenda
         2. [5]Categories of "testing"
         3. [6]Reference implementations?
         4. [7]Levels of "testing"
         5. [8]Actions
         6. [9]Testing manager?
         7. [10]Time slot for the testing call?
     * [11]Summary of Action Items
     * [12]Summary of Resolutions
     __________________________________________________________

   <kaz> scribenick: dape

Agenda

   <kaz> Agenda:
   [13]https://www.w3.org/WoT/IG/wiki/Test_WebConf#Agenda

     [13] https://www.w3.org/WoT/IG/wiki/Test_WebConf#Agenda

   McCool: Any additions to the agenda?
   ... we have plan
   ... 1. step, markup spec with normative assertions

   <kaz> [14]plan

     [14] https://github.com/w3c/wot/blob/master/testing/plan.md

   McCool: RFC 2119 keywords
   ... anyone started with normative assertions ?

   Kaz: think Sebastian and Zoltan started with it...

   McCool: Section "Conformance" needs to be added
   ... particular words MAY; MUST, ... etc need to be used

   <kaz> [15]scripting api draft

     [15] https://rawgit.com/zolkis/wot-scripting-api/229a1bcbc0e08b3af259b18e3d1c3f2ef065877c/index.html

   Kaz: Zoltan added conformance section as Section 9

   <kaz> [16]section 9

     [16] https://rawgit.com/zolkis/wot-scripting-api/229a1bcbc0e08b3af259b18e3d1c3f2ef065877c/index.html#conformance

   McCool: will be protocol binding separate document till to the
   end?

   Koster: plan to provide normative statements for TD

   McCool: for security it is the same
   ... assume normative statements are a big chunk

Categories of "testing"

   McCool: 1. specification validation (required by W3C)
   ... spec implementable and consistent
   ... 2. Implementation testing
   ... check implementation satisfy requirements
   ... a) scripting API
   ... b) TD validation
   ... general useful tool
   ... c) network interface described by TD
   ... would like to have an automated thing that tests this
   ... wonder which is actually necessary

   Koster: do they need to be reference implementation

   kaz: usually we don't have "reference" implementations
   ... just need 2 implementations for each feature

   McCool: 3. category is interop testing
   ... difficult problem

Reference implementations?

   Koster: we can have reference implementation
   ... reference implementation helps to get started

   McCool: Could be online tool

   Koster: PlugFest could be venue

   Kaz: usual W3C specifications are related to Web browsers
   ... so the W3C Testing Framework mainly handles Web browser
   testing
   ... on the other hand, we need server side also for WoT
   ... that's why, reference impl for server side is useful

Levels of "testing"

   Daniel: Wonder whether w3c testing only means spec validation ?

   Kaz: the main purpose of "testing" for Candidate Recommendation
   transition is spec validation

   McCool: we can add levels
   ... and we missed security testing

   Koster: there is our spec we test against but there is more
   (e.g., in security)

   McCool: we can just have recommendation
   ... provide framework how to test

   Kaz: maybe we might want to rename "security testing" to
   "advanced testing" which include security testing

   McCool: I believe we have statement in charter that requires us
   to test security
   ... in our tooling we can point to how to improve security

   Kaz: ah, that's true

   <kaz> [17]WG Charter, fyi

   In order to enhance the security of WoT systems, we will also
   generate and implement a security testing plan which will
   include both functional and adversarial testing of the proposed
   standards and their implementations. We will only recommend an
   implementation of the proposed standards for use in production
   once it has passed such testing.
   ]]

     [17] https://www.w3.org/2016/12/wot-wg-2016.html

   McCool: I think we need validation tool
   ... "functional testing" could be basic authentication ...
   ... hope we can reuse things that already exists

   Matthias: w.r.t. CoAP there is some penetration testing
   ... for CoAP we broke it down by feature
   ... test it pair-wise
   ... also test it with security enabled (difficult and maybe
   also impossible with all combinations)

   McCool: test common combination
   ... w.r.t. reference implementation
   ... we need 2 separate implementations
   ... for example in the case of scripting we might just have a
   limited number
   ... need to focus breaking things down by feature
   ... after that we can create table with common combinations
   ... maybe PlugFest could be used to implement test cases

   Koster: in physical realization: we need system under test and
   test tool
   ... can have scripts for exposed thing and consumed thing using
   node-wot

   McCool: for PlugFest (or after) I would like to have a skeleton
   testing framework in place
   ... implies that before the PlugFest we need test description
   (before that test assertions)

   Koster: need to talk about how to implement
   ... what new tools we need

Actions

   McCool: Next, will create section on wiki for "Actions"
   ... normative assertions
   ... test descriptions
   ... each group should think about this
   ... we might want to have MD template

Testing manager?

   McCool: Q: Is there any test expert who could help us/me?

   Matthias: mainly time issue
   ... talked to Ege Korkan
   ... he could volunteer
   ... will ask him

Time slot for the testing call?

   Koster: should also re-consider time slot

   McCool: Kaz, can you do a doodle?

   Kaz: will do
   ... BTW, we might want to think about collaboration between
   PlugFest and Testing
   ... PlugFest call should start again
   ... possibly done within this testing call, but maybe need a
   separate call

   <kaz> Koster: right. plugfest has a bit different logistics
   cycle

   <kaz> [adjourned]

Summary of Action Items

Summary of Resolutions

   [End of minutes]
     __________________________________________________________


    Minutes formatted by David Booth's [18]scribe.perl version
    1.152 ([19]CVS log)
    $Date: 2018/05/04 05:54:46 $

     [18] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
     [19] http://dev.w3.org/cvsweb/2002/scribe/

Received on Friday, 4 May 2018 05:59:24 UTC