[PlugFest/Test] minutes - 21 November 2018

available at:
  https://www.w3.org/2018/11/21-wot-pf-minutes.html

also as text below.

Thanks a lot for taking these minutes, Matthias!

Kazuyuki

---

   [1]W3C

      [1] http://www.w3.org/

                               - DRAFT -

                              WoT-PF/Test

21 Nov 2018

   [2]Agenda

      [2] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_14.11.2018

Attendees

   Present
          Kaz_Ashimura, Ege_Korkan, Michael_McCool,
          Toru_Kawaguchi, Matthias_Kovatsch, Michael_Koster,
          Kunihiko_Toumura, Michael_Lagally, Tomoaki_Mizushima

   Regrets

   Chair
          McCool

   Scribe
          mkovatsc

Contents

     * [3]Topics
         1. [4]Update on TD Test Plan
     * [5]Summary of Action Items
     * [6]Summary of Resolutions
     __________________________________________________________

   <McCool>
   [7]https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_21.11
   .2018

      [7] https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_21.11.2018

   <scribe> scribenick: mkovatsc

Update on TD Test Plan

   <kaz> [8]PR 290

      [8] https://github.com/w3c/wot-thing-description/pull/290

   McCool: new section on what counts as implementation
   ... added implementation section for Intel
   ... mk added implementation section for Siemens
   ... need to figure out how implementation is counted when a sw
   component is shared

   <McCool>
   [9]https://github.com/mmccool/wot-thing-description/tree/update
   d-test-results/testing

      [9] https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing

   McCool: now using terms "TD Consumer" and "TD Producer"
   ... added several CSV files
   ... changes to the document need to go into template.html
   ... it imports other files
   ... implementations folder contains HTML blocks for each
   Member, listing and describing their implementations and a
   leading testimonial
   ... please give each implementation a unique id
   ("impl-<member>-<impl>")

   <Zakim> kaz, you wanted to mention we should identify if each
   implementation is based on different code-base

   Kaz: two points:
   ... 1. implementation should relate to specific code-base
   ... 2. should we really use this repo for this?

   McCool: code-base aspect is known
   ... my repo is only used to speed up things (merge into master
   requires review)
   ... testing focuses on TD, thus it is in TD repo

   Kaz: you could become moderator for TD repo to speed up

   McCool: it is more about authorization by the Editors, as spec
   was touched by PR
   ... best we wait until PR merge before people open PRs for impl
   information
   ... best start by retrieving the template from my repo and make
   PR to master

   Toru: "TD Producer" has the notion of an automatic generator.
   What about manual TDs?

   <kaz> Matthias: we're learning more and more about the actual
   procedure now

   <kaz> ... the current assertion focused on whether it's correct
   or not

   <kaz> ... need to see proper terms

   <inserted> kaz: During the WoT Chairs call today, I also
   mentioned "consumedThing" and "exposedThing" as possible terms,
   but we can revisit the concrete terminology later.
   ... The point at the moment is the terminology for the
   Implementation Report should be consistent with the definition
   within the WoT Architecture.

   McCool: I understand my implementation with a manual TD still a
   unique code base and an implementation that produces a TD

   Lagally: how stable is the tooling?

   McCool: changes are planned to be backward-compatible
   ... overall the testing plan is still "in flux", as we need
   more information
   ... plan is to stabalize in two weeks for the online TestFest
   ... data files will not change much, assertions will need to
   change

   Lagally: How can I play with this to check the resulting
   document for correctness?

   McCool: adapt data files. script is supposed to pull them in
   automatically at some point.

   Lagally: it is confusing that there are multiple testing
   directories

   Matthias: note that w3c/wot" repo is IG!

   McCool: IG material is general planning. Deliverable repos have
   concrete test documents for the REC process.

   Lagally: is it a valid assumption that w3c/wot is not required
   for REC process?

   McCool: yes
   ... using CSV for test results as they are easy to edit and
   render nicely on GitHub
   ... CSVs can be broken down in multiple files, as each entry
   uses an ID
   ... tool will colorcode the results
   ... red is critical, e.g., only one implementation for security
   ... prefix your test results with your member identifier (e.g.,
   "intel-")

   Ege: having reports per assertions will require to change the
   TD Playground, so that each assertion is in its own JSON Schema
   ... Playground can only check syntax

   Matthias: statements during today's chairs call said that tests
   must focus on behavior, e.g., if a parser must ignore unknown
   terms or must throw an error
   ... need to clarify what kind of assertions are actually
   required from us

   Kaz: I don't think we confirmed that during the chairs call
   today.
   ... the [10]W3C process document requires us to show evidence
   for implementability

     [10] https://www.w3.org/2018/Process-20180201/#implementation-experience

   McCool: there is a file "extra-asserts.html" to file
   new/changes assertions
   ... @all, please have a look at your implementations in the
   context of testing and report to us what you have
   ... need to think about interop and penetration testing and how
   it fits into the report
   ... (something about two-stage process for the test plan impl)

   Kaz: As I (repeatedly) mentioned, CR exit criteria defined by
   the W3C Process is showing that a specification is sufficiently
   clear, complete, and relevant.
   ... On the other hand, I can understand people are interested
   in testing interoperability between various implementations
   itself and describing concrete behavior of each implementation
   (e.g., TD consumer/TD producer).
   ... That's why I'm planning to talk with W3M to confirm what is
   required from us

   McCool: charter states security testing
   ... please clarify what we need to provide

   Ege: isn't interoperability shown when both client and server
   fulfills the assertions?

   McCool: interoperability would go as report into appendix
   ... AoB?

   Toru: Could you please check my PR?

   <inserted> [11]wot PR 596

     [11] https://github.com/w3c/wot/pull/596

   McCool: merging
   ... EoM

Summary of Action Items

Summary of Resolutions

   [End of minutes]
     __________________________________________________________


    Minutes manually created (not a transcript), formatted by
    David Booth's [12]scribe.perl version 1.154 ([13]CVS log)
    $Date: 2018/11/21 18:26:02 $

     [12] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
     [13] http://dev.w3.org/cvsweb/2002/scribe/

Received on Wednesday, 21 November 2018 18:28:26 UTC