See also: IRC log
RESOLUTION: minutes approved
RESOLUTION: The working group agrees to endorse this bug.
<johnarwe_> http://lists.w3.org/Archives/Public/public-sml/2008Apr/att-0069/sml-interop-test-plan.pdf
<scribe> current location: page 3 section 5
<scribe> MSM: does not believe that test results should be only boolean valid/invalid
<scribe> ... e.g. should include errors/messages
<scribe> Ginny: you're talking about manual comparison in this case
<scribe> Pratul: in previous interop workshop, the implementations provided diagnostics and were manually compared
<scribe> Ginny: agree that this information would be useful even if not automate-able
<scribe> Sandy: 2 kinds of validity results - model and SML-IF document
<scribe> Sandy: SML-IF format is the spec and should be validated
<scribe> Kumar: where does SML-IF validation come in?
Kumar: 2 kinds of test cases - sml and sml-if
MSM: suggest discussing this in the future: what the testable assertions are in sml and sml-if
<johnarwe_> we revised 3-4 last week with much discussion
<johnarwe_> MSM text below was pasted from process doc
<MSM> Entrance criteria: The Director calls for review when satisfied that the Working Group has:
<MSM> Fulfilled the general requirements for advancement;
<MSM> Shown that each feature of the technical report has been implemented. Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature. If the Director believes that immediate Advisory Committee review is critical to the success of a technical report, the Director may accept to Call for Review of a Proposed Recommendation even without adequate implementation experience;
<MSM> Satisfied any other announced entrance criteria (e.g., any included in the request to advance to Candidate Recommendation, or announced at Last Call if the Working Group does not intend to issue a Call for Implementations).
<scribe> discussion of # of implementations for optional features
<scribe> moving to section 6
<johnarwe_> perhaps add "...in order to be a complete impl"?
<scribe> MSM: can we use partial implementations?
<scribe> Pratul: suggests taking line 9 out.
<scribe> MSM: can have a partial implementation that implements a feature as contributing toward the "2 implementations" test.
<scribe> The sentence under discussion is "Each participating implementation must pass all tests in this section."
<MSM> Possible new wording:
<MSM> 1. Tests for required features:
<MSM> These features are "required" in the sense that conforming processors
<MSM> must support them. If any partial implementations participate in the
<MSM> testing effort, those partial implementations may not pass all of
<MSM> these tests.
<scribe> RESOLUTION: MSM changes above are accepted by the group
<scribe> discussing section 6, line 14
<scribe> MSM: should think about requiring specific behavior when an implementation does not support an optional feature. ignore? error?
<scribe> ... this is probably a spec change.
<johnarwe_> "spec" change is just words... the interesting question is, would such changes be -substantive-
<scribe> Ginny: preference is to use IF format for tests
<scribe> MSM: also consider embedding metadata if desired
<scribe> Pratul: anyone disagree with using IF format?
<scribe> RESOLUTION: agree to use IF format for test cases rather than defining another test case format
<scribe> discussion of Kirk's comments around interoperability - interoperability of models or implementations
<scribe> MSM: suggests defining a notion of interoperability as a triple of model, producer, and consumer
Pratul: in previous interop, did round-trip testing between 2 implementations
<scribe> John: need to define equivalence of the models, since simple equality of SML-IF file after a round trip does not take into account URI re-writing etc. that may occur.
MSM: just a heads up that W3C is looking at definition of interoperability but we should proceed to define this for our purposes
<johnarwe_> just for context in minutes: this discussion is based on http://lists.w3.org/Archives/Public/public-sml/2008Apr/0087.html
<scribe> ACTION: MSM to define "interoperability" for SML and SML-IF testing purposes [recorded in http://www.w3.org/2008/09/18-sml-minutes.html#action01]
<trackbot> Created ACTION-201 - Define \"interoperability\" for SML and SML-IF testing purposes [on Michael Sperberg-McQueen - due 2008-09-25].
moving to John's comments in second draft http://lists.w3.org/Archives/Public/public-sml/2008May/att-0017/sml-interop-test-plan20080504.doc
<johnarwe_> ok pdf is in the pipe
<johnarwe_> to public-sml and direct cc to msm
Discussion of John's comments in the document
Section 1
Ginny and MSM agree with comments
Section 2
Kumar has some reservations on option 3 (and also option 1 but we've agreed not to use option 1)
Kumar: option 3 requires more work when most test cases are embedded files anyway
John: option 3 does not require that we use locator but gives the option to do the work required if needed
MSM: will need both embedded and locator in the case where we test the locator element
<pratul> I will be representing Microsoft for the rest of the call since Kumar has left the call
Section 3
John: implementation characteristics would be, e.g., features implemented
MSM: xquery does this by saying one possible values of test result is "does not apply"
Ginny: need to consider effort versus value derived
<scribe> John: not discussing final output format at this time
<scribe> Section 6
<scribe> John: in previous interop workshop, they manually reviewed the error output and decided if it was the 'same' or not
Ginny: is comparison of errors in scope for this testing?
<pratul> Manual/human comparison is OK
<scribe> Ginny: Is manual checking part of 'passing the test'?
<scribe> Pratul: this is not a compliance test suite.
<johnarwe_> http://lists.w3.org/Archives/Public/public-sml/2008May/att-0017/sml-interop-test-plan20080504.doc
<johnarwe_> hang on
<johnarwe_> http://lists.w3.org/Archives/Public/public-sml/2008May/0023.html
<johnarwe_> last email in the list :-)
<scribe> ACTION: Kumar to update test case document to incorporate any specific decisions made over the past 2 weeks and to add a list a list of decisions still to be made by the working group [recorded in http://www.w3.org/2008/09/18-sml-minutes.html#action02]
<trackbot> Created ACTION-202 - Update test case document to incorporate any specific decitions made over the past 2 weeks and to add a list a list of decisions still to be made by the working group [on Kumar Pandit - due 2008-09-25].
Last Scribe Date Member Name Regrets pending 2008-05-22 Lynn, James Until further notice 2008-07-10 McCarthy, Julia Until further notice 2008-08-28 Kumar, Pandit 2008-09-04 Gao, Sandy 2008-09-11 Wilson, Kirk 2008-09-18 Smith, Virginia Exempt Arwe, John Exempt Dublish, Pratul Exempt MSM