Raw IRC Log/minutes For Thu AM Boston F2F

Here are the notes Karl (and others) took on IRC during that part of the
meeting:
 Patrick Curran, Sun, is presenting himself.
 s/presenting/introducing/
 Around the table, People introducing themselve.
 we'll try
 8:55 Lofton is starting the meeting.
 Lofton Henderson:
 Olivier Théreaux
 Karl Dubost
 Dominique Hazael-Massieux
 Peter Fawcett
 Mat May
 Paulo Baggia
 Voice browser
 Matt Oschry
 Lynne Rosenthal
 Mark Skall
 Sandra Martinez
 Jim Hallen
 Cathy ???
 Richard ????
 david marston
 Patrick Curran
  John Gunderson
 Daniel Dardailler
 All of our members are here
 Lofton: except for twos
 ... dimitris and kyrill
 ... UAAG will share their test suites experience with us this morning.
 ... I like UAAG and Voice to tell us about their test suites.
 [Lofton showing the outline of a Test Suite presentation (members only)|http://www.w3.org/QA/WG/2003/03/test-desc-skeleton]
 John Gunderson: We'll have a formal presentation without showing what we have done.
 ... our 3 main issues are tools, description languages, EARL tools.
 ... one of our tools is to track dependencies between WG. So it helps to show us what are the comments in other WG.
 LinkTo: Implementation report of the UAAG TS.
 John Gunderson is explaining what is inside the UAAG specification.
 ... Explaining the implementation reports of UAAG.
 http://www.w3.org/WAI/UA/impl-rec/
 ... NR in the table means Not Reported.
 ... We do not have a vocabulary in the test suite to define what's required or not.
 ... We can consider the QA Process is part of the design time of the user agent and not at a later stage with a specification. 
 dom: is it the role of WAI PF to make outreach inside the WGs to have accessibility features in specs.
 John Gunderson: We would like accessibility be part of the design process. 
karl: when you designed the TS? did you identify errors in the HTML spec?
... if so, did you report to the WG?
 John Gunderson: Access Key is an example is of different implementations. The spec is not clear enough.
 ... all of our comments are going to PF, not to the HTML WG.
 ... We will be more than happy to participate to other WGs and Test effort.
 Wendy Chilsom has joined the meeting.
 Daniel Dardailler: Would you generate EARL if you had tools?
 John Gunderson: If we were seeing a group collective benefits, we will move in this direction.
 Description of an individula Test Case.
 Lofton: How do you envision the cooperation between two test suites: for example between HTML and Accessibility. 
 ... HTML WG will certainly not go so far.
 John Gunderson: We will be happy to contribute our test suite to the HTML WG. but it's difficult right now for the WG because they do not specify the behaviour.
 Lofton: Would you see that the future version of HTML refers to UAAG.
 John Gunderson: That's my prospective. 
 ... We had to sastify not only HTML, but other markup languages as well. UAAG is general enough to be help to implement it for every languages.
 Karl Dubost: Is it a solution to put UAAG in the conformance section of the markup Language for Conformant user agents.
 John Gunderson: It's a possible model, but I would likely to have a section on accessibility.
 Dave Marston: What are other models
 John Gunderson: If behaviours are specified in the spec, they should not be only mentionned, but also they should also specify precisely the behaviour of each features.
 *** Switching to Voice Browser people ***
 Matt Arshry
 Matt Oshry
... it took us almost a year to complete the implementation report
comparisons on number of test cases among various groups
 ... we could have written more, but we had to publish our spec. 
 from around 100 to several tens of thousand test cases
 ... we have tried to cover everything.
 ... Some assertions were unable to test.
 to be tested
... we would have liked the test assertions to be developed at the same time as the spec
 Mark Skall: Did you discuss what the form of assertions were?
 Matt Oshry: We just wrote them.
 (plain english, extracted from spec)
 Lynne Rosenthal: Do you maintain a link from the test to the spec?
 Matt Oshry: Yes exactly.
 ... we have create dsubteam to be able to make the tests
 ... because it was huge.
 ... We had to manage data repository, collaboration and workflow.
 We had to manage thing for the assertion data.
  - Spec cross-referencing
 - Ownership
 - Status indicator
 - Conformance level
 - Automatibility
 - Portability
 - Language Dependance
 - Comments
 ... Voice XML has many important tests which are optionnal so we have decided to put them in the test suite,
 ... to verify if it's consistent accross platform.
 (question was from Mark Skall, how to choose what to test? Only optionnal things?)
 ... We have wrote the tests to be sure that internationalization was respected. We abstracted the grammar and as well the example.
 Englis was not considered as the only language.
 Collaboration: Identify leads, Form Subteams, Statistics.
 ... manage people who drop-out, company which resigns, etc.. we had to manage that.
 Status of a Test: New, study, accept, authoring, completed, reviewed, incomplete, rejected.
 Slides will be on the web.
 We had to create our own repository, because it was too much difficult to do it at W3C.
 The test suites had to be used outside from US. and so we installed a toll free number, because of the Voice Browser WG.
 ... Problems we had: Email glut management. many people working on the same test suite.
 ... We had to update tests sometimes.
 ... Test is very USEFUL when start at the begining of the Spec
 s/start/started
 demo and showing test.
 showing "bugzilla"-like test framework
 Dave Marston: notion of cataloging assertions is interesting and seems useful.
 http://studio.tellme.com/vxml2_ir/test_api.html
 http://studio.tellme.com/vxml2_ir/ircgi_api.html
 BREAK
 paolo Baggia speaks
presentation oriented toward subjective testing
 in text-to-speech
 ... doesn't care at all about performances, because it's just an interoperability report
 ... some tests have subjective evaluation. To make this evaluation, you need a panel of experts.
... The degree of expertise needed varies with the various types of test
[quoting a position paper submitted for discsussion in the VB WG]
 link in the agenda
 Paulo said:  some "voting" tool for subjective test evaluation would be useful
-> http://lists.w3.org/Archives/Member/w3c-voice-wg/2003Feb/att-0033/01-SSML_-_Methodological_Contribution.htm
 ... do you think there are the same issues in other technologies?
 Lofton: It's a continuum.
 Peter Fawcett: yes, we had similar problems.
 ... about subjective testing.
 dom: We have similar issues in every specifications depending on the specifity of the tech itself. 
 ... and esp the need of interaction with users
 [Scribe thought: HTML has a lot of subjective testing... for example for Semantic Use of tags... and it's in fact not tested.]
 nor testable
 (that is, automatically testable)
 PB: automation for speech recognition is really hard to achieve
 ... the signal analyzis is usually not flexible enough to be really useful
 DM discussing the modality of subjective testing
 ... (scroging evaluators)
 PB: an evaluation management system would really help solving this issue
 [except if you define a semantic meaning... that's all about the difference between semantic and structure. I think it should be tested in a way.] 
 AI Dom to follow up with VB to get names for QA leads
 AI all visitors to send an overview of their visit the first two days
 Matt's presentation:  http://www.w3.org/QA/2003/03/vxml2-ir.zip
 ------------------------
 Report on out(in)reach 
 lofton: should we summarize our visits?
 ... I have visited WCAG.
 ... I have used the presentation to highlight some of the points we had.
 AI lofton to send QA contact for Karl in the matrix
 ... Ops Guidelines they were very happy about it and they will amend their charter.
 Olivier: two groups have been visited GEO (I18N) and DI
 ... QA Framework is interesting but it's too much far of what they do. 
 ... They will review it
 ... in GEO discussed about strategy of communication and outreach
 ... in DI, they are interested by templates where they can fill and generate for example the conformance section.
 Patrick Curran: Above their head? precise.
 olivier: The topic is not completely related to what they do.
 ... Concerns: "we do not have QA expert at all in our group, how are we supposed to understand that?"
 Patrick Curran: do they think they can do without QA.
 dom, olivier: they are not supposed to produce Spec in the rec track.
 ... using guidelines help finding flaws
 kd: visited 2 WG (foreseen 3, but the XPath TF meeting had some weird modalities)
 ... the other 2 groups are CSS and WSDL
 ... both meetings were both interesting
 ... CSS was interested mostly by the test suite part
 ... I showed them how they were close to conform to SpecGL
 ... just changing a few words can make them comply
 re- DI: they want tools, templates for specs, help for good editing, etc. (more than just guidelines)
 ... the idea is to show that compliance is often a low-hanging fruit
 ... the other part was about testing
 ... I discussed with Tantek about a meta-language for tests
 ... his rationale was that he preferred to create a bunch of tests rather than creating a framework
 ... we'll probably have to find a solution in-between
 ... by defining a data model
 ... without imposing a markup
 ... that was the main point
 ... important since WG don't have much resources
 ... WSDL surprised me!
 ... They already had a look at the 2 documents
 ... they were very interested
 ... [Tim Bolland is the QA Contact assigned in the CSS WG]
 ... in WSDL, they have nominated 2 persons for each review
 ... Chairs and Staff Contact for Ops
 ... and the editors for SpecGL
 ... They also decided to nominate a QA Lead, but didn't actually appoint it
 ... Regarding their work, all the spec is based on extensibility
 ... besides, they are based on SOAP
 ... and SOAP allows extensions to contradict the core spec
 ... It really look like the opposite of what we recommand!
 ... They will write a QA Process document
 ... They'll start to make test cases
 ... about automated testing, they didn't know if the reference should be the spec itself or the schema
 ... they also have decided to insert test assertions in the spec
 LH: went with Karl to CSS
 ... one of the things that Tantek expressed is concerns that with our tools and templates, we'll suppress innovation
 ... my point is that we're only providing helpers
 ... we don't intend to enforce their usage
 ... as long as they come up with good equivalents
 ... that was actually echoed in WCAG
 LR: should that be added in the templates?
 ... that this is not a constraint, just a jump-start
 LH: when talking about our success stories
 ... such as DOM
 ... Tantek pointed that the DOM TS violates several principles
 ... the technology should be isolated from the TS itself
 ... the DOM TS doens't treat all the UA the same way also
 ... besides, the DOM TS contribution process violates the W3C contribution policy
 KD: it was related to JUnit usage
 LH: they have a very good test suite document
 ... 75% overlap with what I wrote with SVG
 ... for TC authoring GL
 ... I was wondering if that was something we should consider
 ... for example: atomicity of tests, ...
 LR: Tim bolland is also writing a process document that I've been reviewing
 dom : visited 3 groups
 WAI EO, VBWG, Xforms
 WAI EO is not producing specs, but they are reviewing other specs, especially WAI specs
 so the presentation was useful
 people asked questions about guidelines, asked how they could help
 didn't ask for a QA contact, mostly informative
 VBWG (with lofton) : were interested...
 were not sure that what we're doing would be useful (asked them to read and find out)
 didn't have to convince them of the usefulness of QA
 QA contact : need to follow-up
 ... had some interest in testing (e.g subjective testing, as Paolo just presented)
 WCAG also liked the LC forms (yay!)
 [thought: we should compare size of the WG with the progress of QA (bigger groups are more positive)
*       dom second sthat
*       olivier too
 dom : Xforms
 ... mildly interested, hadn't read the document
 staff contact agreed to be QA contact (T.Michel)
 wanted framework for reporting tests
 (a job for TTF?)
 clap clap for Patrick Curran who has put together the slides for the QA outline
 QA outreach KIt
 it has been useful for all the people.
 Olivier to add it to the Library
 Lynne: give an overview of the Month in QA as a useful document.
 We have a tech Review planned in QA.
 and we have a plan to visit domain call.
 to explain QA or help people when they need us.
 olivier: visiting is good because it usually helps to have a better contact.
 Daniel: would it be better to have two more days of outreach more than meetings.
 olivier: The two previous have been more useful than all emails.
 Daniel: so should we have a F2F meeting during the TP
 ... and use the two last days for outreach for QA and continue that we have done the two previous days.
 [Discussion about the way to approach the WGs]
 lofton: inreach and outreach were very valuable.

Hope this helps,

Dom
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/ERCIM
mailto:dom@w3.org