[minutes] 20100629 HTML Test Suite Task Force

Available at
  http://www.w3.org/2010/06/29-htmlt-minutes.html

                      HTML Test Suite Task Force

29 Jun 2010

   [2]Agenda

      [2] http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jun/0047.html

Attendees

   Present
          jgraham, krisk, paulc, gsnedders

   Regrets

   Chair
          Kris

   Scribe
          plh

Contents

     * [3]Topics
         1. [4]Check for any bugs on approved tests (currently zero)
         2. [5]Review Current Tests Posted To List For Approval
         3. [6]Ask for feedback on the updates to domtestcase.js
         4. [7]Proposal to change my tracking excel spreadsheet
     * [8]Summary of Action Items
     _________________________________________________________

   oops

   didn't watch the time sorry

   <krisk> that is fine

   <krisk> can't seem to get trackbot and rssagent up

   trackbot-ng, start telcon

   <krisk> jgraham are you going to participate on IRC or dial in?

   <jgraham> I am on IRC

   <scribe> scribeNick: plh

   <krisk> great

Check for any bugs on approved tests (currently zero)

   Kris: none

Review Current Tests Posted To List For Approval

   Kris: first one was video example

   <krisk> I updated the UA sting to canPlayType and added an image of
   the rendering

   [9]http://test.w3.org/html/tests/submission/Microsoft/video/video.ht
   m

      [9] http://test.w3.org/html/tests/submission/Microsoft/video/video.htm

   <krisk> james do you have other feedback?

   s/all time clocks at zero/all time cloks at zero/

   Resolved: video.htm approved

   <scribe> ACTION: Kris to move video.htm into the approved test
   directory

   Kris: next time we meet I'd like to have video_00[1-13]

   <krisk> Let's plan on asking for the rest of video, canvas and
   xhtml5 tests

   plh: is there a reason why we can't approve them today?

   <krisk> How about we state we will post all the video tests as
   meeting notes

   <krisk> If we don't hear negative feedback we can approve all the
   video tests

   <krisk> Which should also include audio0->3

   <krisk>
   [10]http://test.w3.org/html/tests/submission/Microsoft/audio/

     [10] http://test.w3.org/html/tests/submission/Microsoft/audio/

   <krisk> jgraham do you agree that this is OK?

   Resolved: all video* and audio* are approved.

   <jgraham> I haven't looked closely

   <jgraham> sorry

   <krisk> simon peter looked at the tests

   James? do you want more time to look at the tests?

   <jgraham> I think simon is better qualified than me to review video
   tests

   <krisk> He looked at the tests

   <krisk> the one test that will need to be updated is #6

   <krisk> see
   [11]http://lists.w3.org/Archives/Public/public-html-testsuite/2010Ju
   n/0035.html

     [11] http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jun/0035.html

   sounds like we're ok then

   <scribe> ACTION: Kris to update video #6 before moving it

   XHTML5 and Canvas tests have been submitted

   Kris will look at the canvas tests

   <krisk> We also have getElementsByClassName, selection api

   <krisk> Anne's feedback was that we need to update one test to the
   actual parser

   plh: let's approve the foreign tests except for foreign_content_007
   then

   <krisk> that woud be #7
   [12]http://lists.w3.org/Archives/Public/public-html-testsuite/2010Ju
   n/0017.html

     [12] http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jun/0017.html

   Resolved: foreign_content[1-6] and [8-15] are approved

   <krisk> jgraham are you able to look at canvas tests

   <krisk> as a group it's good to get the submitted test backlog under
   control

   <jgraham> We already use many of them, so I guess someone already
   reviewed them. I guess I can get them to look for changes or
   something

   <krisk> that would be good

   <krisk> I think a reasonable approach it to set a date that we can
   all have reviewed the tests

   <krisk> Canvas that is since their is ~800 total tests

   <krisk> we should get agreement on how we want some of the api's to
   be organized

   <krisk> for example we could just cover the document

   <krisk> and a document folder would contain all the api tests for
   document, for example document.getElementByClassName_001.htm

   <krisk> Same goes for the window object

   <jgraham> Yes, I think organisation by chapter would be good

   <jgraham> (also, the annotated spec that Philip made for the canvas
   tests is very nice, we should make something similar)

   [13]http://test.w3.org/html/tests/approved/

     [13] http://test.w3.org/html/tests/approved/

   <krisk> I was going to call it be the name of the html5 feature
   rather than the chapter

   <jgraham> Hopefully name of feature and chapter are quite similar

   <jgraham> I guess there might be edge cases though

   <krisk> I think once we think the chapters (numbers) are not going
   to change we can rename them to include the chapter

   <jgraham> Using chapter-title based rather than chapter-number based
   naming should be forwards compatible

   <krisk> The way I see it is that at some point we create chapters
   and move tests into that folder

   <krisk> OK

   <krisk> It's seems to be easy to move tests around - e.g. hg move
   seems to work just fine and won't lose any data

   <krisk> Let's move on to agend item #3

Ask for feedback on the updates to domtestcase.js

   <krisk> jgraham did you have anymore updates planned for
   domtestcase.js?

   <jgraham> krisk: Yes.

   <jgraham> I just haven't had time to make them yet...

   <krisk> I updated the harness - removed the .name since it's not a
   standard

   <jgraham> Basically some assert functions that check common IDL
   things

   <jgraham> like assertReadonly

   <jgraham> and some more things that I have forgotten right now

   <krisk> do we need to do them before we agree to have this approved
   to be what we want to use for async tests

   <krisk> the reason I say this is adding a readonly assert should be
   able to be done later, since it's just a minor add

   <jgraham> Yeah I think we can more assertions later

   <krisk> ok - since this does help test async stuff like events

   <krisk> I can move the script and the sample test then to the
   approved folder

   <krisk> unless you object james

   <jgraham> No, I think that is fine as long as we can still make
   changes to it

   Resolved: update domtestcase.js with latest dev version

   <krisk> The only issue would be if we want to make some disruptive
   change that breaks a bunch of tests that is used by domtestcase.js

   <krisk> Then this breaking change will need to also update the
   impacted tests

   <krisk> sound fair?

   <jgraham> Yes

   <jgraham> If people are happy with the basic API

   <jgraham> I'm not a good person to judge for obvious reasons

   <krisk> I can take an action item to update the wiki with these
   tests as examples that can cover sync, async, manual (ugh like
   audio)

   <scribe> ACTION: Kris to update the wiki with test information

Proposal to change my tracking excel spreadsheet

   <krisk> Yes though we need more input from the group

   <krisk> Intresting after the meeting Philip Taylor posted results to
   the list for canvas tests

   <krisk> Seems like we can do the same for approved

   <krisk> If we think it will help make the spec get to rec and make
   HTML5 more interoperable

   <jgraham> Yes, it would be great to have

   <krisk> see
   [14]http://lists.w3.org/Archives/Public/public-html-testsuite/2010Ju
   n/0036.html

     [14] http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jun/0036.html

   <krisk> I think we could do this...

   <krisk> We could have browser vendors submit their results in xml
   and then have a page load up and transform these results into a
   result page

   <krisk> I can do this once, though the results moving forward would
   have to come from the vendor

   <jgraham> Yeah, something like that would be good although I would
   choose different specific technologies

   <jgraham> :)

   <krisk> It's in the open - so sure a vendor could lie...but that
   would be quite embarassing of the errors were intentional and not a
   error

   <jgraham> Well it is easy to verify any results

   <gsnedders> Provided they're public builds, it's easy to check

   <jgraham> It doesn't even have to be vendors that submit things
   really since only public builds should be eligible

   <krisk> It could be just submitted to Hg - then it's tracked and
   will come from a w3c member

   <jgraham> Yeah, that sounds like a workable model given some
   convention for where to submit results files

   <krisk> we'll need to not overload the w3c team with this idea

   <gsnedders> See what the webapps WG did for widgets

   <gsnedders> [15]http://dev.w3.org/2006/waf/widgets/imp-report/

     [15] http://dev.w3.org/2006/waf/widgets/imp-report/

   <gsnedders> Just XML files of testfile:testname -> result would do

   <krisk> sounds like then opera and microsoft support this idea

   <jgraham> Personally I would just use json, since it is basically
   key-value pairs and xml is overkill

   the format isn't critical. xml or json is fine

   <jgraham> and I wouldn't format it like the widgets people...

   we could accept more than one in fact

   <jgraham> The bigger question is how we make a unique id for a test

   <jgraham> If we want it to be invariant under directory
   reorganizations and so on

   ah, good point

   <jgraham> But maybe that is a non-issue and a path is good enough?

   <krisk> I'd rather just stick to using the path

   <paulc_> I agree that we might need to handle test suite versions

   <jgraham> Regenerating the results is cheap, coming up with complex
   solutions is hard

   <paulc_> As new tests get added or tests get removed, we need to
   make sure that the results format is robust enough to work.

   <plh> we could agree to use the path to start with and see how it flies

   <plh> in terms of formats, as long as I can integrate whatever I get, I
   don't mind too much

   <krisk> Seems like we agree that this is the right direction to head
   and just have to work out some of the details

   <plh> it's a matter of transformation for me

   <jgraham> Something like {"tests":[{"id":"/path/to/test",
   "result":"pass"}]} seems simple and extensible

   and if start getting too many updates, i'll look into some automatic
   ways

   the test results needs to report the user agent version

   <krisk> some of the details would be...

   since it may change overtime

   <krisk> must include the useragent and build

   and OS ?

   one vendor might want to report several test results depending on
   the device/platform

   [adjourned]

Summary of Action Items

   [NEW] ACTION: Kris to move approved tests into the approved test
   directory
   [NEW] ACTION: Kris to update the wiki with test information
   [NEW] ACTION: Kris to update video #6 before moving it
   [NEW] ACTION: Kris to update update domtestcase.js with latest dev
   version

Received on Tuesday, 29 June 2010 21:45:37 UTC