- From: Deborah Dahl <dahl@conversational-technologies.com>
- Date: Fri, 29 Jan 2010 13:59:30 -0500
- To: <public-hypertext-cg@w3.org>
These minutes are being posted in the public list to record
the HCG discussion of testing in the browser (see topic 3).
- DRAFT -
Hypertext Coordination Group Teleconference
15 Jan 2010
Attendees
Present
Debbie_Dahl, Kazuyuki, Plh, Steven, +95089aaaa, glazou, Bert,
Shepazu, ChrisL
Regrets
Chair
ddahl
Scribe
shepazu
Contents
* [4]Topics
1. [5]Actions
2. [6]HCG time
3. [7]Testing in the Browser
* [8]Summary of Action Items
_________________________________________________________
<trackbot> Date: 15 January 2010
<Steven_> Hi
<Steven_> Muted
<scribe> scribenick: shepazu
Actions
<ChrisL> action-16?
<trackbot> ACTION-16 -- Doug Schepers to coordinate with Mike and
SVG-in-HTML (Issue-37), keep HCG informed -- due 2009-10-31 -- OPEN
<trackbot> [9]http://www.w3.org/MarkUp/CoordGroup/track/actions/16
[9] http://www.w3.org/MarkUp/CoordGroup/track/actions/16
<ChrisL> close action-16
<trackbot> ACTION-16 Coordinate with Mike and SVG-in-HTML
(Issue-37), keep HCG informed closed
shepazu: this is ongoing, probably doesn't need an action
<ChrisL> action-28?
<trackbot> ACTION-28 -- Philippe Le Hégaret to write binding-license
advice for /Guide -- due 2009-12-31 -- OPEN
<trackbot> [10]http://www.w3.org/MarkUp/CoordGroup/track/actions/28
[10] http://www.w3.org/MarkUp/CoordGroup/track/actions/28
plh: can we extend that by a month?
... making progress on the document license
<ChrisL> action-30?
<trackbot> ACTION-30 -- Doug Schepers to report on SVG WG's progress
on test framework -- due 2009-09-11 -- OPEN
<trackbot> [11]http://www.w3.org/MarkUp/CoordGroup/track/actions/30
[11] http://www.w3.org/MarkUp/CoordGroup/track/actions/30
<ChrisL> close action-30
<trackbot> ACTION-30 Report on SVG WG's progress on test framework
closed
shepazu: MS joined SVG WG, and we are suggesting submitting tests in
reftest
<ChrisL> action-32?
<trackbot> ACTION-32 -- Deborah Dahl to follow up on scxml
implementations from KDE -- due 2009-09-30 -- OPEN
<trackbot> [12]http://www.w3.org/MarkUp/CoordGroup/track/actions/32
[12] http://www.w3.org/MarkUp/CoordGroup/track/actions/32
<ChrisL> action-41?
<trackbot> ACTION-41 -- Philippe Le Hégaret to and Chris to work on
a proposed charter for the Group -- due 2009-11-27 -- OPEN
<trackbot> [13]http://www.w3.org/MarkUp/CoordGroup/track/actions/41
[13] http://www.w3.org/MarkUp/CoordGroup/track/actions/41
ChrisL: we decided to do an extension for now
... that's just an email to the AC
plh: yes, but it has to go through W3M first
ddahl: how long is the extension?
plh: 2 years
shepazu: so, we aren't going to make the group public?
plh: we are going to try to be more careful about being publicly
accessible
ddahl: this week's minutes are going to be public
HCG time
<Steven_> public-hypertext-cg@w3.org
ddahl: ArtB has a problem with this time
... he suggested an hour earlier, which is inconvenient for many
west-coast people
<Steven_> and japan
ChrisL: maybe a survey?
<Steven_> This hour is the only one that is anyway acceptible for
west coast and Japan
ChrisL: new person representing ITU based in Japan
<scribe> ACTION: ChrisL to create telcon time WBS survey [recorded
in [14]http://www.w3.org/2010/01/15-hcg-minutes.html#action01]
<trackbot> Created ACTION-42 - Create telcon time WBS survey [on
Chris Lilley - due 2010-01-22].
Testing in the Browser
ChrisL: topic's been around for a while, could mean 2 things
1) browser used as test platform, such as through a harness,
one-by-one, noting results,
scribe: takes an hour or more to collate results
<ChrisL> One aspect is testing by using the browser as a platform,
like the css3.nu tests for css selectors. The browser runs the whole
suite (or some subset that needs no interaction) itself, using
scripting, then displays the result.
ChrisL: alternately, the browser could do the testing
<ChrisL> [15]http://www.css3.info/selectors-test/
[15] http://www.css3.info/selectors-test/
ChrisL: I did the CSS Selectors test suite by hand, and it took me
1.5 hours per browser
... used the automatic harness, and it took only minutes
... you still need to capture the results, and we need to talk about
how to do that
... 2) the other aspect...
<ChrisL> Another aspect is testing things that are needed in
practice but fall between specifications. For example, no-one would
use a browser that can't center a JPEG image in an HTML page. But
HTML says as little as possible about rendering and has no required
image formats. CSS tries to not refer to specific markup languages.
So it all falls between the cracks.
<ChrisL> In some cases this is dealt with by making wrapper
specifications that include a bunch of other specs by reference and
then add extra conformance criteria. The desirability of that
approach can be discussed.
<glazou> yeah
<ChrisL> hmm i was dropped, will redial
ddahl: some of the links focused on testing i general, managing
large test suites... I don't think we need to cover that
plh: my experience is that browser vendors are interested in
submitting tests, but not in adding information or running tests
ChrisL: they do seem reluctant to run them, which is why I've been
doing it to move the specs forward
... but they do seem interested in automatic testing
Steven_: even though the browser vendors haven't shown interest, why
are we doing this? it's for W3C process and progress on the Rec
Track
ChrisL: there are other people who could run tests: staff, members
of the public
<Steven_> We have to test in order to get out of CR
ChrisL: we need to clarify the process for doing so, and the public
could help more
plh: it's a resources problem
<ChrisL> especially if the collection and collation of results is
automated
plh: the browser vendors, to be blunt, aren't interested in W3C
being able to evaluate their browsers
... I haven't been able to get funding
ChrisL: we need to give them the framework first, then ask to help
improve it
shepazu: it may be that they are reluctant to give resources, but
might agree with the goal
... seems to me we could apply for a grant from the Web Foundation,
testing is a fundamental part of making the Web better and more
accessible and i18ned
plh: the deadliine for contributing to the CSS2.1 spec was today,
how do you plan to manage this?
glazou: good question, we have many thousands of tests, need to
discuss this
... should happen next week
plh: any suggestion how we could generate such a report?
glazou: CSS may be easier than others, we can use CSS DOM to get
values
... handling thousands of protocol tests... whew, that's another
matter
... CSS often needs the rendering test done by hand
... we can sometimes use canvas (not on IE), many have reference
images
... one part can often be tested automated, but another needs to be
tested by humans
plh: would making a public announcement calling for aid help?
glazou: maybe, but would it be in a reasonable timeframe
ChrisL: the instructions need to be clear, and a good test makes
that easier ("if you see red, the test fails")
glazou: this depends on what you're testing... animations and
transitions
<ChrisL> svg has animation tests so the end result has things lined
up or covering each other if the animation is correct
glazou: for CSS, we could easily write a framework to test 99% of
tests in a browser with Canvas, without it, forget it
shepazu: what about RefTest? we have good results in automation at
the Test Summit
ChrisL: 99% really helps
glazou: don't forget we need reference images for all those
<ChrisL> especially when there are thousands of tests
shepazu: there are a lot of challenges with maintaining reference
images, especially over versions
... reftest may be better
<ChrisL> reftests avoid variations due to font rendering or subpixel
positioning
ChrisL: what's the next step? how do we move forward?
... you seem a bit pessimistic
plh: it's just going to require a significant level of effort, not
sure where we are going to find funding and resources
ddahl: is it a general problem, or one for each spec/group?
<ChrisL> ach chris
plh: it's general... we have good results in all browsers, even IE,
at the test summit
glazou: even IE?
plh: Sylvain Pashe had a clever way of automating the browsers
glazou: that comparison can't be done in the browser then
shepazu: I don't think that's a problem
glazou: it is for mobile browsers
shepazu: touche
ChrisL: if you are just need 2-4 results for CR implementation
reports, that suffices, but for completeness, this approach won't
work
glazou: there are big differences between mobile and desktop
versions of the same browser
shepazu: question is, are we doing tests to help specs pass CR, or
to improve interop by testing all browsers on all platforms?
ChrisL: both, but CR testing is primary
<scribe> ACTION: shepazu to start a wiki page to summarize the 2
goals, with Use cases and requirements for each goal [recorded in
[16]http://www.w3.org/2010/01/15-hcg-minutes.html#action02]
<trackbot> Created ACTION-43 - Start a wiki page to summarize the 2
goals, with Use cases and requirements for each goal [on Doug
Schepers - due 2010-01-22].
<scribe> ACTION: look into funding from Web Foundation and NIH
[recorded in
[17]http://www.w3.org/2010/01/15-hcg-minutes.html#action03]
<trackbot> Sorry, couldn't find user - look
<scribe> ACTION: plh to look into funding for browser testing from
Web Foundation and NIST [recorded in
[18]http://www.w3.org/2010/01/15-hcg-minutes.html#action04]
<trackbot> Created ACTION-44 - Look into funding for browser testing
from Web Foundation and NIST [on Philippe Le Hégaret - due
2010-01-22].
glazou: when we design a test suite, we should have a manifest that
describes the means of testing for each test: reference image,
reftest, human
ChrisL: and the results need to glued together
<ChrisL> We need to discuss collating multiple test results, but
there is no time
shepazu: we talked about that at the Test Summit
trackbot, end telcon
Summary of Action Items
[NEW] ACTION: ChrisL to create telcon time WBS survey [recorded in
[19]http://www.w3.org/2010/01/15-hcg-minutes.html#action01]
[NEW] ACTION: look into funding from Web Foundation and NIH
[recorded in
[20]http://www.w3.org/2010/01/15-hcg-minutes.html#action03]
[NEW] ACTION: plh to look into funding for browser testing from Web
Foundation and NIST [recorded in
[21]http://www.w3.org/2010/01/15-hcg-minutes.html#action04]
[NEW] ACTION: shepazu to start a wiki page to summarize the 2 goals,
with Use cases and requirements for each goal [recorded in
[22]http://www.w3.org/2010/01/15-hcg-minutes.html#action02]
[End of minutes]
_________________________________________________________
Received on Friday, 29 January 2010 19:00:33 UTC