W3C home > Mailing lists > Public > public-test-infra@w3.org > January to March 2011

[minutes] Call 2011-02-28

From: Francois Daoust <fd@w3.org>
Date: Mon, 28 Feb 2011 15:58:59 +0100
Message-ID: <4D6BB833.8000505@w3.org>
To: public-test-infra@w3.org
Minutes of weekly call available at:
  http://www.w3.org/2011/02/28-testing-minutes.html
... and copied as raw text below.

Agreed:
-  Wiki pages under /Testing:
  http://www.w3.org/wiki/Testing

Actions:
- Mike to classify requirements in the wiki:
  http://www.w3.org/wiki/Testing/Requirements
- Francois to have a look at how Mozilla tests Firefox  and Firefox Mobile

Next meeting:
  Monday 7 March, 13:30 UTC.

Francois.

-----
Testing team call

28 Feb 2011

    See also: [2]IRC log

       [2] http://www.w3.org/2011/02/28-testing-irc

Attendees

    Present
           plh, francois, mike

    Regrets
    Chair
           plh

    Scribe
           francois

Contents

      * [3]Topics
          1. [4]Wiki
          2. [5]Annotations
          3. [6]Timeline
      * [7]Summary of Action Items
      _________________________________________________________

Wiki

    -> [8]Test Infrastructure

       [8] http://www.w3.org/wiki/TestInfra/goals

    mike: I took material written more than one year and half ago when
    we had some meeting and put together some requirements.
    ... Linked from that page are some requirements from other groups.
    ... Goal is to get all the requirements in one place so that we know
    what we want to do

    plh: I had a call on Thursday with accessibility people. We should
    break down the page into "what the harness should do", "how to
    submit the tests", etc.
    ... Not all the WG will submit tests the same way
    ... First priority: what the harness should do.

    <MikeSmith> [9]http://www.w3.org/wiki/Testing

       [9] http://www.w3.org/wiki/Testing

    <MikeSmith> [10]http://www.w3.org/wiki/Testing/Requirements

      [10] http://www.w3.org/wiki/Testing/Requirements

    mike: doug set up another page
    ... because he did not like the name. I don't really care about the
    name

    plh: I don't care either way. Let's take a decision. Do you care?

    [attendees agree to flip a coin]

    plh: ok, please proceed under /Testing, then, Mike.
    ... could you start classifying the topics in the page?

    mike: yes, easy enough to do, happy to do this this week.

Annotations

    <plh> ACTION: Mike to classify the requirements in the wiki
    [recorded in
    [11]http://www.w3.org/2011/02/28-testing-minutes.html#action01]

    <plh> Mike?

    <plh> can't hear you?

    <MikeSmith> the fan on my laptop has quit, so my laptop is
    overheating now and then, to the point where it hits 85 degrees or
    something and then shuts itself down

    <MikeSmith> I have a new laptop on the way but I won't get until for
    at least one more week

    <plh> can you rejoin the call or should we continue on irc?

    mike: let me paste in the URL for annotations

    <MikeSmith>
    [12]http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canv
    as/

      [12] http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/

    mike: here's the system that Philipp Taylor's set up for canvas.
    ... not scalable to the whole spec as it does regular expression
    checks specific to this section.
    ... what we could do is to scope the tests with the IDs of a
    specific section and then we can have it scale for finer-grained
    addressing.
    ... If it works out for HTML5, it should work for everything else,
    since there's hardly anything bigger than the HTML5 spec.

    <MikeSmith>
    [13]http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-
    spec/canvas.html

      [13] http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html

    <MikeSmith>
    [14]http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-
    spec/canvas.html#testrefs.initial.colour

      [14] http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html#testrefs.initial.colour

    <MikeSmith>
    [15]http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/ini
    tial.colour.html

      [15] http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/initial.colour.html

    mike: you'll see annotations in the result specifications, attached
    to testable assertions.

    plh: I thought this was hard to replicate on a larger specification?

    mike: not really. The problem is if we don't have any way to scope a
    test case to a specific section of the spec. If we do, then this
    thing can scale.
    ... the point is that you have at the beginning of the test case the
    id of the section in the spec, and then the regular expression would
    apply to this section only.
    ... If we do it that way, it would scale.

    plh: but the specification is a moving target!

    mike: when the prose changes, we need to come up with an error
    message that says "I cannot find this assertion anymore".

    plh: so that's what you're working on?

    mike: yes.

    <MikeSmith>
    [16]http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canv
    as/tests2dtext.yaml

      [16] http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/tests2dtext.yaml

    francois: in short, I'd introduce a level of indirection with a
    mapping table between testable assertions, extracted by a regular
    expression, and an ID, that tests can then reference.

    mike: it seems doable and is not far away from what I've been
    contemplating do.
    ... I need to take a closer look at that.
    ... I should be able to work on that later this week.

Timeline

    plh: in terms of timeline, I'd like us to be done with requirements
    by the end of March.
    ... I'm hoping that Shadi and Michael are going to be able to show
    us what they need in the coming week.
    ... Once done with requirements, April to June should be spent
    implementing the bouzin.
    ... We may need some things running in May, for instance for
    Michael.
    ... At the minimum, we should try not to change the format of the
    tests.
    ... Because of the parsing tests, we'll need some way to have tests
    completely separated from metadata about the tests.

    francois: will we need to setup some hybrid system? I'm all for
    keeping things separated, but some people may think otherwise.

    plh: Right. About parsing tests, we may need tests that are exactly
    1 byte long, and we cannot include metadata in there.

    mike: yes, I think we already discussed and agreed about the need to
    have a manifest file that describes the tests.

    plh: what I would suggest for this metadata information is to look
    at what the CSS working group did, because they are pretty advanced
    there.

    mike: ok.

    plh: After June, once we are done with setting up the framework, we
    should focus on integrating tests in there, such as HTML parsing
    tests.
    ... tests done outside of W3C, for instance.

    mike: getting the parsing tests in is definitely an important goal.

    plh: yes, if we manage to get parsing tests in, the rest should be
    relatively easy to integrate.
    ... Some tests may apply to places where JavaScript is not available
    and may be tricky, but the rest should be fine.
    ... On the framework front, we need to identify all the frameworks
    that are out there. I believe that the Wiki page already has such a
    list.
    ... Someone needs to look at what Mozilla is doing and understand
    how Mozilla is testing their implementations.
    ... They are probably the most advanced ones out there.

    francois: happy to have a look

    <MikeSmith> Ms2ger

    <scribe> ACTION: francois to see how Mozilla is testing Firefox
    [recorded in
    [17]http://www.w3.org/2011/02/28-testing-minutes.html#action02]

    mike: you may get in touch with Ms2ger, as he is a key contributor
    to testing efforts.

    plh: I think we should still have a direct look at it. Francois,
    would be good to spend time and see e.g. the test format, how they
    test their mobile implementation, etc.

    <MikeSmith> Mark Finkle

    mike: the guy to get in touch with to have pointers to the right
    things is Mark Finkle.

    plh: is there an IRC channel for testing?

    <MikeSmith> irc.mozilla.org, ask on #developers

    mike: don't know, but you should be able to find the info on
    Mozilla's main developer irc channel.

    plh: don't know how long it would take, probably two weeks. I think
    that's valuable. We need it before end of March.

    francois: ok, will do asap.

    plh: anything else on the testing framework?
    ... I'll ping Michael and Shadi to make sure they get their
    requirements right.

    mike: next meeting?

    plh: how about this day and time?

    [should work]

Summary of Action Items

    [NEW] ACTION: francois to see how Mozilla is testing Firefox
    [recorded in
    [18]http://www.w3.org/2011/02/28-testing-minutes.html#action02]
    [NEW] ACTION: Mike to classify the requirements in the wiki
    [recorded in
    [19]http://www.w3.org/2011/02/28-testing-minutes.html#action01]

    [End of minutes]
Received on Monday, 28 February 2011 14:59:30 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 28 February 2011 14:59:30 GMT