- From: Francois Daoust <fd@w3.org>
- Date: Mon, 28 Feb 2011 15:58:59 +0100
- To: public-test-infra@w3.org
Minutes of weekly call available at:
http://www.w3.org/2011/02/28-testing-minutes.html
... and copied as raw text below.
Agreed:
- Wiki pages under /Testing:
http://www.w3.org/wiki/Testing
Actions:
- Mike to classify requirements in the wiki:
http://www.w3.org/wiki/Testing/Requirements
- Francois to have a look at how Mozilla tests Firefox and Firefox Mobile
Next meeting:
Monday 7 March, 13:30 UTC.
Francois.
-----
Testing team call
28 Feb 2011
See also: [2]IRC log
[2] http://www.w3.org/2011/02/28-testing-irc
Attendees
Present
plh, francois, mike
Regrets
Chair
plh
Scribe
francois
Contents
* [3]Topics
1. [4]Wiki
2. [5]Annotations
3. [6]Timeline
* [7]Summary of Action Items
_________________________________________________________
Wiki
-> [8]Test Infrastructure
[8] http://www.w3.org/wiki/TestInfra/goals
mike: I took material written more than one year and half ago when
we had some meeting and put together some requirements.
... Linked from that page are some requirements from other groups.
... Goal is to get all the requirements in one place so that we know
what we want to do
plh: I had a call on Thursday with accessibility people. We should
break down the page into "what the harness should do", "how to
submit the tests", etc.
... Not all the WG will submit tests the same way
... First priority: what the harness should do.
<MikeSmith> [9]http://www.w3.org/wiki/Testing
[9] http://www.w3.org/wiki/Testing
<MikeSmith> [10]http://www.w3.org/wiki/Testing/Requirements
[10] http://www.w3.org/wiki/Testing/Requirements
mike: doug set up another page
... because he did not like the name. I don't really care about the
name
plh: I don't care either way. Let's take a decision. Do you care?
[attendees agree to flip a coin]
plh: ok, please proceed under /Testing, then, Mike.
... could you start classifying the topics in the page?
mike: yes, easy enough to do, happy to do this this week.
Annotations
<plh> ACTION: Mike to classify the requirements in the wiki
[recorded in
[11]http://www.w3.org/2011/02/28-testing-minutes.html#action01]
<plh> Mike?
<plh> can't hear you?
<MikeSmith> the fan on my laptop has quit, so my laptop is
overheating now and then, to the point where it hits 85 degrees or
something and then shuts itself down
<MikeSmith> I have a new laptop on the way but I won't get until for
at least one more week
<plh> can you rejoin the call or should we continue on irc?
mike: let me paste in the URL for annotations
<MikeSmith>
[12]http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canv
as/
[12] http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/
mike: here's the system that Philipp Taylor's set up for canvas.
... not scalable to the whole spec as it does regular expression
checks specific to this section.
... what we could do is to scope the tests with the IDs of a
specific section and then we can have it scale for finer-grained
addressing.
... If it works out for HTML5, it should work for everything else,
since there's hardly anything bigger than the HTML5 spec.
<MikeSmith>
[13]http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-
spec/canvas.html
[13] http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html
<MikeSmith>
[14]http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-
spec/canvas.html#testrefs.initial.colour
[14] http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html#testrefs.initial.colour
<MikeSmith>
[15]http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/ini
tial.colour.html
[15] http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/initial.colour.html
mike: you'll see annotations in the result specifications, attached
to testable assertions.
plh: I thought this was hard to replicate on a larger specification?
mike: not really. The problem is if we don't have any way to scope a
test case to a specific section of the spec. If we do, then this
thing can scale.
... the point is that you have at the beginning of the test case the
id of the section in the spec, and then the regular expression would
apply to this section only.
... If we do it that way, it would scale.
plh: but the specification is a moving target!
mike: when the prose changes, we need to come up with an error
message that says "I cannot find this assertion anymore".
plh: so that's what you're working on?
mike: yes.
<MikeSmith>
[16]http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canv
as/tests2dtext.yaml
[16] http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/tests2dtext.yaml
francois: in short, I'd introduce a level of indirection with a
mapping table between testable assertions, extracted by a regular
expression, and an ID, that tests can then reference.
mike: it seems doable and is not far away from what I've been
contemplating do.
... I need to take a closer look at that.
... I should be able to work on that later this week.
Timeline
plh: in terms of timeline, I'd like us to be done with requirements
by the end of March.
... I'm hoping that Shadi and Michael are going to be able to show
us what they need in the coming week.
... Once done with requirements, April to June should be spent
implementing the bouzin.
... We may need some things running in May, for instance for
Michael.
... At the minimum, we should try not to change the format of the
tests.
... Because of the parsing tests, we'll need some way to have tests
completely separated from metadata about the tests.
francois: will we need to setup some hybrid system? I'm all for
keeping things separated, but some people may think otherwise.
plh: Right. About parsing tests, we may need tests that are exactly
1 byte long, and we cannot include metadata in there.
mike: yes, I think we already discussed and agreed about the need to
have a manifest file that describes the tests.
plh: what I would suggest for this metadata information is to look
at what the CSS working group did, because they are pretty advanced
there.
mike: ok.
plh: After June, once we are done with setting up the framework, we
should focus on integrating tests in there, such as HTML parsing
tests.
... tests done outside of W3C, for instance.
mike: getting the parsing tests in is definitely an important goal.
plh: yes, if we manage to get parsing tests in, the rest should be
relatively easy to integrate.
... Some tests may apply to places where JavaScript is not available
and may be tricky, but the rest should be fine.
... On the framework front, we need to identify all the frameworks
that are out there. I believe that the Wiki page already has such a
list.
... Someone needs to look at what Mozilla is doing and understand
how Mozilla is testing their implementations.
... They are probably the most advanced ones out there.
francois: happy to have a look
<MikeSmith> Ms2ger
<scribe> ACTION: francois to see how Mozilla is testing Firefox
[recorded in
[17]http://www.w3.org/2011/02/28-testing-minutes.html#action02]
mike: you may get in touch with Ms2ger, as he is a key contributor
to testing efforts.
plh: I think we should still have a direct look at it. Francois,
would be good to spend time and see e.g. the test format, how they
test their mobile implementation, etc.
<MikeSmith> Mark Finkle
mike: the guy to get in touch with to have pointers to the right
things is Mark Finkle.
plh: is there an IRC channel for testing?
<MikeSmith> irc.mozilla.org, ask on #developers
mike: don't know, but you should be able to find the info on
Mozilla's main developer irc channel.
plh: don't know how long it would take, probably two weeks. I think
that's valuable. We need it before end of March.
francois: ok, will do asap.
plh: anything else on the testing framework?
... I'll ping Michael and Shadi to make sure they get their
requirements right.
mike: next meeting?
plh: how about this day and time?
[should work]
Summary of Action Items
[NEW] ACTION: francois to see how Mozilla is testing Firefox
[recorded in
[18]http://www.w3.org/2011/02/28-testing-minutes.html#action02]
[NEW] ACTION: Mike to classify the requirements in the wiki
[recorded in
[19]http://www.w3.org/2011/02/28-testing-minutes.html#action01]
[End of minutes]
Received on Monday, 28 February 2011 14:59:30 UTC