RE: HTML Testing Task Force Conf Call Agenda 1/25/2011

Meeting Notes

* Removed approved mathML test case (use of innerHTML) bugzilla bug #11825
* Agreement to add setup/tear down to test harness (http://lists.w3.org/Archives/Public/public-html-testsuite/2010Dec/0012.html)
* Discussion on server names and need to update a few approved X-Domain canvas tests due to retirement of http://test2.w3.org/

IRC Log
[08:05] <krisk> Lets start the meeting
[08:05] <gsnedders> Ms2ger: Nobody here would let me borrow their deck :)
[08:06] <krisk> Agenda: http://lists.w3.org/Archives/Public/public-html-testsuite/2011Jan/0013.html
[08:06] <Ms2ger> Oh, so that's why you're still hanging around with us ;)
[08:06] <gsnedders> Ms2ger: Heh, I was actually at a lecture for the past hour :)
[08:06] <plh> test.w3.org doesn't execute php files, so I think we're fine with keeping around
[08:06] <gsnedders> jgraham: ping?
[08:06] <plh> for test2.w3.org, I need to ask
[08:06] <@jgraham> pong
[08:06] <krisk> So test2.w3.org has been turned off
[08:06] <gsnedders> (Also, I'm vanishing at around half past)
[08:07] <@jgraham> Neat trick
[08:07] <plh> either turned off, or the webmaster missed it
[08:07]  * jgraham meant gsnedders vanishing fwiw
[08:07] <krisk> Though we have about 8 canvas tests that use this for images
[08:07] <Ms2ger> One is enough, I guess
[08:07] <krisk> Well we need two since the tests are for canvas X-Domain
[08:08] <plh> with test.w3.org and w3c-test.org, this should be enough
[08:08] <@jgraham> Doesn't that have the same trust problems that using just test.w3.org had?
[08:08] <@jgraham> e.g. for cookies and so on
[08:09] <krisk> plh how about we have these tests use http://w3c-test.org/html/ for these images
[08:09] <Ms2ger> Let me restate that, one non-w3c-test.org-domain is enough
[08:09] <plh> james, not unless we don't run php on it I think
[08:09] <plh> s/unless/as long/
[08:09] <krisk> The theory is that since test.w3.org doesn't have php it can't be used as an attack vector
[08:10] <@jgraham> Hmm. OK. I thought that we were concerned that scripts in general could be used for XSS type attacks
[08:10]  * gsnedders changes his vanishing plan, and vanishes now
[08:10] <@jgraham> As well as obvious problems with PHP
[08:10] <plh> oh
[08:11] <plh> ok, so we need to retire test.w3.org then
[08:11] <krisk> Yes
[08:11] <plh> and come up with test2.w3c-test.org
[08:11] <plh> or something like it
[08:11] <krisk> The would work...
[08:11] <@jgraham> Yeah
[08:11] <plh> ACTION: plh to retire test.w3.org
[08:11]  * RRSAgent records action 1
[08:11] <krisk> then the canvas tests can't be updated to point to test2.w3c-test.org
[08:11] <@jgraham> There may even be some cases where we want something not under w3c-test.org
[08:11] <@jgraham> e.g. document.domain tests
[08:12] <krisk> s/can't/can/
[08:12] <plh> so, would we need an other domain?
[08:12] <plh> or could we deploy something specific on w3.org ?
[08:13] <plh> something that doesn't get automatically updated through mercurial
[08:13] <Ms2ger> If we're sure we'd catch problems in review, I suppose that would work
[08:13] <@jgraham> I think a second domain would be good
[08:13] <krisk> yes
[08:14] <@jgraham> But it's not a priority since only a subset of cross-domain tests will need it
[08:14] <Ms2ger> But a second domain would probably be safest, once we need it
[08:14] <@jgraham> A subdomain should be fine for many cases
[08:15] <krisk> for example w3-testdomain.org
[08:15] <plh> ok
[08:15] <plh> I guess we'll figure out something once we get there. shouldn't be too difficult or too long either
[08:16] <krisk> So to solve the current problem at hand
[08:16]  * Zakim sees So on the speaker queue
[08:17] <krisk> We should have philip taylor point to w3-test.org for the approved canvas tests that are now failing
[08:17] <plh> q- So
[08:17]  * Zakim sees no one on the speaker queue
[08:18] <krisk> Looking at bugzilla
[08:18] <krisk> http://www.w3.org/Bugs/Public/show_bug.cgi?id=11825
[08:18] <krisk> we have a bug for one of the recently approved innerHTML uses on mathml
[08:19] <krisk> They are xml based elements - so no innerHTML
[08:20] <krisk> gsnedders/jgraham do you concur?
[08:20] <plh> so, the test needs to be removed until it is fixed
[08:20] <krisk> Agree - let's remove it and have the test owner fix the test
[08:21] <krisk> ms2ger I assume you agree since you opened the bug
[08:21] <Ms2ger> Indeed
[08:21] <krisk> Moving on to harness changes...
[08:22] <krisk> jgraham: ping?
[08:22] <plh> I'm trying to get the web performance folks not to fork the harness btw
[08:22] <@jgraham> Yes, I concur
[08:22] <@jgraham> plh: You saw my email about the harness?
[08:22] <krisk> I like the idea of having one harness
[08:22] <plh> yep
[08:22] <@jgraham> I don't understand exactly what their requirements are
[08:23] <krisk> Anderson will respond back  with a use case
[08:23] <@jgraham> Great
[08:24] <krisk> Any update on adding support for setup/tearDown?
[08:24] <krisk> to the harness
[08:25] <krisk> from http://lists.w3.org/Archives/Public/public-html-testsuite/2010Dec/0012.html
[08:25] <@jgraham> I haven't tried it. What is the exact requirement? That each file should be able to define one setup function that is run before any tests and one teardown function that is run after any tests?
[08:26] <@jgraham> And that exceptions in those functions should be handled gracefully? (how?)
[08:26] <@jgraham> I see why setup will be useful, but not teardown, tbh
[08:27] <@jgraham> Web pages typically don't have explicit cleanup steps because you can't enforce that they actually run
[08:27] <krisk> The idea is you can do a bunch of work in a function - then later assert that it is all correct
[08:27] <krisk> today this has to be inlined
[08:28] <krisk> Seems reasonable to have a 'testsetup' function called
[08:28] <krisk> that can do this and be undercontrol of the harness
[08:28] <@jgraham> Right, I think I agree that setup is useful
[08:29] <krisk> I would think that then if this testsetup throws an exception, the test() function would still need to be called
[08:29] <krisk> so that we get proper failures
[08:30] <krisk> Basically eat the exceptions from testsetup
[08:31] <@jgraham> I'm not sure that will work in all cases
[08:31] <@jgraham> But I see the idea
[08:31] <krisk> then make the testsetup configurable or add two potential types of testsetup
[08:32] <krisk> one that upon exception fails the test and another that ignores exceptions
[08:32] <@jgraham> I mean reporting the failures from setup as failures in the tests
[08:32] <Ms2ger> I'm not sure why you'd want to ignore something going wrong
[08:32] <@jgraham> Since you might now know what tests are defined
[08:33] <@jgraham> So there might have to be a way to report that the file as a whole failed
[08:33] <@jgraham> and that you don't know what specific tests failed
[08:33] <@jgraham> (or at least that you might only have a subset of them)
[08:34] <krisk> maybe report that test setup failed and then still runs the tests
[08:34] <krisk> if a test passes when the setup fails that would be a bad test
[08:34] <@jgraham> That would just be weird
[08:35] <@jgraham> Not impossible of course; some tests might only depend on some of the setup
[08:35] <krisk> So it would be possible to fail test setup and pass some tests but not all
[08:35] <krisk> which i think would be OK
[08:36] <krisk> We could also just fail all tests when test setup fails
[08:36] <@jgraham> Many things are possible :) However I think there is enough to go on to make an implementation of this for deeper critique
[08:36] <krisk> surely it's easier to 'test' the tests
[08:36] <krisk> great so do you want to propose this to the list?
[08:37] <Ms2ger> Or implement it? :)
[08:38]  * jgraham will implement it and ask for feedback :)
[08:38] <Ms2ger> Excellent
[08:39] <krisk> Shall we adjorn the meeting?
[08:40] <plh> where are we on the reftest support?
[08:40] <krisk> I'll make the update - got blocked on Hg not proping to test.w3.org
[08:40] <plh> ok
[08:41] <plh> I guess we'll setup a redirect from test.w3.org to w3c-test.org
[08:41] <Ms2ger> ETA on submitting your tests for <section> and friends, Kris?
[08:42] <krisk> I should be able to submit them this week
[08:43] <krisk> rrsagent, generate minutes
[08:43] <RRSAgent> I have made the request to generate http://www.w3.org/2011/01/25-htmlt-minutes.html krisk


-----Original Message-----
From: public-html-testsuite-request@w3.org [mailto:public-html-testsuite-request@w3.org] On Behalf Of Kris Krueger
Sent: Monday, January 24, 2011 8:19 PM
To: 'public-html-testsuite@w3.org'
Subject: HTML Testing Task Force Conf Call Agenda 1/25/2011

Agenda

#1 Check for any bugs on approved tests
#2 w3-test, test.w3.org and test2.w3.org (it seems that hg no longer syncs to test.w3.org?)
#3 Harness changes - 
	http://lists.w3.org/Archives/Public/public-html-testsuite/2011Jan/0011.html
	http://lists.w3.org/Archives/Public/public-html-testsuite/2010Dec/0012.html

If you have other items you would like, please email me directly.

-Thanks!

IRC #HTMLT
Time 16:00-17:00 UTC (11:00am-12:00pm Boston local) Zakim Bridge +1.617.761.6200, conference 48658

Received on Thursday, 3 February 2011 16:33:00 UTC