RE: HTML Testing Task Force Conf Call Agenda 10/5/2010

Notes - 

* Meeting Schedule extended, see list and WIKI
* All of Philips except for case http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/2d.canvas.readonly.html
     + Need to remove the TODO, technically it could throw in strict mode
* Need up update Microsoft Canvas test to use same format as Philip's tests

IRC Log - 
<krisk> will anyone be dialing in?
<krisk> looks like zakim is not up..
<jgraham> Not dialing in
<jgraham> But around
<jgraham> Don't think gsnedders is around
*** Zakim [rrs-bridgg@128.30.52.169] has joined #htmlt
*** trackbot [trackbot@128.30.52.169] has joined #htmlt
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> zakim, list confrences
<Zakim> I don't understand 'list confrences', krisk
<krisk> zakim, list conferences
<Zakim> I see SW_(SPARQL)10:00AM, WAI_PFWG(HTML_TF)11:00AM, W3C_VALID()11:00AM, WAI_UAWG(CHAIRS)10:30AM, VB_VBWG()10:00AM, W3C_(W3SC)11:00AM active
<Zakim> also scheduled at this time are SW_HCLS(COI)11:00AM, SW_RIF()11:00AM, HTML_WG(HTMLT)11:00AM, XML_ET-TF()11:00AM, DIG_weekly()11:00AM, T&S_XMLSEC()10:00AM, Team_(bigm)14:08Z
<krisk> zakim, this is htmlt
<Zakim> krisk, I see HTML_WG(HTMLT)11:00AM in the schedule but not yet started.  Perhaps you mean "this will be htmlt".
<krisk> zakim, this will be htmlt
<Zakim> ok, krisk; I see HTML_WG(HTMLT)11:00AM scheduled to start 7 minutes ago
<krisk> trackbot-ng, start telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<Zakim> HTML_WG(HTMLT)11:00AM has now started
<Zakim> +[Microsoft]
<krisk> zakim, Microsoft is krisk
<Zakim> +krisk; got it
<krisk> looks like no one else is dialed in
<krisk> let's just do this on IRC - assume that no one will dial in
<krisk> Agenda http://lists.w3.org/Archives/Public/public-html-testsuite/2010Oct/0003.html
<krisk> Agenda item #1
<krisk> No new bugs exist in bugzilla
<krisk> item #2 Approval for next 25 Phillip Taylor tests (one issue see below)
<krisk> http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/2d.canvas.readonly.html
<jgraham> I think that test is OK
<jgraham> as is
<jgraham> Unless I am missing something
<jgraham> Setting to readonly attributes doesn't throw
<jgraham> (well, possibly it will in ES5 strict mode, but ignoring that for now since WebIDL doesn't cover that yet)
<krisk> So then the TODO should go away
<jgraham> Yes
<krisk> If WebIDL changes then the test can get fixes
<jgraham> I'm pretty sure that WebIDL can't change for the non-strict-mode case here
<krisk> why would that be?
<jgraham> Back-compat
<jgraham> I assume pages depend on setting reasonly attributes not throwing
<jgraham> *readonly
<krisk> In my experiecne I have not seen this issue arise very often
<jgraham> Maybe. I would consider it a very scary change. But anyway I don't think anyone is asking for that change in WebIDL
<krisk> if it were to throw then everytime a web dev added a noopt line of script they would get caught
<jgraham> Yes, but that's the point of strict mode
<krisk> just like in more modern languages won't compile a if(d=5) - like you can in c
<jgraham> Making that kind of backwards-compatibility affecting change behind a supposedly safe pragma
<krisk> Ok then let's just have philip remove the todo
<krisk> moving on to agenda item #3
<krisk> Lots of feedback - which is good since it will help browser interop
<krisk> james you seemed concerned about some of the non-rendering cases being manual
<krisk> they will fit into the test runner - just like the getElementsByClassName tests
<jgraham> I am concerned about all manual tests
<krisk> I understand your concern - though the non-rendering cases are not manual
<jgraham> The ones that reported a result using javascript are not a problem
<jgraham> I mean from the point of view of automation
<jgraham> The ones that require human interaction are a problem. CSS currently has a huge problem with this
<jgraham> They have thousands of unautomated tests that is blocking progress on making implementaion reports and may delay them getting to Rec.
<jgraham> So they are having a last-minute attempt to automate tests
<jgraham> We don't want to be in that situation in a few years time
<jgraham> Also, from the point of view of a browser maker, tests that require manual work are more effort, and more prone to break, than automated tests. So they are a big cost
<jgraham> That's assuming that you have a system for running such tests in a semi-automated way, which not all browser makers have
<krisk> I think the other problem with the css group is all the test work was done at the very end
<jgraham> (as it turns out, Opera do have such a system, and we still want to avoid using it wherever possible)
<krisk> a lot of the tests in the css suite have been around for a long time
<jgraham> The upshot of all of this is that I am loathe to approve tests that are not automated and do not come with a clear explaination of why they *cannot* be automated
<krisk> not sure why running them and doing implementation reports had to wait till the very end
<jgraham> You have to run them at the very end because the testsuite will change right up to the very end
<jgraham> and implementations will keep changing
<krisk> But the churn goes way down...like any really big software project
<jgraham> The churn on implementations might not
<jgraham> I mean the testsuite might converge gracefully
<jgraham> (or might not, anyone could dump thousands of tests at the last minute)
<krisk> dumping 1000's of tests at the last minute is not good
<jgraham> But implementations move at a steady pace and we can assume it will be the bleeding edge implementations that we need for the IR
<jgraham> Even if things work out for the IR, the QA point is IMHO more significant
<krisk> Did someone from google run all the css tests in 3 days?
<jgraham> Don't know
<krisk> yep - look in the css lists - it was done by tab atkins
<jgraham> In any case, 3 days is an insane amount of time for a single run on a single platform
<jgraham> We need tests that we can run continuously on a variety of platforms and configurations
<krisk> really - surely google can afford to hire a person for less than a week once every few years to get a spec to rec
<jgraham> I'm not sure what the relevance of what Google can afford to do is
<krisk> my point of view is that the reviewing of the tests are far bigger than the cost to run the tests
<jgraham> The point is that it's not in anyone's best interests to have tests that are cumbersome to run
<jgraham> Because a test is written once and run thousands of times
<jgraham> If we want implementations that actually pass all the tests then we need them to be easy to run
<jgraham> So people can make changes without regressing other tests
<jgraham> Just running the tests is only half the battle; having the interoperable implementations is the other (harder) part
<jgraham> That's why I am more concerned about the QA aspect than the IR aspect
<krisk> So does opera have a had time running the css tests?
<krisk> I would think that you (Opera) would have found bugs in your implementation a while back
<krisk> especially from tests that have been submitted years ago
<jgraham> Yes. We have a system that allows us to automate running them after the first time, but getting the initial data about what is a pass and what is a fail is difficult and error-prone. Then small changes in the implementation can invalidate a lot of the data and require all the manual effort to be redone
<jgraham> We are actively trying to replace the manual CSS tests with reftests where possible
<jgraham> We don't want to have to lead a similar cleanup operation again
<krisk> so can't you build your automation system in // to the html5 wg progress?
<jgraham> The problem with the automation system is fundamental
<jgraham> It depends on screenshots and there are a lot of things that can cause a screenshot to change other than a failed test
<krisk> sure that it software development
<krisk> when a product churns - regressions can blow up 1000's of tests (for example a fault in the network stack)
<jgraham> But it's not a regression
<jgraham> It can just be a change
<jgraham> Different rounding behaviour across platforms
<krisk> keeping it more relavant looking at http://test.w3.org/html/tests/submission/Microsoft/canvas/canvas_text_font_002.htm
<jgraham> Or due to an optimisation
<krisk> this is a visual test
<jgraham> Or due to a font rendering library change
<krisk> how do you think this test shoud be written?
<jgraham> krisk: That could be a reftest. Or Philip has offered to make it into a havascript test using his framework
<jgraham> *javascript
<krisk> his canvas suite is not automated
<krisk> it also is all manual
<krisk> anyhow lets have the visual tests move to philip's framework
<jgraham> Philip's tests all report their results via javascript
<jgraham> They might not be integrated with the harness correctly
<jgraham> But they could be
<krisk> ok
<krisk> The feedback was the case with an img tag that points to SVG
<krisk> I don't think the SVG WG will be testing canvas with SVG
<krisk> I do think that it's important to test this case - thus the reason Microsoft submitted the case
<jgraham> I don't think we can require support for this case in the testsuite since the spec doesn't require support for it
<jgraham> Even SVG 1.2 Tiny has dropped that requirement
<krisk> surely you think that having an img tag support SVG is good for the web
<jgraham> Yes, I do
<jgraham> But lots of things would be good for the web that we can't test
<jgraham> Like common support for a freely-redistributable video codec
<krisk> well a test exists for this case and more than one browser supports canvas being used in this case
<Zakim> disconnecting the lone participant, krisk, in HTML_WG(HTMLT)11:00AM
<jgraham> I don't disagree. But there is no normative requirement in the spec to support this feature
<Zakim> HTML_WG(HTMLT)11:00AM has ended
<Zakim> Attendees were krisk
<krisk> we'll we have a difficult judgement call - just like with all the webIDl tests
<krisk> strictly speaking we could reject all the cases that end up depending up other w3c specs that are not recommended.
<jgraham> The WebIDL tests are different. There there exists a spec that is used by HTML5 to assert normative critera
<jgraham> That's quite different to a third party spec that is not normatively required by HTML5 adding features to HTML5
<krisk> So then with all the effort done with SVG and in HTML5 we can't make help browser interop by testing canvas with inmg tags that point to svg content
<krisk> That seems like a good item to bring back to the co-chairs
<jgraham> It seems that way, yes. We can of course have some way of marking tests that are for non-normative requirements
<jgraham> And it would certainly be appropriate for the SVG WG to test this, since they are the ones making up the normative criterion
<krisk> No that makes stuff too mushy
<jgraham> What does?
<krisk> non-normative requirements
<krisk> non-normative requirements lead to bad interop
<jgraham> Well they exist of course
<jgraham> The question is whether we should test them
<jgraham> I agree that avoiding them in the spec is good
<jgraham> For exactly that reason
<jgraham> Your other course of action, if you believe that SVG support should be required, is to file a bug stating that
<krisk> we will see..
<krisk> Last item - meeting time(s)
<krisk> I assume that this time still works - if not respond to the list
<krisk> Also the 11/2 meeting won't happen due to TPAC
<krisk> any other items or shall we adjourn?
<jgraham> Nothing from me
<jgraham> Thanks

-----Original Message-----
From: Kris Krueger 
Sent: Monday, October 04, 2010 4:42 PM
To: Kris Krueger; 'public-html-testsuite@w3.org'; Philip Taylor <pjt47@cam.ac.uk> (pjt47@cam.ac.uk)
Subject: HTML Testing Task Force Conf Call Agenda 10/5/2010

Agenda

#1 Check for any bugs on approved tests
#2 Approval for next 25 Phillip Taylor tests (one issue see below)
#3 Ask for feedback from canvas tests, see -> http://lists.w3.org/Archives/Public/public-html-testsuite/2010Oct/0002.html
#4 Meeting Times - plan on extending schedule same time, etc.. and cancel the 11/2/2010 meeting (TPAC)

If you have other items you would like, please email me directly.

-Thanks!

IRC #HTMLT
Time 16:00-17:00 UTC (11:00am-12:00pm Boston local) Zakim Bridge +1.617.761.6200, conference 48658

Canvas Tests with a Bug
#1 http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/2d.canvas.readonly.html

Test has a TODO: -> Note that indeed it should throw // TODO: not sure whether this should throw or not...

Received on Thursday, 14 October 2010 14:07:21 UTC