W3C home > Mailing lists > Public > public-html-testsuite@w3.org > September 2010

RE: HTML Testing Task Force Conf Call Agenda 9/7/2010

From: Kris Krueger <krisk@microsoft.com>
Date: Tue, 7 Sep 2010 17:03:15 +0000
To: "'public-html-testsuite@w3.org'" <public-html-testsuite@w3.org>
Message-ID: <FA9085650D2F8B4A9D501ED9D9706E9A14AD75CA@TK5EX14MBXW652.wingroup.windeploy.ntdev.microsoft.com>
Notes:

* Remove approved test case for bug #10532 (http://www.w3.org/Bugs/Public/show_bug.cgi?id=10532)
* Review another batch of Philip Taylor canvas tests
* Clarification on test submission (tests MUST not contain personal views, tests containing personal views will be rejected)
* Discussion on pass rates and reporting with to multiple tests per page

Attendees - krisk (Microsoft), jgraham (Opera), gsnedders (Opera)

-Kris

IRC Log is below
*** krisk [836b0066@128.30.52.43] has joined #htmlt
*** krisk trackbot gsnedders @jgraham
*** Channel created on Sun Aug 15 01:32:28 2010
<krisk> zakim, this is htmlt
<krisk> trackbot-ng, start telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> zakim, this is htmlt
<krisk> zakim, who is here?
<krisk> Looks like the call is not setup...
* jgraham will just be on IRC
<krisk> Looks like I am the only one dialed in, will others be dialed in?
<jgraham> gsnedders is not planning to afaik
<krisk> zakim, list conferences
<krisk> Looks like zakim is not on IRC
<gsnedders>  /invite Zakim does magic
*** Zakim [rrs-bridgg@128.30.52.169] has joined #htmlt
<krisk> thanks
<krisk> zakim, this is htmlt
<Zakim> ok, krisk; that matches HTML_WG(HTMLT)11:00AM
<krisk> trackbot-ng, start telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> trackbot-ng, start telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> trackbot, associate this channel with #htmlt
<trackbot> Associating this channel with #htmlt...
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> zakim, list confreneces
<Zakim> I don't understand 'list confreneces', krisk
<krisk> zakim, list conferences
<Zakim> I see Team_(hcls)16:55Z, W3C_Global(Vision)10:00AM, VB_VBWG()10:00AM, Team_(a11y-bugs)15:01Z, T&S_XMLSEC()10:00AM, XML_ET-TF()11:00AM, HTML_WG(HTMLT)11:00AM, SW_RIF()11:00AM active
<Zakim> also scheduled at this time are W3C_VALID()11:00AM, WAI_UAWG(CHAIRS)10:30AM, SW_HCLS(COI)11:00AM, SW_(SPARQL)10:00AM
<krisk> zakim, this is htmlt
<Zakim> krisk, this was already HTML_WG(HTMLT)11:00AM
<Zakim> ok, krisk; that matches HTML_WG(HTMLT)11:00AM
<krisk> let's get going - looks like this will be all on IRC
<krisk> Agenda-> http://lists.w3.org/Archives/Public/public-html-testsuite/2010Sep/0005.html
<krisk> Agenda Item #1 Check for any bugs on approved tests
<krisk> Unlike the past - we have a bug
<krisk> see http://www.w3.org/Bugs/Public/show_bug.cgi?id=10532
<krisk> jgraham/gsnedders can you take a peek at this 'bug'
<krisk> Seems like the spec should clarify this behaviour
<krisk> Today IE/Opera honor this in html and xhtml
<krisk> though Firefox and webkit don't honor this attribute
<jgraham> Which behaviour?
<krisk> valign
<jgraham> Right, iirc the spec makes no normative conformance requirements here
<jgraham> So we can't have tests for it
<krisk> since the behavior is not consistent - don't you think the spec should make this a normative requirement?
<jgraham> Isn't it presentation?
<jgraham> The spec can't require specific presentation
<gsnedders> jgraham: It's still possible to have tests, as CSS 2.1 does, for should/may/expected behaviour
<krisk> css won't want to touch this since it's tables
<jgraham> I am quite opposed to us testing SHOULD behaviour
<jgraham> in general
<jgraham> At the very least such tests much be kept entirely seperate
<krisk> Agree testing SHOULD, LIKE or MAY doesn't lead to interop
<krisk> IMHO seems that we can ask the spec to change - is this not the whole point of testing?
<jgraham> The spec can't require presentation though
<jgraham> It just doesn't make any sense
<krisk> why not - at least for some elements
<gsnedders> jgraham: I think there should be tests, but they _MUST_ be marked in someway to not get mixed up (be that metadata within the tests as is the CSS case, or in a separate folder)
<gsnedders> jgraham: Well, the advice it gives should at least follow UAs
<jgraham> Sure if the spec has non-normative advice that doesn;t follow UAs
<gsnedders> jgraham: Which is the case here
<jgraham> gsnedders: Not from what krisk said
<jgraham> UAs are split 50/50
<gsnedders> And that's the whole question, is the spec with the right half of UAs
<jgraham> Well, sure you can argue that it isn't
<krisk> why does the valign attribute exist - if it doesn't actually do vertical alignement?
<jgraham> krisk: Historical reasons
<krisk> I suspect that is why half the UA's tested actually do alignment
<jgraham> But vertical alignment is highly media dependant and generally HTML is media-independent
<jgraham> e.g. a tty browser might not be able to lay out tables at all
<jgraham> hence the rendering section is non-normative
<krisk> that doesn't seem to help the vast majority of actual UA's though..
<jgraham> In any case I think people are free to file bugs on the spec if they think it is wrong, but that doesn't change the fact that the test is wrong because it tests something non-normative
<jgraham> +at least
<krisk> that is true
<krisk> I'll open a bug and remove the test from the approved folder
<krisk> let's move on 2 agenda item #2
<krisk> Agenda Item #2 Call to review the next 25 Philip Taylor's Canvas tests
<krisk> I'll review another batch of philip's tests by the next meeting
<krisk> if folks have objections to these tests please raise them in the list or at the next meeting
<krisk> Agenda item #3 Discuss test submission edict
<krisk> Never thought that I'd have to actually clarfy that tests should not state personal views in tests
<krisk> Test that state personal views will not be approved
<krisk> Especially ones that don't help collaboration
<krisk> jgraham/gsnedders - do you agree?
<jgraham> That sounds sensible...
<krisk> OK then moving on....
<jgraham> (depending on what you mean by personal views. I mean it seems OK to say "I believe the spec means X")
<jgraham> (but I assume that is not what you mean)
<krisk> that is not...
<krisk> for example is the test author decided to promote or demote a product, for example...
<krisk> in America alot of debate occurs around which is better a Ford or a Chevy
<krisk> Basically the test should not have any other that test information - all others text would be grounds for rejection.
<jgraham> Yes it seems sensible that tests should be written neutrally
<krisk> Agenda item #4 getElemtnsByClassName integration
<krisk> I have the start_callback implemented (not pushed)
<jgraham> What are you doing in start_callback?
<krisk> though today we track a pass/fail per page load
<jgraham> it is completion_callback once all the results are known...
<krisk> I understand that part - I should have just stated  ***_callback
<jgraham> OK, just checking
<jgraham> :)
<jgraham> The test harness allows multiple tests per page to cover cases where e.g. you have several very similar things to test in a loop
<jgraham> I think the runner needs to deal with that too
<jgraham> Somehow
<jgraham> Like accepting an array of results per page
<krisk> That is fine - though for reporting purposes we should discuss how this a page with some  passes and some failures is represented
<krisk> I will track each test in the pass rate
<krisk> so if you have 9 tests on page 1
<jgraham> Right
<krisk> and 1 test on page 2
<krisk> passing 5 of the first tests on page one
<krisk> and none on page 2
<krisk> the pass rate would be 50% (5/10)
<jgraham> Yes, that seems right
<jgraham> Do we currently have any mechanism for presenting test results?
<krisk> yes
<jgraham> where?
<jgraham> I see the xml files?
<jgraham> s/?//
<krisk> I'll send you the link
<krisk> http://test.w3.org/html/tests/reporting/report.htm
<krisk> This is come up alot in the past meetings here and at the weekly HTML5 meeting
<krisk> The % we discussed would apply to the top of the page
<jgraham> I didn;t realise we had actually implemented it already
<jgraham> (and missed the file in my checkout)
<jgraham> So instead of PASS / FAIL put 1/1 or 3/5 or whatever
<krisk> Ideally if a page has more than one test - it can be take a param
<krisk> e.g. canvas?1
<jgraham> That could be problematic in theory unless people are very careful
<krisk> Seems like the 1/1 or 3/5 will work then for the bottom
<jgraham> and in strange cases might lead to different results if you run a single test compared to running all tests
<jgraham> even if people are careful
<jgraham> I think just using numbers is the simple solution for now
<krisk> simple is best
<krisk> any other items for discussion?
<krisk> if not we can adjurn
<gsnedders> krisk: Do you mind if others edit that page?
<krisk> yes
<jgraham> Not really. I was vaugely considering making some experimental changes to the harness you wrote, but I won't change anything without review
<krisk> In theory you can just update the xml file for a UA
<jgraham> And might not get round to it anyway
<jgraham> Oh, I remember something
<krisk> Though the co-chairs asked for a date
<gsnedders> krisk: I mean the HTML viewer, not the XML data
<krisk> for when the tests were ran
<jgraham> The list of tests needs more metadata
<krisk> like what?
<jgraham> specifically if we want to support reftests, which we agreed we do, we need type, url refurl
<jgraham> rather than just url
<krisk> I don't agree to ref tests
<jgraham> I thought we agreed to that last time?
<krisk> the whole concept doesn't work
<jgraham> Particularly self-describing reftests
<Zakim> disconnecting the lone participant, [Microsoft], in HTML_WG(HTMLT)11:00AM
<Zakim> HTML_WG(HTMLT)11:00AM has ended
<Zakim> Attendees were [Microsoft]
<krisk> since you need many 'references'  - one for a unix box, one for a phone, one for MAC, etc...
<krisk> The ref image ends up not really being a reference at all
<jgraham> Well CSS are planning to use them almost exclusively for CSS3
<krisk> just like you stated about about tty devices...
<gsnedders> krisk: It's not an image
<jgraham> And I don't think your concern plays out if you design the test well
<gsnedders> krisk: Just something the spec defines should render the same
<jgraham> Since both the refernce and the test should be adjusted in the same way
<gsnedders> (some simpler way to get the same rendering)
<jgraham> I'm pretty sure e.g. Mozilla use the same reftests on phones and on desktop
<krisk> that works in theory until you deal with lossless rendering (for example jpg)
<jgraham> s/lossless/lossy/?
<gsnedders> Where does JPG come into it, though?
<jgraham> In any case I don't see why we would be testing jped genering
<jgraham> *rendering
<jgraham> Maybe we should take this to the list
<krisk> any other items - not ones that have been going on for a long time (ref images)
<krisk> another simple case is buttons...
<krisk> another simple case is inset on a table or border...
<krisk> let's adjurn
<gsnedders> Yes, not everything can be reftests. But most things can be

-----Original Message-----
From: public-html-testsuite-request@w3.org [mailto:public-html-testsuite-request@w3.org] On Behalf Of Kris Krueger
Sent: Monday, September 06, 2010 9:30 PM
To: 'public-html-testsuite@w3.org'
Subject: HTML Testing Task Force Conf Call Agenda 9/7/2010

Agenda

#1 Check for any bugs on approved tests
#2 Call to review the next 25 Philip Taylor's Canvas tests
#3 Discuss test submission edict
#4 Discuss the getElementsByClassName tests - integration with the test runner

If you have other items you would like, please email me directly.

-Thanks!

IRC #HTMLT
Time 16:00-17:00 UTC (11:00am-12:00pm Boston local) Zakim Bridge +1.617.761.6200, conference 48658
Received on Tuesday, 7 September 2010 17:03:58 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 7 September 2010 17:03:59 GMT