- From: Michael[tm] Smith <mike@w3.org>
- Date: Thu, 3 Nov 2011 09:00:19 +0900
- To: public-test-infra@w3.org
http://www.w3.org/2011/10/28-testing-minutes.html
28 Oct 2011
Agenda
http://lists.w3.org/Archives/Public/public-test-infra/2011OctDec/0014.html
Attendees
Present
Jeane_Spellman, Bryan_Sullivan, Wilhelm_Anderson,
James_Graham, Elika_Etemad, Jason_Leyba, Simon_Stewart,
Kris_Krueger, John_Jansen, Peter_Linss, Mike_Smith,
Alan_Stearns, Narayana_Babu_Maddhuri, Duane_O'Brien,
Charlie_Scheinost, Ken_Kania, Jeff_Hammel, Clint_Talbert,
Tab_Atkins, Michael_Cooper, Philippe_Le_Hégaret
Chair
Wilhelm_Andersen
Contents
* Topics
1. Introductions
2. Agenda Overview
3. WebDriver API
4. Testing IE
5. Testing Firefox
6. Testing Opera
7. Testing in the CSS WG
8. Testing Chrome
9. krisk_
10. testharness.js
11. How should we organize public test suites so that they
are as easy as possible to contribute to and reuse?
12. Additional Items
13. Conclusions and Action Items
* Summary of Action Items
_________________________________________________________
<MichaelC_SJC> scribeNick: MichaelC_SJC
Introductions
wa: testing helps everybody
figure out how to make best possible test suites
<plh> Wilhelm: I'd like to figure how to make the best possible test
suite, how to make the Web better
I work for Opera as testmonkey, test manager
in various parts
jg: also work for Opera
<missed the rest>
ee: also known as fantasai
work on testing in CSS WG
jl: work on testing in Google
want to improve the ecosystem so it all works better
ss: created Webdriver, working Selenium
very aware of the differences between browsers, would love to sort
it out
kk: worked in testing at Microsoft
more recently on Web standards
jj: also at Microsoft
interested in automation, test suites
pl: co-chair of CSS WG
have contributed extensively to that test suite
and working on test shepherd for <missed>
ms: work for W3C, staff contact to HTML WG
work on testing for HTML, extensive contributions to framework
as: working for Adobe
interested in tests working across browsers
represent Nokia
nm: learn what's up
do: <missed>
<MikeSmith> https://browserlab.adobe.com/en-us/index.html <-
Adobe BrowserLab
https://browserlab.adobe.com/en-us/index.html
cs: represent adobe
<simonstewart> Ken_Kania
kk: work for google, Webdriver
bs: AT&T, mobile data services
interoperability in various fora
want to understand the challenges browser vendors have in automation
and how to leverage tools in repeatable continuous framework
to certify new devices as they come out, get updated, etc.
jh: Mozilla, test automation
ct: Mozilla, testing
ta: Google, work on Chrome
not as closely involved in testing, but have worked in CSS on some
<plh> involed in WAI. zstaff contact for PF, developping ARIA> we're
struggling in testing. hoping to contribute to the test framework
<plh> ... we have reuirements that we'd like to bring as well
plh: W3C, Interaction Domain, lots of your favourite groups
want a common framework, common way to write tests
Agenda Overview
wa: first, want browser vendors to introduce how they do testing
then, presentations of a few testing approaches
finally, discussion of how to write tests for different types of
functionality
90% of tests cover how something rendered to screen in a particular
way
or script returns an expected result
or user fills out form and certain result
WebDriver API
ss: WebDriver is an API for automation of WebApps
developer-focused, guides people to writing better tests
Merged with Selenium a couple years ago
fairly simple, load page, find element, perform actions like focus,
click, read, etc.
kk: does it simulate user input at driver level, or elsewhere?
ss: in past user interactions were done by simulating events in DOM
but browers inconsistent in how they handle those
when they do what etc.
so events at script level not feasible
so do events at OS level
that is high fidelity but terrible machine utilization
and wastes developer's time
so now, allow window not to have focus and send events via various
OS APIs
but OS not designed to send high fidelity user input to background
window
so now, Opera and Chrome pump events into event loop of browser
<scribe not sure that was caught right>
Webdriver has become a de facto standard for browser automation
most popular open source framework
as can be seen by job postings requiring familiarity with it
has reasonable browser support
Opera, Chrome, and Android add-on, Mozilla starting
uses Apache2 license
business-friendly license
nm: tried on mobile browsers?
ss: yes, in various <lists>
it's a small team
covering wide range of browsers and platforms
see 3 audiences for automation
1) App developers are vast majority
need to test applications
hard to get developers to write tests, and can only get them to
write to one API when you get it at all
first audience for WebDriver
2) browser vendors
desire to automate their testing as much as possible
bs: how does Webdriver related to qunit <sp?>
ss: <didn't catch details>
bs: so Webdriver isn't a framework, it's an API for automating
events
ss: clearly a browser automation API
e.g., understand Opera runs 2 million tests / day with this
3) Spec authors
some specs can be articulated entirely in script
and tested that way
others need additional support, this provides that
ee: more spec testers than authors?
ss: yes, those focusing on test aspects
... user perspective
it's a series of controlled APIs
to interrogated DOM
execute script with elevated priveleges
and provide APIs to interact, so not just read-only
jj: <question missed>
ss: <answer missed>
jj: avoids cross origin vulnerability?
ss: yes
bs: good, some complicated scenarious
ss: implementer view
neutral to transport and encoding
provide JSON
which bring clients that can handle immediately
also have released JavaScript APIs
ss: Security
<JohnJansen> My question was regarding the bypass of the x-origin
security restriction
ss: automation and security are opposite concerns
<JohnJansen> answer: the jscript still honors that restriction,
though webdriver itself ignores it.
generally, build support into browser
and enable it via an additional component
or command line features
ss: Demo
<shows short script, then executes>
kk: how Opera?
ss: Watir on top of WebDriver
... API designed to be extensable
expose capabilities via a simple interface or casting
jj: How are visual verifications handled?
ss: can take a screenshot, platform-dependent
Opera has extended with ability to get hash of the screenshot
attempt to capture entire area described by DOM, not just viewport
deals with difficulties like fixed positioning etc.
but very browser specific
jj: human comparison mechanism?
ss: in google, teams of people do that
we just provide the mechanism
don't want to over-prescribe how to process images, as state of the
art continually changes
bs: to compare layout between different browsers
capture screens, or query position of elements?
ss: can do both
can get location of an element
and size
bs: how about different screens sizes
interested in specifically how things rendered in various
circumstances
ss: the locatable interface can provide various types of measures
kk: differences among browsers are wide for many reasons
it's part of the landscape
ss: was able to use same tests using same APIs
at rendering level can be different
plh: platform AAPIs use similar services
hope e.g., ARIA can use WebDriver
ss: have looked at AAPIs, can look at elements by ARIA role etc.
on relationship to AAPIs
sometimes they're enough, sometimes not
one of the next big things in hybridized apps, part native and part
Web
may need to use AAPIs to test
plh: think ARIA can be tested using this
ss: have applied Webdrive to native app testing using AAPIs
kk: there has been a path starting with MSAA
ss: AAPIs are extremely low-level
e.g., a combobox is represented as a few different controls together
kk: developers create all kinds of crazy things
so UI automation allows patterns
mc: can speak to AAPI from WebDriver
ss: Webdriver sits on top of AAPI
but because of script interface, could talk back and forth a bit
wa: Opera has a layer "Watir" on top of WebDriver
<shows sample>
test file looks like a manual test, e.g., a human could interact
with it
<demos manual execution of test>
<that can also be executed using the script showed previously>
for each test file, there's a block in the automation script
ss: Webdriver simlilar
nm: <missed>
ss: <answer related to webelement.gettext>
jj: why wrapping in Watir
wa: was done before projects had merged
now doesn't matter as much
plan to submit Opera set of tests to HTML WG for official test suite
but want them in a format other browser vendors could use
Opera uses Ruby bindings, Mozilla uses Python bindings
need to automate in all browsers, Webdriver seems way to go
for official W3C tests, question of what language binding to use?
ss: Javascript is hugely known
Python is the other one being explored by Mozilla and Chrome
also is "politically unencumbered"
vs some other candidates out there
<MikeSmith> I vote for Javascript
wa: how complete are JS bindings?
js: still finalizing
kk: <something detailed>
js: API stable
loading script within browser is the part that still needs working
on, to get around sandbox
it's usable now, but have debugging etc. to do
ss: so maybe Python preferable?
jg: having dependency on core could be a big stability issue
<^ not sure that's scribed right>
kk: dangerous to build on things that are changing
otoh, need bindings to be something that's available on all targets
ss: normally test and browser communicate like a client / server
can do over a web socket
and run test on machine independent of browser
wa: was able to test a mobile device on a different continents this
way
plh: if we set up a test server on W3C site, could you allow it to
just run tests at you?
ss: can connect from browser to a test server
so in theory, this works
but security concerns
need a manual intervention to put browser in testing mode
mc: have to trust W3C server from security POV
how we allow tests to be contributed needs to be careful
<general view of usefulness of this approach>
as: <missed>
<JohnJansen> as: is there support for IME? how good is it?
ss: support varies by platform as we prioritize development
<mentions wherefores and whynots>
do support internationalized text input
for testing I18N but could be used to test other stuff
do: how well documented is JS API?
ss: fairly extensive
<jhammel>
http://code.google.com/p/selenium/wiki/JsonWireProtocol
http://code.google.com/p/selenium/wiki/JsonWireProtocol
Facebook developed PHP bindings using this documentation
Selenium stuff hosted under software freedom conservancy
can use w/o the open source stuff, but also handy to use the open
source stuff
wa: Just started browser tools and @@ WG
<jhammel> http://www.w3.org/2011/08/browser-testing-charter
http://www.w3.org/2011/08/browser-testing-charter
primary goal is to standardize Webdriver API at W3C
<jhammel> (i think)
welcome you all to join to make this happen
also want to explore whether all browser vendors can handle official
test suites using Webdriver API
ss: aware of support from Google, Opera, Mozilla
explicit non-support from Microsoft, Apple, Nokia, HP
also support from RIM
plh: would Microsoft be able to accommodate tests using this?
kk: depends
standardization of the API will help a lot
<Another link for the WG is http://www.w3.org/testing/browser/>
http://www.w3.org/testing/browser/%3E
also need tests structured in certain ways we can work with
<fantasai> kk: having the tests be self-describing is very
important. If I was a TV browser vendor that doesn't support
webdriver, I would want to be able to leverage the W3C tests as well
jg: tests always structured so you could run manually, though would
be ridiculous to do so with them all in practice
ms: first thing we need is a spec
doesn't matter where editors draft hosted, can do at W3C
IP commitments kick in when we publish a Working Draft
ss, wa: ready to move right away on that
kk: W3C would own code?
ss: W3C would maintain spec
and a reference implementation
but there could be other implementations
mc: reference implementation doesn't necessarily have to be W3C
plh: spec is most important for W3C
ss: all Google testing in some way related to Webdrive
bs: supported in mobile?
ss: chrome and android
wa: also opera for mobile
bs: so other platforms is just lack of implementation?
ss: right; Nokia and Apple haven't implemented
just need a driver
kk: support IE6? want to get rid of that
ss: drop support when usage drops below a certain level
plh: support from Microsoft for Webdriver API will help HTML WG a
lot
jj: even if Opera submits tests and HTML adopts, they're
self-describing so still testable manually
plh: what does Nokia think?
nm: Nokia not really interested
focused on Webkit stuff
today is first time hearing about it
ss: it's not just about testing a spec, it's about ensuring users
can use content in your browser
so that market force should drive interest even if internal interest
is elsewhere
nm: how is performance?
ss: rapid on Android, but slow on emulator
Iphone is fast directly and in emulator
<something else> fast
nm: <missed>
<jhammel> ^ pixel verification
ss: haven't seen a lot of pixel verification on mobile devices
<scribe having a hard time hearing or understanding remainder of
discussion>
<MikeSmith> agenda:
http://lists.w3.org/Archives/Public/public-test-infra/2011OctDec
/0014.html
http://lists.w3.org/Archives/Public/public-test-infra/2011OctDec/0014.html
<dobrien> Could we get the minutes updated again as well please?
jj: propose not requiring webdriver in first version of test suite
<bryan> Scribenick: bryan
Testing IE
kk: To walk thru testing of IE
... shows slides "Standards and Interoperability"
<fantasai> IE testing diagram: Standards, Customer Feedback,
Privacy, Accessibility, Performance, Security
<fantasai> (these are pictured as hexagrams around a central
"Internet Explorer" label)
kk: IE testing has various chunks as shown on the slide (slides to
be shared)
<fantasai> "Internet Explorer Testing Lab" w/ photo
<fantasai> IE5 -> IE10
<fantasai> 948 Workstations
<fantasai> 119 servers
<fantasai> 1200 virtual machines
<fantasai> remotely configurable
<fantasai> 152 versions of IE shipped every "Patch Tuesday"
<fantasai> Green Lab Initiative saves ~218 tons of CO2/Year
kk: IE testing lab using a lot of machines with a lot of IE versions
tested every week
<fantasai> "Standards Engagement"
<fantasai> ECMA
<fantasai> TC39 (Ecmascript 5)
<fantasai> W3C
<fantasai> - CSS
<fantasai> -WebApps
<fantasai> -HTML
<fantasai> -SVG
<simonstewart> Slides for the webdriver notes:
https://docs.google.com/present/edit?id=0AVrYfCxRNKUGZGc5Nm1ocGh
fNzFnaGd2bmZnYw
https://docs.google.com/present/edit?id=0AVrYfCxRNKUGZGc5Nm1ocGhfNzFnaGd2bmZnYw
<fantasai> -XML
<fantasai> cycle diagram: Testing -> spec editing -> implementations
-> (loop back to Testing)
<fantasai> "Standard Contributions"
<fantasai> - Spec editing
<fantasai> -co-chairing
<fantasai> -test case contributions w3c and ecma
kk: encourage standards engagement and participation in various
groups
<fantasai> -- 14623 tests submitted
<fantasai> -- across IE9/IE9/IE10 features
<fantasai> - hardware (Mercurial server)
<fantasai> - IE Platform Preview Builds
kk: have contributed a lot of tests and hardware
... preview builds allow early access and feedback
<fantasai> "IE10 Standards Support"
<fantasai> CSS2.1 , 2D Transofrms, 3D Transforms, Animations,
backgroudns and Borders, Color, Flexbox, Fonts, Grid alignment,
hyphenation, image values gradients, media querie,s multi-col,
namespaces, OM Views, positioned floats, selectors, transitions
Value sand Units
<fantasai> DOM element traversal, HTML, L3 Core, L3 Events, Style,
Traversal and Ragne
<fantasai> ECMASCRIPT 5
<fantasai> File Reader API
<fantasai> FIle Saving
<fantasai> FormData
<fantasai> Geolocation
kk: IE 10 will support a lot of standards CSS, HTML5, Web APIs, ...
http://ietestdrive.com
http://ietestdrive.com/
<fantasai> HTML5 appcache, asycn cavnas, drag and drop, forms and
validation, structure clone, history API, parser sandbox, selection,
semantic element,s video and audio
<fantasai> ICC Color profiles
<fantasai> Indexed DB
<fantasai> Page Visibliity
<fantasai> Selectors API L2
<fantasai> SVG Filter Effects
<fantasai> SVG standalone and in HTML
kk: also look at the IE blog
<fantasai> Web Sockets
<fantasai> Web Workers
<fantasai> XHTML/XML
<fantasai> XMLHttpREquest L2
<fantasai> "Items for Discussion"
<fantasai> * WG Testing Inconsistent
<fantasai> - when are test created? Before LC? CR?
<fantasai> - Whena re tests reviewd?
<fantasai> - vendor prefixes
<fantasai> - 2+ impl passing test srequired for CR/
<fantasai> * Review Tools (none)
kk: issues are inconsistent testing across WGs
<fantasai> Note -- that's not quite true anymore, plinss wrote one
for csswg :)
kk: when tests are created e.g. related to last call or earlier
... soft rules for how a spec is allowed to progress are maybe not
enough
plh: these are soft rules currently
jj: test tools recently developed have helped with consistency,
flushing our remaining inconsistencies is a goal
... different test platforms result in different tests as submitted
to W3C
Michael_Cooper: experience has convinced that tests should be
available by last call
Kris_Krueger: why would this not be a rec across W3C?
plh: its not easy to enforce
... some WGs will complain
jj: amping the expectations on testing will help
mc: it should be the rule, with exceptions allowed
<Zakim> MichaelC_SJC, you wanted to say I now believe tests need to
be ready by Last Call
Elika_Etemad: implementations are needed to see how tests are
working
James_Graham: the process does not map to browser development
reality
Elika_Etemad: its difficult to say when spec development is done
thus making a hard deadline
<dobrien> @
<dobrien> Mhmv @7
John_Jansen: problems often cause the specs to move backward
<dobrien> Sorry about that.
Elika_Etemad: CR is test the spec phase, not fixing bugs in browsers
... having to move CR back due to bugs is an issue, we need an
errata process to allow edits in CR
plh: we are not here to fix the W3C process
John_Jansen: the more times you go thru the circle
(edit/implement/test) the better, and also the earlier
James_Graham: when we implement we write the tests... test suites
should not be closed
<fantasai> James_Graham: The state of the spec is irrelevant to when
we write tests
Mike_Smith: the Testing IG is scoped broadly perhaps too much so.
The IG will decide what its products will be, e.g. a best practice
on when test suites are developed.
... writing this down even if we do not fix the process will help
others avoid the same mistakes of the past
... it will still have some value
Wilhelm_Anderson: how do you run tests, what is automated, is
development inhouse
Kris_Krueger: write our own tests
plh: from JQuery?
Kris_Krueger: no, customer feedback is also considered
... e.g. Gmail support provides feedback
... have a lot of automated tests, ship every Tuesday, and get quick
feedback from users/developers
Narayana_Babu_Maddhuri: is there any review of the test cases to
determine is the test a valid test, validation of the test results?
plh: the metadata of the test log should clarify what is being
tested
Kris_Krueger: pointing to where the test relates to the spec is
helpful
plh: we cannot force metadata into tests, but we can encourage this
info to help ensure test value clarity
Narayana_Babu_Maddhuri: good reporting would be helpful
plh: knowing e.g. what property works across devices and platforms
is a goal, and matching tests to specs would support that
James_Graham: knowing why something is failing is sometimes
difficult, dependencies are not clear and why the test failed is
unclear
<plh> [lunch]
<MichaelC_SJC> == Lunch break is 1 hour ==
<ctalbert_>
http://people.mozilla.org/~ctalbert/automationpresentation/Autom
ation.html
http://people.mozilla.org/~ctalbert/automationpresentation/Automation.html
Testing Firefox
<krisk_> Firefox Testing Presentation
<krisk_> clint: Tools automation lead at Mozilla
<krisk_> Clint: overview of their testiong
<krisk_> Grown over the years
<krisk_> Test Harnesses
<fantasai> "Automation Structure: Test Harnesses"
<fantasai> - C++ Unit
<krisk_> C++ Unit testing, XPCShell, no too intresting for this
group
<fantasai> - XPCShell (javascript objects)
<fantasai> - Reftest
<fantasai> -Mochitest
<fantasai> -UI Automation Frameworks
<fantasai> - Marionette
<krisk_> Mochitest - tests dom stuff
<krisk_> New UI automation framework - Marionette
<krisk_> Reftest drill down
<fantasai> "Reftest: style and layout visual comparison testing"
<fantasai> Reference: <p><b>This is bold</b></p>
<fantasai> Test: <p style="font-weight: bold">This is bold</p>
<fantasai> clint: The test and the reference create the same
rendering in different ways.
<fantasai> clint: Then we take screenshots and compare them pixel by
pixel
<fantasai> clint: Mochitest is an HTML file with some javascript in
it.
<fantasai> clint: One of the libraries it pulls in is the SimpleTest
library.
<fantasai> clint: It has the normal asserts: ok, is, stuff to
control whether asynchronous or not
<fantasai> clint: This other file here (in this example) turns off
the geolocation security prompts
<fantasai> clint shows a geolocation test
<jhammel> ^
http://mxr.mozilla.org/mozilla-central/source/dom/tests/mochites
t/geolocation/test_allowWatch.html
http://mxr.mozilla.org/mozilla-central/source/dom/tests/mochitest/geolocation/test_allowWatch.html
<fantasai> plh: How does this route around the security checks?
<fantasai> clint: uses an add-on
<fantasai> clint: has a special powers api
<fantasai> "Marionette: Driving Gecko into the future"
<fantasai> This is a mechanism we can use to drive any gecko-based
application either by UI or by inserting scrit actions into its
various script contexts.
<fantasai> How it works -
<fantasai> 1. socket opened from inside gecko
<fantasai> 2. Connect to socket from test harnes, either local ro
remote
<fantasai> 3. Send JSON protocol to it
<fantasai> 4. Translates JSON protocol into browser actions
<simonstewart> uses webdriver json protocol streamed over sockets
directly
<fantasai> 5. Send results back to harness in JSON
<jhammel> wiki page:
https://wiki.mozilla.org/Auto-tools/Projects/Marionette
https://wiki.mozilla.org/Auto-tools/Projects/Marionette
<jhammel> (WIP)
<fantasai> clint: We run all of these test on every check in every
tree we build on.
<fantasai> clint: Goes into a dashboard
<fantasai> slide: shows screenshot of TinderboxPushLog
<fantasai> wilhelm: Can we steal your Mochitests? What do we need to
do to do so?
<fantasai> clint: Check them out of the tree and see how well they
run in Opera
<fantasai> clint: Some of the stuff we did, e.g. special powers
extension,
<fantasai> clint: but it's now a specific API (used to be scattered
randomly throughout tests)
<fantasai> clint: If you had something similar and named it
specialpowers, then you could use that to get into your secure
system
<fantasai> clint: So should be possible.
<fantasai> clint: A lot of tests we have in the tree are completely
agnostic; don't do anything special at all, should work today
<jhammel> mochitests are at
http://hg.mozilla.org/mozilla-central/file/tip/testing/mochitest
http://hg.mozilla.org/mozilla-central/file/tip/testing/mochitest
<fantasai> wilhelm: Are there plans to release these tests to
geolocation wg?
<fantasai> clint: I think they already did. guy wrote tests is on
that wg
<fantasai> kk: ... they're hard-coded to use the Google service. If
you don't use it, they don't run...
<fantasai> kk: Not too many though
<fantasai> some discussion of sharing tests
<fantasai> Alan: I think WebKit is using some Mozilla reftests, but
not using them as reftests
<fantasai> kk: I'm fine w/ reftests. But of course won't work for
everything.
<fantasai> kk: CSS tests we wrote are self-describing.
<fantasai> Alan: do you have automation?
<fantasai> kk: Yes
<fantasai> rakesh: Do you run the tests every day?
<fantasai> clint: Every checkin
<fantasai> clint: Different trees run different numbers of tests.
<jhammel> https://tbpl.mozilla.org/
https://tbpl.mozilla.org/
<fantasai> clint: Our goal is to have test results back within 2
hours. Right now we're averaging 2.5hrs
<fantasai> fantasai: You're responsible for watching the tree and
backing out if you broke something.
<fantasai> discussion of test coverage
<fantasai> discussion of subsetting tests during development
<fantasai> wilhelm: How much noise do you have?
<fantasai> clint: Don't know about false positives
<fantasai> clint: Probably not many; once we find one, we check for
that pattern elsewhere
<jhammel> orange factor, for tracking failures:
http://brasstacks.mozilla.com/orangefactor/
http://brasstacks.mozilla.com/orangefactor/
<fantasai> clint: Thing we really have is intermittent failures
<fantasai> clint: We're trying really really hard to bring it down
<fantasai> clint: Used to be on every checkin you'd get, on average,
8 intermittent failures
<fantasai> clint: we pushed it down to 2
<fantasai> clint: And then we added the Android tests
<fantasai> clint: trying to bring it down again
<fantasai> duane: Can I instrument Marionette today in FF7?
<fantasai> clint: No, code we're depending on now is landing
currently on Nightly
<fantasai> clint: Released probably... May?
<fantasai> clint: Depending on work done by Developer Tools group
<fantasai> clint: They have a remote debugging protocol they're
implementing
<fantasai> clint: Will be really nice; decided this would be great
to piggyback on. Don't need two sockets in lower-level Gecko.
<fantasai> clint: So won't be available until that's released.
<fantasai> clint: Currently in a project repo... land in Nightly in
~2.5 weeks
<fantasai> plh: Marionnet is only for Fennec, not for desktop
version?
<fantasai> clint: For Fennec right now. Planning to go backwards and
use for Desktop as wel.
<fantasai> clint: My goal is to move all our infrastructure towards
that
<fantasai> kk asks about reducing orange
<fantasai> clint: It's mostly a one-by-one effort of fixing the
tests
<simonstewart> Interesting comment about avoiding using setTimeout
in tests
<fantasai> kk: Are you going to take Mochitests into W3C? Anything
preventing you?
<fantasai> clint: Nothing right now. We'd have to clean them up and
make them cross-browser. Good for everyone, not opposed, j ist a
matter of finding people and time
<fantasai> jgraham: there's a bug on making testharness.js look like
Mochitest to Mozilla
Testing Opera
<fantasai> "This looks vaguely familiar"
<fantasai> wilhelm: Say a few words about testing at Opera
<fantasai> wilhelm: We have a mainline, which is supposedly always
stable, and then when we're developing a feature, it gets branched
and at some point tests start passing (that's the yellow, b/c out of
sync with mainline) and then we merge and that becomes mainline
<fantasai> diagram shows mainline with six green dots going forward
<fantasai> branch goes off, two red dots, one yellow
<fantasai> arrow from mainline to green dot on feature branch
<ctalbert_> The wiki page we(mozilla) wrote that details our
"lessons learned" from fixing intermittently failing tests is here:
https://developer.mozilla.org/en/QA/Avoiding_intermittent_orange
s
https://developer.mozilla.org/en/QA/Avoiding_intermittent_oranges
<fantasai> arrow from green dot back to green dot on mainline
<fantasai> jgraham: ...
<fantasai> jgraham: Our setup's a bit different
<fantasai> jgraham: All the tests are in subversion in their own
repository that's separate from the code. It's just a normal
webserver: apach, php
<fantasai> jgraham: When you ask for tests to be run, they get
assigned from the server and we send them out to a couple hundred
virtual machines
<fantasai> jgraham: not quite MSFT's setup
<fantasai> jgraham: And then we store every result of every test
<fantasai> jgraham: I think you just store did all the tests past..
we store, in this build this test passed.
<fantasai> jgraham: We have a huge database of this information
<fantasai> jgraham: Theoretically we can delete stuff, but we store
everything.
<fantasai> jgraham: In a mainline build from yesterday, we ran
quarter of a million tests
<fantasai> jgraham: That's not quarter million files -- it's 60,000
files, some of which produce multiple results
<fantasai> jgraham: e.g. some tests from HTML5 test in W3C, one file
might produce 10,000 results
<fantasai> jgraham: Typically it's a JS thing and it just runs a
bunch of code and at the end it has some results
<fantasai> jgraham: Dumps them to the browser in some way
<fantasai> jgraham: The way we do that right now is pretty stupid,
so I won't talk about it
<fantasai> slide: Visual tests, JS tests, Unit tests, Watir tests,
Manual tests :(
<fantasai> jgraham: System was designed 7 years ago or sth
<fantasai> jgraham: For visual tests, you just take a screenshot,
and then we store the screenshot.
<fantasai> jgraham: Someone manually marks whether that screenshot
was a pass or fail.
<fantasai> jgraham: Don't do that. You have to do it once per test,
and then once any time anything changes very slightly
<fantasai> jgraham: e.g. introduce anti-aliasing test, have to
re-annotate all tests
<fantasai> jgraham: this format is deprecated
<fantasai> wilhelm: We have 20,000 tests on 3 different Opera
configurations...
<fantasai> wilhelm: We want to kill these tests and use reftests
instead
<fantasai> jgraham: Oh, reftests should be on that list too
<fantasai> jgraham: Recently we implemented reftests, and we're
actively trying to move tests to reftests.
<fantasai> jgraham: You can't test everything with reftest, but when
you can it's much better
<fantasai> Alan: Do you keep track of when the reference file bitmap
changes?
<fantasai> Alan: What if both the reference and the test change
identically such that the test should fail but doesn't?
<fantasai> plinss: In the case of the CSSWG when we have a fragile
reference, we have multiple references that use different techniques
<fantasai> jgraham: We have a very lightweight framework we used to
use for JS tests. Only allowed one test per page.
<fantasai> jgraham: Easy to use, but required a lot of convoluted
logic for each pass/fail result.
<fantasai> jgraham: For new test suites, we're using testharness.js
<fantasai> jgraham: similar to Mozilla's MochiKit
<fantasai> jgraham: Unit tests are C++ level things not worth
talking about here
<fantasai> jgraham: When things need automation, we use Watir --
discussed this morning
<fantasai> jgraham: When all else fails, we have manual tests
<fantasai> wilhelm: Notice that the monkey looks really unhappy
<fantasai> jgraham: For the core of Opera, we schedule a test day
and just run tests
<fantasai> plh: How many manually tests do you have?
<fantasai> wilhelm: around 2000 before, less now...
<fantasai> wilhelm: Probably spend about a man-year on manual tests
per year
<fantasai> wilhelm: Say some things about challenges we have, things
we need to take into account when writing tests internally and for
W3C
<fantasai> wilhelm: First thing is device independence
<fantasai> wilhelm: We run 3 different configurations of Opera:
Desktop profile, Smartphone profile, and TV profile
<fantasai> wilhelm: Almost every time someone requests a build, it
will be tested on those three profiles
<fantasai> wilhelm: We notice that if you have a static timeout in
your test, e.g. wait 2s before checking result, that will break on
stupid profile with low resources
<fantasai> wilhelm: On some platforms we automatically double or
triple it, and we hope it works, but it's not really good solution
<fantasai> jgraham: How do you deal with ... ?
<fantasai> clint: we time out our tests after a set time period and
mark it as failed
<fantasai> jgraham: Most assumption is don't depend on device size
or speed -- test will randomly fail.
<fantasai> wilhelm: Brings me to the next problem: random
<fantasai> wilhelm: If you have so many tests and even small
percentage fail randomly, going to spend man-years investigating
those failures
<fantasai> wilhelm: When we add new configurations, when we steal
tests from source of unknown quality, we spend many man-years
stamping out randomness in the tests
<fantasai> wilhelm: The more complex the test, the more likely to
randomly fail
<fantasai> wilhelm: Simplest tests are JS.
<fantasai> wilhelm: For imported tests from random sources, could be
very bad
<fantasai> wilhelm: Then comes visual tests
<fantasai> wilhelm: Sometimes complexity is needed, but if can
simplify will do that
<fantasai> wilhelm: We have a quarantine system: run 200 times on
test machines first to make sure its good
<fantasai> wilhelm: Still, sometimes things slip through.
<fantasai> wilhelm: We steal your tests. Thank you.
<fantasai> slide: jQuery, Opera, Chrome, Microsoft, mozilla, W3C
<fantasai> wilhelm: Keeping in sync with the origin of the test is
difficult
<fantasai> wilhelm: When someone updates a test elsewhere, w don't
automatically get that
<fantasai> wilhelm: When we muck about the test to get it to work on
our system, we have to maintain patches
<fantasai> wilhelm: If we fix bad tests, sometimes easy to
contribute back, but sometime not
<fantasai> wilhelm: Automating tests to use our Watir scripts, can
also become a problem.
<fantasai> wilhelm: Our current approach is not usable
<fantasai> wilhelm: need a better way for us all to keep in sycn
<fantasai> kk: This is why we have submitted and approved folders
<fantasai> jgraham: The problem from our POV is really .. part of it
is version control problem on our
<fantasai> end
<fantasai> jgraham: Don't have a good way to keep our patches
separate from upstream changes
<fantasai> jgraham: If we have w3C tests, and we pull new version,
don't have a way to say "these are bits we changed ot make it work
on our version"
<fantasai> jgraham: ... reporting and script file separate
<fantasai> jgraham: if we pull some tests from Mozilla, say, and
they're JS engine tests andthey update them, if we try and merge
them.. someone has to work out how to do that by hand. It's kindof a
nightmare.
<fantasai> wilhelm: Last thing about randomness, esp imported
<fantasai> wilhelm: Some tests rely on external tests.
<fantasai> wilhelm: Great when we only had a few tests
<fantasai> wilhelm: But now it's a problem. Servers go down, etc.
<fantasai> wilhelm: Conclusion there is: don't do that. :)
<fantasai> wilhelm: That's it!
<fantasai> jhammel: Wrt upstream tests, standardizing on formats and
standardizing on process
<fantasai> wilhelm: We set up time at 3:15 today to discuss this
exact issue
<fantasai> mc: You say you have to fix tests to work on your
product.
<fantasai> mc: Question is how do you separate fixing test to be not
random, vs. making them work on a particular product
<fantasai> jgraham: When we pull in tests, we try not to change
anything to do with the test.
<fantasai> jgraham: We don't require the tests to pass to be in our
system.
<fantasai> jgraham: The thing we need to change is, can this test
report back to our servers.
<fantasai> jgraham: But external tests are usually not designed that
way.
<fantasai> wilhelm: I think testharness.js approach is good, because
those are separated.
<krisk_> That is the end of Opera
<MichaelC_SJC> 's presentation
<krisk_> The next person up is peter from HP on css wg update (10
minutes)
<krisk_> Then a discussion on rendering tests for about 1 hour
Testing in the CSS WG
<krisk_> test.csswg.org
<krisk_> has lots of information on CSS WG testing
<krisk_> Tests are 'built' from xml into multiple formats - html,
xhtml, etc...
<krisk_> Test harness is a wrapper around the tests that are loaded
in an iframe
<krisk_> It loads the tests that have the least number of tests
<krisk_> The harness has a filter for spec section, etc..
<krisk_> The harness has meta-data description for each of the tests
<stearns> test format requirements:
http://wiki.csswg.org/test/css2.1/format
http://wiki.csswg.org/test/css2.1/format
<krisk_> The harness also has test results that can be shown for
each of the browser/engine versions
<krisk_> Build process has requirements that will be improved
overtime - meta data, ref test, title, etc...
<krisk_> Adding meta-data helps review process, though most
submitters don't like to add this data
<krisk_> Multiple refs for the same test exist and a negative ref
test as well
<krisk_> You can have two ref tests if the spec has two different
results - for example margin collapsing
<krisk_> If a ref test can't be used then in some cases a
self-describing test works
<plinss> http://test.csswg.org/annotations/css21/
http://test.csswg.org/annotations/css21/
<krisk_> Spec annotations are used that map back to the annotated
spec
<krisk_> The annotated spec has total tests and results for each
section of the spec
<krisk_> Now on to the test review system
<krisk_> http://test.csswg.org/shephard/
http://test.csswg.org/shephard/
<krisk_> Very tight coupling to the css test metadata
<krisk_> Tracks history and other information about a test case
<krisk_> jgraham: is this tied to the test file?
<krisk_> peter: no it's possible to have this information in another
file
<krisk_> jgraham: can this handle a case when multiple files are
used to create alot of tests
<krisk_> peter: yes we have the same issue for the media query test
cases
<krisk_> Wilhelm: So does css still use visual non-ref tests?
<krisk_> fantasi: for css3 we require ref-tests, so no
<Alexia> http://b39b5112.thesegalleries.com
http://b39b5112.thesegalleries.com/
<plh> s|http://b39b5112.thesegalleries.com||
http://b39b5112.thesegalleries.com/
<krisk_> peter: The system is built to save time and automate parts
<krisk_> peter: for example when a test is approved it is moved from
submitted to approved
<krisk_> Michael: Does the system have access control checks for
approval?
<krisk_> peter: yes
Testing Chrome
<krisk_> Ken: Chrome Testing Information
<simonstewart> kk: works on the chrome automation team
<simonstewart> kk: not an automation group in the same sense as
mozilla
<simonstewart> chrome depends on webkit
<krisk_> kk is not krisk
<simonstewart> webkit layout tests, pixel-based tests
<simonstewart> kk == ken_kania
<simonstewart> kk: dom dump tree tests
<simonstewart> kk: not got a lot of insight into the specifics of
the webkit tests. Focuses mainly on the chrome browser
<simonstewart> kk: couple of layers of testing
<simonstewart> kk: lowest layer is the c++ browser tests
<simonstewart> kk: probably more than other browsers do. Special
builds of chrome which will run C++ in the ui thread
<simonstewart> kk: relatively low level, though
<simonstewart> kk: beyond those, there are the ui test framework.
Based on the automation proxy (AP)
<simonstewart> kk: ap is pretty old, but is an ipc mechanism
<simonstewart> kk: very much internal facing
<simonstewart> those tests are still fairly low level, depsite being
called ui tests
<simonstewart> kk: higher than that, Ken's team work on something
called the chrome bot
<simonstewart> kk: runs on real and virtual machines
<simonstewart> kk: cache of a large number of sites in a cache.
Often used for crash testing. Also include tests that perform random
ui actions
<simonstewart> kk: a little bit smarter than pure random, but that's
the gist
<simonstewart> kk: qa level tests. Tests that are done by manual
testers. Piggy back off the ui test automation framework. things
ilke creating bookmarks, installing extensions, etc
<simonstewart> kk: break down manual testing to test parts. First
app compat. Push a new release of chrome it continues to work, and
testing chrome at the ui level
<simonstewart> Most of the ui is "based on the web"
<simonstewart> For the chrome specific native widgets there are
manual tests
<simonstewart> kk: app compat depends on webdriver
<simonstewart> kk: lots of google teams depend on webdriver to
verify that sites work.
<simonstewart> kk: guess that at a high level, the testing strategy
tends to be developer focused.
<simonstewart> kk: devs should write the tests in whatever tool and
harness is most expedient for their purpose
<simonstewart> kk: piggy back a lot on the fact that chrome does
rapid releases. 4 channels release to users (canary, dev, beta,
stable)
<simonstewart> kk: different release schedules
<simonstewart> kk: depend a lot on user feedback from the canaries
<simonstewart> kk: that's the gist of it
<simonstewart> tab: sounds good to me
<simonstewart> jhammel: do chrome do performance testing?
<simonstewart> kk: we do. Using the AP and the ui testing framework
mentioned earlier
<simonstewart> http://build.chrome.org
http://build.chrome.org/
<simonstewart> to view the tests that have been run
<simonstewart> plh: do we run jquery tests
<jhammel> ^ correction: http://build.chromium.org
http://build.chromium.org/
<simonstewart> kk: not really. webkit guys might, and we pick that
up
<simonstewart> krisk_: do you create tests and feed them back
<simonstewart> TabAtkins: we don't do much, but we do
<simonstewart> krisk_: is that because it doesn't fit with the
systems
<simonstewart> TabAtkins: the ways we write and run tests isn't
really compatible with the existing w3 systems.
<simonstewart> TabAtkins: would like to change that!
<simonstewart> TabAtkins: some tests are html/js. which might be
used where possible. Doesn't ahppen that regularly
<simonstewart> krisk_: how do you know that you're interoperable?
<simonstewart> TabAtkins: in terms of webkit stuff, it's a case of
testing being done by different browser vendors
<simonstewart> kk: lots of c++ tests that are specific to chrome
<jhammel> simonstewart: np :)
<simonstewart> krisk_: v8?
<simonstewart> TabAtkins + kk: v8 team live in europe. Who knows?
<simonstewart> wilhelm: also has legacy stuff for opera. New tests
written in a way that (in theory) is usable outside. Can chrome do
the same thing?
<simonstewart> TabAtkins: will agitate for that. Involved in spec
writing rather than active dev, so might be tricky
<simonstewart> wilhelm: This is a great forum to raise those issues.
Opera happy to share with Chrome if Chrome does the same :)
<simonstewart> krisk_: do chrome try and pass a bunch of the w3c
test suites?
<simonstewart> TabAtkins: yes. Some of the might be integrated into
the chromium waterfall. Some of them might be run by hand
<simonstewart> ?? does anyone know about webkit testing
<simonstewart> TabAtkins: the people who'd I'd like to ask aren't
around
<simonstewart> webkit does seem to take in test suites from mozilla.
They're running against a bitmap that's different from the moz
rendering
<simonstewart> TabAtkins: we don't have a good infrastrcuture for
ref tests
<simonstewart> TabAtkins: the test infrastructure people _do_ want
to fix that
<simonstewart> TabAtkins: every time a new port is added to webkit,
there are more pixel tests. Provides pressure to do better
<simonstewart> plh: any other questions?
<simonstewart> 15 minute break coming up
Info available from webkit: https://trac.webkit.org/wiki
https://trac.webkit.org/wiki
also see http://www.webkit.org/quality/testing.html
http://www.webkit.org/quality/testing.html
<krisk_> Next agenda Item jgraham talking about testharness.js
<MichaelC_SJC> scribe: testharness.js
<MichaelC_SJC> scribe: krisk_
krisk_
testharness.js
<MichaelC_SJC> s/topic: krisk_//
<fantasai> scribenick: fantasai
jgraham: testharness.js is something I wrote to run tests.
... It runs JS tests specifically
... It's a bit like MochiTest or QUnit which JQuery uses, or various
things
<plh> --> http://w3c-test.org/resources/testharness.js
testharness.js
http://w3c-test.org/resources/testharness.js
jgraham: Every JS framework has invented its own testharness
... This has slightly different design goals
... The overarching goal is that it's something we can use to test
low-level specs like HTML and DOM
... So it can't rely on lots of HTML and DOM :)
... The design goals were to provide some API for writing readable
and consistent tests
in JS
jgraham: Our previous harness at Opera, as I mentioned, didn't resul
in very readable
tests
jgraham: The other is to support testing the entire DOM level of
behavior
... There are 2 test types : asynchronous tests and synchronous
tests
... second us purely syntactic sugar
... Another design goal was to allow possibility of the test to have
multiple assertions, and all have to be true for test to pass
... typical example might be checking that some node has a set of
children.
... Might want to first test for any children before testing that
4th child is a <p>
... Multiple tests per file was a requirement; learning from Opera's
1/file, which was painful for test writers and discouraged many
tests
... ... runs everything in try-catch blocks
... One feature of that is that every bit of the test is like a
function, basically
... it tries to handle some housekeeping.
... if you have 1000 tests in a file, nice if you can time out those
tests individually
... Uses settimeout(); can override that if you want, e.g. if
running on slow hardware
... and a design goal was easy integration with browsers' existing
test systems
... Should be easy to use on top of MochiKit or whatever you use for
reporting results
... next thin I thought I'd do is go through creating a test.
jgraham's text editor:
<script src="resources/testharnessreport.js"></script.
<script src="resources/testharness.js"><script>
<div id="log"></div>
jgraham: By default testharnessreport.js is blank. It's for you to
integrate into your testing system.
... the order is not at the moment relevant
... we might later check in testharness.js that testharnessreport.js
was included
added to file:
(at the top)
<title> Dispatching custom events</title>
(at the bottom)
<script>
var t = async_test("Custom event dispatch");
</script>
jgraham: Each test has a number of tests, and each step is a
function that gets called
... It gets called inside a try-catch block, and we can check if the
test failed. We don't put anything as top-level code.
(added at the bottom)
t.ste(function() {
(ok, that's too much to type)
jgraham: Here it's adding an event listner before the second step
... When it gets called, it'll cal lthis other function here, which
will run this other step, which is another function. Can get a bit
verbose.
... There's a convenience method that will make this easier.. all
documented in testharness.js
... Simple assert_equals() with value we get, value we expect, and
then you can optionally have a string that describes what it is
you're asserting.
... At this point everything we want done is done, so we say
t.done();
... If you load this in a browser, because we have div#log, it will
show whether it passes or fails and what assert failed
<plh> -->
http://w3c-test.org/webapps/ElementTraversal/tests/submissions/W
3C/Element-childElementCount.html Example of testharness.js
http://w3c-test.org/webapps/ElementTraversal/tests/submissions/W3C/Element-childElementCount.html
jgraham: That's all
jj: Is there an id on the steps, so that you can say you failed step
4 of test foo?
jgraham: If there's demand, there could be a second argument there.
jj: would be nice to know where it failed so I can set a breakpoint
there
jgraham: If you get a huge number of tests per file, it's usually
auto-generated
... if it's failing in an assert, then it'll tell you which assert
failed
plh shows his example
plh: everything shown here is generated by testharness.js
jgraham: There's a failure in this, and it seems everyone fails
that.
plh: Bug in testharness.js
jj: Easiest way to debug the test. Is there an error in the test,
error in testharness.js, or error in browsers
jgraham: There are various types of assertions. Usually corresponds
to webIDL
... But what's in webIDL isn't always the same
kk: It's pretty well-written, only 700 lines or so
clint: If it's synchronous, you don't have to do t.step()
jgraham: A test that is synchronous implicitly creates a step
wilhelm: Opera currently uses this tool for all the new tests that
we write. Can others use this?
clint: Yeah, I think so
kk: There use to be some nunit or something that W3C had
... Was in IE, but some browsers couldn't run it.
... Very complicated
[server problems]
plinss: Are tests grouped by section into files?
jgraham: In this case, it checks reflection section, plus section of
each part of the spec that defines a reflected attribute
topic change
wilhelm: plh wanted to talk about test harness, fantasai wanted to
talk about syncing problem
How should we organize public test suites so that they are as easy as
possible to contribute to and reuse?
htp: //w3c-test.org/framework/
MikeSmith: This is an instance of the framework peter demoed
Mike: I'm going to show you what has been added here to make it
easier for test suite maintainers to add data to the system.
... There's this area called Maintianer Login
... It'll give you an http_auth, which authenticates against W3C's
user database
... Email me if you want access to the system
... Once you go in there you'll see 2 options: add metadata, change
metadata
... Can add a specification
... one early piece of feedback I got was they have tests they want
to run that are not associated with a spec.
... So in this instance of the system, it's not a requirement to
have a spec for your test suite
... You can give it an arbitrary ID as long as not a duplicat
... Title of the spec
... URL for the spec
... It expects you'll point it to a single-page version of the spec
... If you have a multi-page spec, don't point it at the TOC. You
need the full version of the spec.
... Could change later, but initially set up this way 'cuz easier
... This will get added to the list here
... Next thing you can do is needed if you want to do what Peter was
demoing earlier, which was associating testcases with specific
sections of the spec -- or specific IDs in the spec
... Structured around idea that you put your IDs per section
... But some WGs like WOFF WG they're putting assertions at the
sentence level
... They don't actually have section titles, so needed to
accommodate that too
Peter: Alan and fantasai did some work on that, too.
... Shepherd tool will be able to parse out spec to find test
anchors
... and then can report testing coverage of the spec, so this is
something we will automate
Alan: What fantasai and I worked out was based on WOFF work, but
will be simpler for spec editors. A bit harder to automate, though
Mike: This part add spec metadata.
... Instead of a form to fill out, it lists existing specs in the
system
... once you go here, if there's already data in the system, will
show you data in the system alread
... otherwise it'll show you generated data
... This parses the spec and pulls out the headings. If it looks ok,
you press submit
... It'll put these section titles into the database.
... If you have IDs below the section title level, then you'll have
to use a different way to get it into the DB
... You might have to get me to do it for now :)
... Those steps are optional right now.
... What is necessary is going in and giving info about the test
suite itself.
... you can give it an arbitrary ID
... Title, longer description
... to explain better thet est suite
... base URL of where your test suites are stored
... Difference from CSS is, that one requires format subdirectories
plinss: it's optional
Mike: This one doesn't expect subdirectories. Expects all tests in
this one directory
... If you have separate subdirectories...
... Need to make different test suites or ...
... Simplest case you have all tests in one directory
plinss: The code's actually a lot more flexible wrt formats. We'll
talk offline.
MikeSmith: Then you have contact information for someone who can
answer questions about test suites
... Then you indicate format of the test suite
... Then you have a list of flags, you can select which ones
indicate optional tests
... There are ways to add flags to the system
... No ui for it, so contact me
... Last thing you then do is upload a manifest file
... You have to have a test suite
... You select a test suite
... and then what I have it do right now is that you need to point
it to the url for a manifest file, and it'll grab that and read it
in
... Right now two forms of manifest files that it will recognize
... second one here is just a TSV that expects path/filename,
references, flags, links, assertions
... links are the spec links
... The other big change is, I was talking with some people e.g.
annevk and ms2ger
... the format they're using is just listing the filenames
... it marks support files as support files
kk: Mozilla guys wanted to know what files were needed to pull to
run a test case
plinss: In the CSSWG, the large manifest file with metadata -- that
gets built by the build system
MikeSmith: This form expects the full filename, not just the
extensionless filename
... Because that's what they had
... Once you have that, you should be able to get your test cases
into the test database
... and it'll show up on the welcome page
... Long way to go on this.
... Goal when I started on this was to get it to the point where I
didn't have to manually do INSERT in SQL to get specs into the
database
... What would be really nice is if ppl start using this and getting
more test suites in there so that we can ..
plinss: But right now only limited set of ppl can contribute to that
code
MikeSmith: I created two groups in our database
... I created a group for developers -- anyone who wants to
contribute to framework
... That'll give you write access to hg repo for the source code for
this
... Take a look at source code and see problems, send me patches or
I'll give you direct access
... Second thing is if you want to have access to use this UI to
submit test suite data, I'll have to add you to a particular group
fantasai: how is this code related to plinss's code?
MikeSmith: It's forked from that.
... I've just been pulling the upstream changes
... been able to merge everything without it breaking.
... Think it's in good enough shape that we could port it back
upstream
plinss: This system and the Shepherd share a lot fo the same base
code
... Lots of things I was going to port Shepherd system back into
this system, and then pull your stuff in too
... Mike also has code that ties into the testharness.js code, and
will automatically submit results from that
MikeSmith: If you go to enter data, it gives you some choices about
whether you want to run full test suite or not
... There's a button here that will pull automatic results where
possible
... Be careful, this will submit the data publicly!
jgraham: Not saying it's a bad idea, but from our POV, we're not
going to use it offline.
(Brian was talking about trying out the system privately offline)
plinss: The system tracks who's submitting the data. By login if
you're logged in, by IP if not
Brian: Privacy is useful
plinss: goal is for pulling data from as may sources as possible
wilhelm: fantasai wanted to talk about keeping things in sync
<dobrien> Is someone scribing? I can't keep up on the iPad
<ctalbert_> This is the writeup that we are planning to set up at
Mozilla for the CSS tests specifically:
https://wiki.mozilla.org/Auto-tools/Projects/W3C_CSS_Test_Mirror
ing
https://wiki.mozilla.org/Auto-tools/Projects/W3C_CSS_Test_Mirroring
<krisk_> Mozilla has a way to move tests from mozilla -> w3c ->
mozilla
<ctalbert_> wilhelm: how will this cope with local patches?
<krisk_> fantasi: The master copy only lives in one place...
<ctalbert_> jgraham: probably not a problem with the css tests
<krisk_> fantasi: approved is the master in w3c
<krisk_> fantasi: submitted is the master for submissions
<ctalbert_> jgraham: opera is thinking of having the master from w3c
which is intact, and our checkout from that master will have the
local patches, and when we pull we'll rebase our patches atop the
w3c master
<ctalbert_> this should be possible now that hg is in the w3c side
and our (opera) side
<ctalbert_> fantasai: we'll probably have to do something similar
<krisk_> wilhelm: how does this handle local patches?
<ctalbert_> jhammel: is there a technical limitation to not have
people editing the w3c tests
<ctalbert_> fantasai: no
<krisk_> fantasi: this is only for css which don't seem to have this
problem
<ctalbert_> jgraham: probably make it a commit hook
<ctalbert_> ctalbert_: agreed
<ctalbert_> peter: if someone pushes to the approved directory
without actually being approved then the system just automatically
denies them
<ctalbert_> that may be incorrect ^ (scribe error)
<ctalbert_> wilhelm: might be an idea to split test suites down at
lower granularity levels so that you can have test suites with
differnt levels of maturity
<ctalbert_> jgraham: don't think that would make a difference tbh
<ctalbert_> peter: our repo would keep all the data from all the
suites in the repo so that our build system could build any version
of them from any suite
<ctalbert_> wilhelm: are there other things we can do to make it
easier to contribute test suites?
<ctalbert_> fantasai: one problem on the mozilla side - there's no
place to put tests that should go to the w3c - we depend on a manual
process to sort out which should be submitted and then it is done
later
<ctalbert_> fantasai: these tests just sit in a random place and are
forgotten
<ctalbert_> fantasai: once we have a directory that goes to w3c and
we tell the reviewers, then it will help quite a bit.
<ctalbert_> fantasai: the basic idea is to make the process obvious
what developers need to do with that test to indicate that it is
appropriate and ready for w3c then it should "just happen"
<ctalbert_> jgraham: we have a similar problem. it's hard to surface
those tests and bugfixes without a policy and a place for those
tests
<ctalbert_> peter: if we have a standard format among the test
writers then it will be easier to help developers to upload the
tests to the w3c. If the developers have to convert the tests it's
too difficult and people won't expend the effort to make it happen
<ctalbert_> krisk_: sometimes it depends on the editors as to when
they allow tests into the spec, and you find that tests sometimes
lag the spec by quite a bit
<ctalbert_> fantasai: we found that with the css - the person
writing the spec is often nominally tasked with also writing the
test suite but because the skill sets are different and the spec
editor is usually swamped, then the tests get neglected
<ctalbert_> fantasai: we really need a dedicated person to manage
these tests and testing effort for each spec
<ctalbert_> MikeSmith: is there some way to motivate people to do
that?
<ctalbert_> MikeSmith: maybe we should publicly track the testsuite
owner?
<ctalbert_> fantasai: we can do that, but the burden is on getting
resources for that, really.
<ctalbert_> MikeSmith: yeah, the question is how do you encourage
the managers allow their people to spend times on w3c work
<ctalbert_> MichaelC_SJC: you might be able to convince your company
to do that, but we also need to have the working group chairs
understand that this needs to happen
<ctalbert_> jgraham: if we have them already in an interoperable
format then it's pretty easy, but for our existing tests that are in
a different format, we aren't going to spend the time to convert
them
<ctalbert_> fantasai: we might just have a place at w3c to take
those tests, and just post them publicly and have someone else do
the conversion work
<ctalbert_> jgraham: I suspect that's a wide problem
<ctalbert_> krisk_: if you getin the habit of submitting stuff as
you're doing development, tat seems reasonable.
<ctalbert_> krisk_: keeping things not super complex is a win, and
being consistent will pay dividends
fantasai^: Because for Opera it may not be valuable to do the
conversion, but e.g. Microsoft might want those tests, and decide
that the cost of converting is less than the cost of rewriting tests
from scratch, so to them it'll be worth it to do the conversion
<ctalbert_> fantasai: thanks, I'm not too good at this :/
<ctalbert_> (scribe note ^)
<ctalbert_> wilhelm: the more I think of this, the more I realize
that facilitating the handover of tests is a full time job
<Zakim> MichaelC_SJC, you wanted to ask how much should there be a
"W3C format" vs how much does W3C framework need to format (nearly)
any format?
<ctalbert_> wilhelm: if we could get every browser vendor to commit
one person to do this work on their team then that would be good.
<ctalbert_> fantasai: the problem we're at now, people havne't
adopted the w3c ofrmats internally
<ctalbert_> it will be less work once that happens
<ctalbert_> it's not w3c's responsibility to convert your tests to
w3c
<ctalbert_> fantasai: you can write a conversion script to convert
your test to w3c format
<ctalbert_> better to do that than to have w3c to accept all
differnt formats
<ctalbert_> jgraham: the problem is that many of these harnesses are
not built for portability
<ctalbert_> MichaelC_SJC: the problem with a common format (and I
may be wrong) is that you run into things you can't test
<ctalbert_> jgraham: if we run into that, then in that case maybe we
can find some lightweight format for those tests, or in that case
maybe we use a different type of harness
<ctalbert_> scribe: ctalbert has to step out
<ctalbert_> fantasai: ^
scribe:
kk: If you can write it with testharness.js, do that. If not, try
reftest, if not, try self-describing test
... In your case you have the difficulty of needing a screenreader
or something
...
jgraham: If you can get ppl to contribute in one format, at least
you solve the problem once per platform rather than once per test
mc: I think there's a hierarchy of goodness
... The framework should have at least thepossibility of hooking in
new formats
general agreement
wilhelm: For the Watir cases, we noticed areas where we'd want to
addtests for something very obscure and specific. What we've done is
add support at a low level in Opera and use an API
... Such things could be later added to WebDriver
<MichaelC_SJC> s/I think there's/I can agree with the idea that/
Alan: For tests where there isn't a w3c version, but browsers have
something, is there a list of most-wanted specs that need tests on
the w3c site
fantasai: All of them? :)
Alan: We were talking about poking ppl, committing ppl to
translating browser tests to w3c tests
... Would be more successful to getting resources if we have a
specific list of things we need
jj: Also possibility to ask specific people.
... Rather than saying, please call all submit tests for HTml5
... Say, can you submit tests for WebWorkers
... need a specific ask to get things done
... It might not cause immediate surge in test submissions, but for
me from outside to inside, the idea of submitting tests was
impossible to me. Didn't know where to submit them, figured they'd
be rejected, didn't know what a reftest was, etc.
... So process was hard, and not being specific
... Better way to get things done is asking
... Would like Opera to submit WebWorker tests
wilhelm: Can I get that in writing so I can show it to my manager?
Alan: Identify the tests, see who has those tests, then request them
plh: We've been corrsponding on testing framework a little bit, but
part of task is also going out there in the wild and finding tests
and getting them to W3C
... Need to get to point where we have framework and start on asking
tests
Alan: Use framework to identify areas, since it annotates the spec
jj: We have no idea how much coverage those 47 tests have -- number
isn't meaningful from a coverage perspctive
... 1 is better than 0, but maybe 100 is needed not47
?: Test coverage is a negative covered only know when something is
not covered, not how well something is covered
jj: Even if you say you have 100% on that normative statement, still
doesn't tell you if you got all the edge cases
jgraham: At the moment for HTML we have nothing, though.
<simonstewart> ^^ simonstewart: test coverage is a negative thing.
It'll only say what's not covered, not how well the covered areas
are tested
jgraham: We have our tests organized by section in the repo, but
it's not explicit
... Being able to say per normative statement, do we have a test for
this, is pretty nice
<plh> --> http://www.w3.org/2011/10/timer.html (annoying) timer
http://www.w3.org/2011/10/timer.html
jgraham: If you look somewhere, there's an annotation per sentence
in the spec showing tests for section X
... But that's really complicated, because spec isn't marked up to
make that easy
... and testing dozens of disconnected statements
kk: The problem we're struggling with is not that how do we get
perfect coverage. There's a spec, and there's no coverage.
... Browsers all have this feature, and they don't work the same. So
having some is a good start.
Bryan: If you look at most of WebAPIs near LC or at LC, only 1/3
have tests available
<jhammel> fantasai: setup a process for getting tests from *your*
organization to w3c, and *going forward*, you should write
w3c-submittable tests *and* submit the tests. Once that is in place,
we can go back and convert legacy tests
<plh> s/corrsponding/working/
<jhammel> fantasai: we need to get the webkit people to commit to
this
<jhammel> fantasai: you can require that when checked into repo,
they become reftests
<jhammel> fantasai: plan going forward is to convert to reftest
<jhammel> jgraham: if you're comparing to something bitmap-based, it
may take 2x time, but it will save time going forward
fantasai^: Because then the number of legacy tests that are not
w3c-formatted stops growing, and we can work on making that number
smaller
Additional Items
example of a test that has to be self-describing: This tests that
the blurring algorithm produces results within 5% of a Gaussian blur
http://test.csswg.org/source/contributors/mozilla/submitted/css3
-background/box-shadow/box-shadow-blur-definition-001.xht
http://test.csswg.org/source/contributors/mozilla/submitted/css3-background/box-shadow/box-shadow-blur-definition-001.xht
bryan: We developed a number of specs for device APIs
... We recognize these APIs are quite sophisticated, an it'll take
some time, but we're continuing the development of these
capabilities for web runtimes
... We have developer program, global ... ecosystem
bryan (from AT&T): wanted very briefly ...
bryan: show you these links to the specs, the APIs, but more
importantly the test framework
... Test framework is based on QUnit
... Pulls in a file from a test directory, which has the list of
test associated with this particular API.
... Tests individual JS filesin the same directory
... will run them one by one
... This is packaged up as a widget file, whcih is available for
download
... So we can run all the tests for example using this widget
framework.
bryan shows pie charts of resutls
bryan: Automatically uploaded and made available to vendor
plh: Say 1000 tests for core web standards?
bryan: No for APIs
... What comes for underlying platform is inherently tested by that
community
... We need to cover device variation
... identify things that we reference
... We have individual tests for these, test scripts
... this is more than acid level test, but not what we hope to see
from W3C in long run
... We don't want to develop and maintain this level of detail in
WAC. Want to leverage W3C test suites
... If you look at the tests, you can see for example the
geolocation test suite, which we reference.
... We want to auto-generate the tests as widget
jj: So if hte test suite changes, do you update your widget?
bryan: Our goal is to create frameworks where we can pull in tests
and run them in this runtime environment without havng to
necessarily maintain the tests ourselves
... We would benefit from a common test framework
... What exactly these tests are is basically just a JS procedure
... We test existence of methods, call qunit functions for
pass/fail, not necessarily married to this format, but it was the
most common one at the time we developed this.
... So to summarize our goal is to have the scalability to support
this widget-based ecosystem across dozens of devices across the
world
... So we have to have scalability
... To depend on the core standards as something we don't spend a
lot of effort on
... Duplicate things that eventually come from W3C.
... We'd like to see this developed at W3C so we can directly
leverage it.
fantasai comments on how this shows having a few common formats is
better than having w3c accept many similarly-capable formats -- it
better supports reuse of the tests
Conclusions and Action Items
1. Vendors commit to running W3C tests
2. Vendors push internally to adopt W3C test formats
plh says W3C should make ti easier for vendors to import suites
fantasai: what does that entail?
plh: make guidelines for WG
jgraham: I feel the problem is more on our side than on W3C side
wilhelm, jgraham: but of course, using hg instead of cvs is
important for tests
wilhelm: W3C should commit resources to get tests from vendors
plh: start with webapps
wilhelm: Any conclusions on WebDriver discussion?
... We commit to work on the spec, and get that into our browser
plh: MS and Apple should look into that
Mike: normal people at apple are interested, but they're not the
ones who sign off on things
kk: Using testharness.js seems to me a very low-hanging fruit,
rather than writing a whole bunch of APIs
<jhammel> "not buy Apple" would be more effective
wilhelm: There should be a spec that talks about it, for the IP
stuff, we need to get a spec out so there's less risk for those
implementing
jgraham: There was some discussion, but no decision, about which
bindings W3C would accept tests in
wilhelm: I'd list that as an open issue
MikeSmith: We want to follow up with testing IG , [other grou]
s/grou/group/
MikeSmith: Spec discussion would go to [... mailing list ...]
wilhelm: Dumping ground for non-W3C-format tests
kk: You can put whatever you want in submitted folder
<MikeSmith> public-browser-tools-testing@w3.org
jgraham: It would be nice if ppl dump random test suites in random
formats, to separate those out from thing sthat would be approved in
roughly their current form
<MikeSmith>
http://lists.w3.org/Archives/Public/public-browser-tools-testing
/
http://lists.w3.org/Archives/Public/public-browser-tools-testing/
kk: We should have an old_stuff directory
jgraham: And encourage people to dump stuff there
<MikeSmith> for the Testing IG,
http://lists.w3.org/Archives/Public/public-test-infra/ and
public-test-infra@w3.org
http://lists.w3.org/Archives/Public/public-test-infra/
plh: We can associate a repo with the testing IG, and then anyone in
that IG can push to the repo
<plh> ACTION: Mike to create mercurial repositories for Web Testing
IG and Browser Tools WG [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action01]
fantasai: Should be clear that dumping things here is not the same
as submitting to an official W3C test suite
bryan: Should also have a wiki that documents what's there
<ctalbert_> TabAtkins_: I accidentally locked myself on the patio,
could you come rescue me?
jj: Right, should be clear these are not submitted for review;
they're there, and someone can take them and convert them and submit
them
<MikeSmith> http://www.w3.org/wiki/Testing
http://www.w3.org/wiki/Testing
jgraham: Come up with a prioritized list of things that need tests
jj: anything that's in CR? :)
plh: I'll take an action item to do that
<scribe> ACTION: plh to make a list of things that need tests
[recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action02]
bryan: Need a list of what's available, what are the key gaps, what
do we need to get there
kk: Identify specs that are in a bad situation.
fantasai: Also want to track not just what needs testing, but ask
vendors whether they have tests for any of these.
... Can then go pester people to submit those tests
<scribe> ACTION: MikeSmith to Create repos for testing IG and
testing framework group [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action03]
plh: Need places to dump tests for groups that don't have repos atm
... more and more groups have their own test repo
<plh> ACTION: plh to convince the geolocation WG to use mercurial
for their tests [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action04]
3. Vendors commit to finding a person to facilitate submission and
use of W3C tests
wilhelm: need to make a formal request to each organization
bryan: Someone should pull together format descriptions and include
the guidelines
<plh> --> http://www.w3.org/html/wg/wiki/Testing/Authoring/
Authoring Tests
http://www.w3.org/html/wg/wiki/Testing/Authoring/
dicussion of where to collect this information
<plh> --> http://www.w3.org/testing/ Testing
http://www.w3.org/testing/
jgraham: should be in a place not specific to a given working group
...
plinss: There's a lot to be gained by standardizing metadata
jgraham: hard to do the CSS way for an HTML test
... Could have n ways to do it, where n is a small number
Alan: It would be nice to have everything on a wiki so we don't have
to go through a staff member
... What if this page was a redirect to a wiki?
jgraham: Could have that page be a link to a wiki
MikeSmith: I like redirect idea, minimizes work I have to do :)
wilhelm: So when should we meet again?
jj: I think we should definitely make this a regular meeting.
... Seems like everyone in every WG is going to be solving the same
problems
...
plh: WebDriver will be under browser tools WG
mc: Who's "we"?
wilhelm: I don't know, but this crowd is great.
plh: We can put under the IG
fantasai: We can say at last meet again next TPAC
plh: Would be in France next year
fantasai: Since not everyone will be travelling to TPAC, would we
want to do another place at at different time as well?
jj: Does everyone agree we should meet?
kk: Depends on deliverables.
MikeSmith: If we meet 6 months from now, when would that be?
?: April
mc: Just want to be sure who the "we" is the invite would go out to
wilhelm is designated in charge
Meeting closed.
RRSAgent: make minutes
Summary of Action Items
[NEW] ACTION: Mike to create mercurial repositories for Web Testing
IG and Browser Tools WG [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action01]
[NEW] ACTION: MikeSmith to Create repos for testing IG and testing
framework group [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action03]
[NEW] ACTION: plh to convince the geolocation WG to use mercurial
for their tests [recorded in
http://www.w3.org/2011/10/28-testing-minutes.html#action04]
[NEW] ACTION: plh to make a list of things that need tests [recorded
in http://www.w3.org/2011/10/28-testing-minutes.html#action02]
[End of minutes]
_________________________________________________________
--
Michael[tm] Smith
http://people.w3.org/mike/+
Received on Thursday, 3 November 2011 00:02:30 UTC