[minutes] Weekly Testing Call - 2011-07-18

 Hi,

 The minutes of yesterday's call are available at:
  http://www.w3.org/2011/07/18-testing-minutes.html

 ... and copied as raw text below.

 Thanks,
 Francois.

 -----
 18 Jul 2011

    See also: [2]IRC log

       [2] http://www.w3.org/2011/07/18-testing-irc

 Attendees

    Present
           Shadi, Plh, francois, MikeSmith, wilhelm, Michael_Cooper,
           Jeanne

    Regrets
    Chair
           Philippe

    Scribe
           francois

 Contents

      * [3]Topics
          1. [4]Scope missing APIs relevant to accessibility in WG
             charter
          2. [5]Framework
          3. [6]Informal summer meeting
          4. [7]Next call
      * [8]Summary of Action Items
      _________________________________________________________

 Scope missing APIs relevant to accessibility in WG charter

    shadi: comment 2 should be the one we should focus on
    ... related to accessibility, UI automation
    ... other ones are more editorial.
    ... or can be handled at your discretion.

    mike: UI automation is something that is quite different from what
    we have in the current scope section. My understanding is that is a
    vendor specific platform and an OS specific API.
    ... Am I wrong about that?

    Michael: yes and no. There's a Microsoft and an Apple API that are
    vendor-specific.
    ... A testing API needs to be aware of these APIs.

    Mike: The WebDriver API should be at the core of the charter. The
    scope of it is well precised. It's something that runs in a Web
    browser.

    <shadi>
    [9]https://lists.w3.org/Archives/Team/w3mreq/2011Jul/0083.html

       [9] https://lists.w3.org/Archives/Team/w3mreq/2011Jul/0083.html

    plh: I think Wilhelm said before that they take inspiration from
    accessibility APIs.

    mike: right, what I do not understand here is how UI automation
    relates to this. I mean at the technical level.
    ... If I don't understand it, I can't write in the charter. If
    someone else can propose text, then I could include it in the
    charter, but I can't do that on my own.
    ... Another thing is that it seems kind of weird to me to mention UI
    automation in a charter focused on Web browsers.
    ... I do not have enough details to be able to write up a
    description to how this work relates to what we have in the charter.

    wilhelm: I'm familiar with what we do here, at Opera. The different
    parts of the spec are at various maturity level. The core of the
    spec is only interested at what happens at the Web page level.
    ... In addition to the core API, there are some experimentations
    going on but no consensus yet among browser vendors.
    ... I'd prefer to postpone extensions until we have more feedback
    experience.

    plh: ok, so maybe we're not clear enough in the charter. If we say
    we're not going to standardize the UI part, then we should make it
    clear in the charter.

    <plh> [10]http://www.w3.org/2011/08/browser-testing-charter.html

      [10] http://www.w3.org/2011/08/browser-testing-charter.html

    [reviewing draft charter text: "there is a specific need to simulate
    user actions such as clicking links, entering text, and submitting
    forms"]

    plh: what did you have in mind, Mike, with "any other potential APIs
    useful in performing automated testing of Web applications"?

    mike: I didn't have anything specific in mind.
    ... Happy to remove.

    plh: that's the part you're most interested in, shadi, I believe. Is
    your comment still relevant now that we know the API is only
    intended to drive things in the HTML page and not in the browser
    chrome?

    Michael: I would say yes as you can still drive the HTML content
    through the accessibility APIs.
    ... I think we need to acknowledge that.

    plh: section 1.1 is about the scope. Trying to extend this section
    is a mistake, I think. It should actually be tighter.

    mike: I agree we need review. But what does "dependency" mean?

    Michael: It's more a point of coordination.

    plh: ok, let's see where we can put it, then.

    [discussion about "draft notes" section that is to go away]

    Michael: I would add a new sub-section under Dependencies and
    Liaisons section, such as "relevant API".

    <plh> - Accessibility Interoperability Alliance (AIA)

    <plh> -- <[11]http://www.atia.org/aia/>

      [11] http://www.atia.org/aia/%3E

    <plh> - Open Ajax Alliance

    Mike: that's easy enough to do, yes.

    <plh> -- <[12]http://www.openajax.org/>

      [12] http://www.openajax.org/%3E

    Mike: Do they have some task force dedicated to accessibility I
    could point to?

    Michael: there is an accessibility sub-group of the Open Ajax
    Alliance, but I would not restrict the coordination to it as the
    whole things is relevant to our work.

    Mike: I have enough information to update the charter. I would still
    ask Michael to send more precise wording. "consider any other
    potential APIs" is too loose, I think.

    Shadi: the idea is "consider" as looking at the appropriate
    relationship with existing solutions if, of course, organizations
    are willing to contribute their work anyway.

    Michael: one reason to mention vendor-specific APIs is to have
    everyone be conscious of the overlap.

    Mike: that won't happen. It is not duplicating anything that
    everyone else has done. There is no existing API that works across
    browsers right now.

    plh: one way could be to expose these API in JavaScript.

    Mike: right, but there is no standard mechanism that does what this
    API does.
    ... It would be good to get some direct review from people in these
    companies.

    plh: Mike, to summarize, you need to run through Shadi's message.

    Mike: ok, I can do that by tomorrow.

    <MikeSmith> francois: what's the status on the IG charter:

    <MikeSmith> plh: it's stable, no changes needed

 Framework

    <MikeSmith> [13]http://w3c-test.org/framework/

      [13] http://w3c-test.org/framework/

    mike: so, Francois deployed something not working and then cowardly
    left for vacation leaving everything on my plate
    ... I had a look, and ran into weird problems, apparently because
    PHP version was not 5.3, and also Apache version was too late.
    ... I upgraded everything on the server.
    ... The thing is working.
    ... I also replaced it with a clone of the CSS server Mercurial
    repository.
    ... Now we can pull and push updates directly.
    ... What we could do this week is to setup things for it to work
    with the Web Performance WG spec.

    plh: Yes, good guinea pig. That's a good test.

    Mike: everybody else who is interested in getting the framework
    populated now that it's set up, I'm happy to help with that.
    ... It's team only because of SSH access.
    ... I think it's bad practice to all act as root on that server.
    ... Better to create a user account on that machine.

    plh: sounds good to me.

    shadi: if you can write this up somewhere, that would help.

    plh: you still need someone with root access to the machine.

    shadi: just an email that explains what to request from systeam and
    what to setup on one's machine would be good not to forget steps
    later on.

    plh: how to add a group in the framework would be a good thing to
    document

    mike: pretty well documented, actually.
    ... The process is clear but requires a lot of manual work. We could
    provide a better interface later on.
    ... You work with text files, and have to work with a few mysql
    statements.

    plh: ok, I don't think we need to focus so much on making things
    automated at this stage.

    mike: there are two pieces: the "framework" which Peter calls
    "harness". The second is "shepherd" wich is the test suites
    management system.
    ... I'm not sure I see that as a requirement.

    plh: well, that's why we picked up the framework in the first place.
    ... So would be good to deploy.

    mike: once we get the framework setup for Web Performances WG, we'll
    be clearer as to what we need.
    ... That should be doable this week.

    plh: we may need to update the tests, so could take longer.
    ... Another thing is multi-WG support.

    <MikeSmith> here is Peter's how-to:

    plh: Actually, I wonder if we want to tie this framework tied up to
    working groups.

    <MikeSmith>
    [14]http://hg.csswg.org/dev/harness/raw-file/tip/docs/index.html

      [14] http://hg.csswg.org/dev/harness/raw-file/tip/docs/index.html

    plh: better to tie it to specifications to handle cases when a
    working group get splitt up for instance.

    mike: here's the thing. That's more or less how it works today.
    ... It's per spec and test suite. We could simply add a column to
    link it to a working group.

    <MikeSmith> francois: the harness is already set up to work with
    different specs

    <MikeSmith> … and we could easily add a column for WG info if we
    want to

    [francois thanks mike a lot for taking up the hard task of debugging
    the deployment while he was enjoying sun on vacation!]

    <MikeSmith> francois, debugging wasn't so hard, and Tom helped some.
    I'm glad you got some vacation man

 Informal summer meeting

    shadi: I'm wondering what the process and expectations are.
    ... which kind of organizations are to attend the meeting. What's
    team attendance expectations, that sort of things.

    wilhelm: the general idea is to get an overview of what browser
    vendors handle tests today. We need to know what vendors need to
    test. And what they'd be ready to contribute.

    shadi: are we only focusing on browser vendors? Aren't we going to
    miss something?

    wilhelm: to answer the two questions, that's good to narrow the
    scope. We will have to extend things afterwards. In one or two days,
    there are only so many stuff you can cover. I hope we'll kickstart a
    long discussion here.

    shadi: The fix-up later approach is typically what makes
    accessibility added afterwards, which always creates problems. I'm
    concerned we'd be leaving a bunch of stakeholders.

    plh: this is an informal meeting. Not a group coming with a set of
    requirements.

    shadi: we want browser vendors to be more considerate about
    accessibility issues.

    <Zakim> MikeSmith, you wanted to say we have the consideration
    already, and this is not a case of risking "fix it later"

    Mike: we are already considering accessibility. Any group
    considering work in W3C already considers accessibility, even if not
    explicitly mentioned. I don't like the implication that we are
    ignoring accessibility here, and using a fix-it-up later approach.
    ... It's not a workshop, it's an informal meeting.

    plh: What I heard Wilhelm say is that we have browser vendors which
    have an extensive set of test cases and do not know how to leverage
    those in W3C.

    wilhelm: that's correct

    plh: We could say that anyone who has vast amount of tests is
    welcome to attend to see how these can be contributed.
    ... It's not a meeting about coming up with requirements.

    <gsnedders> I wonder how much of the infrastructure is going to be
    shared for a11y testing, because if you're testing what's passed to
    the OS APIs that's inevitable going to be a separate API to test.

    Michael: what's missing here is that there are all ways to tackle
    testing things. For me, it's not optional to have other stakeholders
    here, it's important to have them.

    <MikeSmith> which people

    shadi: we are missing people who are doing accessibility testing.

    <jeanne> there is a outside group doing accessibility testing - I
    will look up the name

    <MikeSmith> bingo

    shadi: The more we shortcut the deadlines, the more difficult it is
    to get people on board.

    <MikeSmith> there is interest from browser vendors

    plh: the goal is to see if there's interest from browser vendors in
    the first place.

    <MikeSmith> the lack of a response from them so far does not
    indicate they are not interested

    shadi: I do think we do need to take that we are trying to involve
    other players, and we should take care not to leave them aside
    unintentionally.

    plh: what we want to learn is how people do testing rather than how
    we should be doing testing.
    ... That's why I'm more inclined to attract people that have good
    amounts of tests.

    <Zakim> MikeSmith, you wanted to ask Michael to explain exactly what
    things and to say the idea is to keep this meeting small

    plh: I need someone to take an action item to come up with a precise
    list.

    Mike: in some comments you're making, Shadi, you're telling us that
    we may be forgetting accessibility but you're staying vary vague.

    <jeanne> jeanne found the group, and it is the same group MichaelC
    has been discussing. Sorry for distraction.

    Mike: Another point is that it is to remain an informal meeting,
    focused, with fewer people. The idea is to get these people into the
    same room.
    ... The goal is to save time, see how we can stop reinventing the
    wheel across groups. It would be a good idea to see people
    representing accessibility concerns but, as mentioned before, people
    involved in browser development are already addressing accessibility
    concerns in their everyday work.

    shadi: we're really not having diverging points at all.
    ... Having accessibility as part of every developer day job is a
    good goal, but it's still a goal, not the situation today, even
    within W3C.
    ... There has to be a compromise between the number of people
    attending this meeting and the number of viewpoints we can have.
    ... I think we're all having the same vision for the future.

    plh: What I'd like to have to move forward is to identify
    individuals, including accessibility.
    ... That's what Wilhelm did, targeting specific individuals, not
    companies.
    ... It would be good if you could come up with a list of people,
    Shadi.

    <MikeSmith> while I agree that making specific mention of
    accessibility coordination in charters in activity statements and
    other places does not imply that accessibility concerns are being
    neglected or that there is not a high level of accessibility
    consideration among the people who produced the activity statements
    and charters

    shadi: I'm asking Jeanne if she can take this up.
    ... We'll try to come up with a list.

    wilhelm: goal is to have the meeting as soon as humanly possible.

    mike: one thing about timing we should agree on is that when we have
    critical mass, we still need to announce it, about a month in
    advance for travel arrangements.

    plh: we're going to have to announce it as well in some fashion.
    ... We're going to have to strike a balance between open and closed
    door.
    ... Not a proper workshop (no position paper).
    ... We should have a list of questions ready for people interested
    to show up to check whether we need them in the meeting.
    ... We need people with a large number of tests who would like to
    contribute them.

    <shadi> [15]http://www.w3.org/2005/10/Process-20051014/events.html

      [15] http://www.w3.org/2005/10/Process-20051014/events.html

    shadi: note there is no requirement for position papers to call some
    event a workshop.

 Next call

    plh: I'm on the road next call, so propose to cancel it.

    <jeanne> jeanne is on vacation

    plh: Next call on August, 1st.

    [call adjourned]

 Summary of Action Items

    [End of minutes]

Received on Tuesday, 19 July 2011 15:05:19 UTC