Re: Ideas of useful deliverables for this group

HI All:

I think identifying which specifications we are 
going to address will make it easier to
tackle the execution/harness issues.

Carmelo



At 12:55 PM 1/18/2007, Dmitri Silaev wrote:
>Hi Dom,
>
>I believe we need to add to F2F meeting one more 
>task: we need to create the list of the 
>specifications that need to be covered by 
>delivery of our working group. Now this 
>published list (on web site) is not final. This 
>list could be extended in future during the WG activity.
>
>It is also useful to create the initial 
>requirements for test harness. There are 
>different approaches for test suite runs on w3c. 
>Some tests suite are automatically executed, 
>other - interactively. Could we find the general 
>approach for the test execution and result storage during the F2F?
>
>Thanks,
>    Dmitri.
>
>
>Carmelo Montanez wrote:
>>Hey Dom:
>>
>>Thanks for this.  See some of my ideas/comments embedded below.
>>
>>Carmelo
>>
>>At 10:03 AM 1/17/2007, Dominique Hazael-Massieux wrote:
>>
>>>Hi,
>>>
>>>As promised, here is a summary of my ideas on what this group could be
>>>developing over the upcoming few months.
>>>
>>>I think we have several options, some of which can be mixed together:
>>>* we can look at the existing conformance tests suites out there as I
>>>started to describe in my previous message [1], and try to re-package
>>>them, maybe contributing to make them more complete and more useful for
>>>user agent developers.
>>>
>>>* we can focus on making these tests suites more easily available/usable
>>>by the web community at large, so that we can invite individuals to test
>>>the conformance of the user agents they use and make it generate reports
>>>that help web developers know what bugs exist in what browsers; the idea
>>>would be to generate reports à la:
>>>http://www.westciv.com/style_master/academy/browser_support/basic_concepts.html 
>>>
>>>but on a bigger scale (many more browsers), and with a collaborative
>>>approach
>>
>>This should be the least we should strive for 
>>and definitely should be a part of our
>>deliverables.  Attached is a starting frameworks that I put together last
>>year.  It is by no means complete or normative.
>>
>>>* we can try and create a set of "acid tests" for mobile web browsers, Ã
>>>la:
>>>http://www.webstandards.org/action/acid2/
>>>with the idea of testing the integration of several technologies in
>>>rather complex arrangements
>>
>>This approach seems fine for complex 
>>arrangements  and interaction of features.
>>This particular approach (acid) seems to be combining a lot of things
>>at the same time.
>>
>>>* another approach that I think would be interesting to consider is to
>>>focus on non-conformance tests suites (!); the idea would be to assemble
>>>and create tests cases that wouldn't focus so much on whether a given
>>>browser conforms to a given specification, but instead, to identify
>>>common browsers behaviors for things that are un- or ill-specified, and
>>>that web developers need data on.
>>>For instance, during our work on the mobile web best practices, there
>>>were at least 2 occasions where we needed a good survey of behaviors of
>>>existing mobile user agents, and came up with a series of simple tests
>>>to identify these behaviors:
>>>http://www.w3.org/2005/MWI/BPWG/techs/XhtmlBasic11Support
>>>http://www.w3.org/2005/MWI/BPWG/techs/EncodingDeclarationSupport
>>>(while I have used a pass/fail color scheme in these reports, it doesn't
>>>actually mean that the user agent is buggy per se, but only that it
>>>didn't yield a behavior one may have wished it would)
>>>Generally speaking, should we take that option, I think we would need a
>>>strong collaboration with web developer communities, so that we could
>>>get contributions of tests cases for well-known authoring techniques.
>>>There are several existing other small tests suites that would fall
>>>under that category, and that I think would be a good starting basis for
>>>such an effort:
>>>http://www.paxmodept.com/pan/index.xhtml
>>>http://t.wurfl.com/
>>>http://www.cameronmoll.com/mobile/mkp/
>>>(and most likely, more of them)
>>
>>I would love to see mobile browser developers 
>>provides us with a set of areas/issue they
>>will like to see address and developed test 
>>for.  It can be a starting point for a test suite
>>of this sort.
>>
>>>I think each of these approaches has its own merit, and would certainly
>>>be happy to work on any of these, although I confess I have a slight
>>>preference for the last one as it would probably fill a need that no
>>>other existing efforts have filled so far.
>>>
>>>Some of these plans may require that we use rather specific software
>>>solutions:
>>>* if we invite the web community to submit tests results, we would need
>>>a system to log tests results, and probably doing so in a
>>>mobile-friendly fashion
>>>* if we invite tests cases contributions, we would need a submission
>>>system that takes into account the various policies in place at W3C on
>>>this topic (e.g. [2])
>>>* if we develop tests cases or review existing tests cases, we would
>>>probably need some form of tests cases management system
>>
>>All of these are great topics for the face to face next week.
>>
>>>Sorry this mail is so long, but I think it summarizes most of my
>>>thinking on the topic at this point; I would very much like to get
>>>feedback on the various ideas exposed here.
>>>
>>>Thanks,
>>>
>>>Dom
>>>
>>>1. http://lists.w3.org/Archives/Public/public-mwts/2007Jan/0007.html
>>>2. http://www.w3.org/2004/10/27-testcases.html

Received on Thursday, 18 January 2007 19:38:41 UTC