Re: Ideas of useful deliverables for this group

Hi Dom,

I believe we need to add to F2F meeting one more task: we need to create 
the list of the specifications that need to be covered by delivery of 
our working group. Now this published list (on web site) is not final. 
This list could be extended in future during the WG activity.

It is also useful to create the initial requirements for test harness. 
There are different approaches for test suite runs on w3c. Some tests 
suite are automatically executed, other - interactively. Could we find 
the general approach for the test execution and result storage during 
the F2F?

Thanks,
    Dmitri.


Carmelo Montanez wrote:
> Hey Dom:
>
> Thanks for this.  See some of my ideas/comments embedded below.
>
> Carmelo
>
> At 10:03 AM 1/17/2007, Dominique Hazael-Massieux wrote:
>
>> Hi,
>>
>> As promised, here is a summary of my ideas on what this group could be
>> developing over the upcoming few months.
>>
>> I think we have several options, some of which can be mixed together:
>> * we can look at the existing conformance tests suites out there as I
>> started to describe in my previous message [1], and try to re-package
>> them, maybe contributing to make them more complete and more useful for
>> user agent developers.
>>
>> * we can focus on making these tests suites more easily available/usable
>> by the web community at large, so that we can invite individuals to test
>> the conformance of the user agents they use and make it generate reports
>> that help web developers know what bugs exist in what browsers; the idea
>> would be to generate reports à la:
>> http://www.westciv.com/style_master/academy/browser_support/basic_concepts.html 
>>
>> but on a bigger scale (many more browsers), and with a collaborative
>> approach
>
> This should be the least we should strive for and definitely should be 
> a part of our
> deliverables.  Attached is a starting frameworks that I put together last
> year.  It is by no means complete or normative.
>
>> * we can try and create a set of "acid tests" for mobile web browsers, Ã
>> la:
>> http://www.webstandards.org/action/acid2/
>> with the idea of testing the integration of several technologies in
>> rather complex arrangements
>
> This approach seems fine for complex arrangements  and interaction of 
> features.
> This particular approach (acid) seems to be combining a lot of things
> at the same time.
>
>> * another approach that I think would be interesting to consider is to
>> focus on non-conformance tests suites (!); the idea would be to assemble
>> and create tests cases that wouldn't focus so much on whether a given
>> browser conforms to a given specification, but instead, to identify
>> common browsers behaviors for things that are un- or ill-specified, and
>> that web developers need data on.
>> For instance, during our work on the mobile web best practices, there
>> were at least 2 occasions where we needed a good survey of behaviors of
>> existing mobile user agents, and came up with a series of simple tests
>> to identify these behaviors:
>> http://www.w3.org/2005/MWI/BPWG/techs/XhtmlBasic11Support
>> http://www.w3.org/2005/MWI/BPWG/techs/EncodingDeclarationSupport
>> (while I have used a pass/fail color scheme in these reports, it doesn't
>> actually mean that the user agent is buggy per se, but only that it
>> didn't yield a behavior one may have wished it would)
>> Generally speaking, should we take that option, I think we would need a
>> strong collaboration with web developer communities, so that we could
>> get contributions of tests cases for well-known authoring techniques.
>> There are several existing other small tests suites that would fall
>> under that category, and that I think would be a good starting basis for
>> such an effort:
>> http://www.paxmodept.com/pan/index.xhtml
>> http://t.wurfl.com/
>> http://www.cameronmoll.com/mobile/mkp/
>> (and most likely, more of them)
>>
>
> I would love to see mobile browser developers provides us with a set 
> of areas/issue they
> will like to see address and developed test for.  It can be a starting 
> point for a test suite
> of this sort.
>
>> I think each of these approaches has its own merit, and would certainly
>> be happy to work on any of these, although I confess I have a slight
>> preference for the last one as it would probably fill a need that no
>> other existing efforts have filled so far.
>>
>> Some of these plans may require that we use rather specific software
>> solutions:
>> * if we invite the web community to submit tests results, we would need
>> a system to log tests results, and probably doing so in a
>> mobile-friendly fashion
>> * if we invite tests cases contributions, we would need a submission
>> system that takes into account the various policies in place at W3C on
>> this topic (e.g. [2])
>> * if we develop tests cases or review existing tests cases, we would
>> probably need some form of tests cases management system
>
> All of these are great topics for the face to face next week.
>
>> Sorry this mail is so long, but I think it summarizes most of my
>> thinking on the topic at this point; I would very much like to get
>> feedback on the various ideas exposed here.
>>
>> Thanks,
>>
>> Dom
>>
>> 1. http://lists.w3.org/Archives/Public/public-mwts/2007Jan/0007.html
>> 2. http://www.w3.org/2004/10/27-testcases.html

Received on Thursday, 18 January 2007 17:48:11 UTC