RE: Ideas of useful deliverables for this group

Hello all,
	I just wanted to share my cents of experience.

Thank you Dom for adding a link to my simple testing site, t.wurfl.com. The site is far from complete, but what is in and will be in is the result of a previous experience.
In the past a test-suite had been built for the WURFL community. It had a fairly advanced engine that was able to show a page or a content, let the user know what the data in WURFL said about it and the user was able to confirm or request a change. The original site was fairly complete and included markup tests for WML, XHTML, iHTML, images and sounds.

The results were poor and I think for 2 main reasons:
1) there were MANY tests and users got bored. The result was that most users entered the section they were most interested in and only completed those tests;
2) tests were not very clear.

I would also like to add that it was a totally "distributed" contribution system and I mean that the URL was published on mailing lists and the WURFL site and everyone was invited to test his device.
Often we happened to receive requests for changes that we thought were not correct, but the system did not include any way to recognize the user and go back and ask how the test happened, if he could try again and so on.
In the mobile space you are using a device on-the-move accessing through a carrier that is not always totally open and in most cases you have a proxy in the middle. Chances of failure are high and the possibility of testing again are needed.

My suggestion, to test mobile content, is to provide a number of very basic tests, even only to verify very basic compliance to a certain standard such as XHTML Basic 1.0, 1.1, MP and so on.

I agree that testing general behaviour can be interesting. The small screen and reduced keyboard often result in strange behaviour especially with tables or colors.

I obviously support the idea of an open environment and the request to anyone to join us and test their browser. I think it is important to provide a way to the user to give an e-mail address or some reference. A very basic login system might be useful for users that test a lot of devices.
I support the idea of a back-end to manage test results. I think data should be linked to the usernames or e-mails provided. This will let us get a chance to get in touch with users at a later stage, if needed.

I also suggest the development of an online manual for the test-suite. A Wiki will be more than enough. Screenshots in the wiki of how we expect the page to look like will be specifically useful for the "behaviour tests". Sometimes it could be hard to explain what we expected in a test.

I think that the development of one or more test-suite sites should be part of this group's work. This is a very practical work, if we just go with a spec, I can't think of who will do the implementation. If I were a third party the first thing I'd ask myself would be "why didn't they implement it themselves?". Also, isn't the W3C's site the best place to host a test-suite about markup's and browser behaviour?

- Andrea
 

> -----Original Message-----
> From: public-mwts-request@w3.org 
> [mailto:public-mwts-request@w3.org] On Behalf Of Rosenthal, Lynne
> Sent: Friday, January 19, 2007 1:30 PM
> To: public-mwts@w3.org
> Subject: RE: Ideas of useful deliverables for this group
> 
> 
> All
> 
> This is a nice problem to have -- that is, having a few 
> possibilities and being able to choose and an excellent group 
> of people to work on it.  It may be worthwhile to consider a 
> few things in deciding what to pursue:
> 1. What would do the most good - e.g., be useful, have an 
> impact, make a difference, etc. 
> 2. What is the problem we are trying to solve 3. Is there a 
> specification, use case or requirements for which we are 
> building tests/tools (this is necessary to have if we do 
> conformance tests) 4. What is the interest of the group 5. 
> What is the timeframe for producing something useful
> 
> It would be great to be able to produce something in a very 
> short timeframe.  This would establish the WG and possibly 
> get others interested in the WG. Thus, it may make sense to 
> pursue 2 efforts - one a short term deliverable (which may 
> not be the ideal, most useful tool) and the other, a longer 
> term development of a needed tests/tool that would a) target 
> conformance, b) improve devices, c) foster adoption, d) any 
> or all these.  
> 
> Regards
> Lynne
> 
> 
> > -----Original Message-----
> > From: public-mwts-request@w3.org 
> [mailto:public-mwts-request@w3.org] 
> > On Behalf Of Dominique Hazael-Massieux
> > Sent: Friday, January 19, 2007 3:10 AM
> > To: Allen Wang
> > Cc: public-mwts@w3.org
> > Subject: Re: Ideas of useful deliverables for this group
> > 
> > 
> > Hi Allen,
> > 
> > Le jeudi 18 janvier 2007 à 16:54 -0800, Allen Wang a écrit :
> > > >* another approach that I think would be interesting to 
> consider is 
> > > >to focus on non-conformance tests suites (!); the idea 
> would be to
> > assemble
> > > >and create tests cases that wouldn't focus so much on whether a 
> > > >given browser conforms to a given specification, but instead, to 
> > > >identify common browsers behaviors for things that are un- or 
> > > >ill-specified, and that web developers need data on.
> > > >
> > >
> > > This certainly is an interesting idea. However, it seems that 
> > > focusing on non-conformance test suites may contradict to 
> our charter:
> > >
> > > "The test suites ... could be suitable for checking 
> conformance of 
> > > user agents to specifications in the mobile Web space."
> > 
> > Note the "could"; in other words, this is not an obligation of our 
> > charter, but a possible direction.
> > 
> > > So I suggest we as the members still focus on the 
> conformance part 
> > > of the test suite but leave the non-conformance part 
> largely to the 
> > > community, as Dom suggested.
> > 
> > That sounds good in theory, but if we want the latter part 
> to happen, 
> > we would need to provide the infrastructure to do so - e.g. 
> a system 
> > to contribute tests cases, a test harness, a test reporting system, 
> > etc. We would also likely need to review the submissions of test 
> > cases; could you clarify whether you think we shouldn't 
> even provide 
> > that infrastructure and only focus on conformance tests suites?
> > 
> > > Also, we may want to get input from the Mobile Web 
> Initiative group 
> > > as they may have a better understanding of the priorities 
> of our group.
> > 
> > At this point, I think it's really up to us to decide, based on our 
> > evaluation of the needs from the community, vendors, etc.
> > 
> > Dom
> > 
> > 
> 
> 
>

Received on Tuesday, 30 January 2007 02:05:05 UTC