Re: Evolving ringmark

On Fri, 22 Jun 2012 17:03:05 +0200, Thomas, Gavin, Vodafone Group
<Gavin.Thomas@vodafone.com> wrote:
> Someone had said...
>> From my perspective, one of the most useful deliverables of CoreMob  
>> would be to improve implementation transparency - ie what web features  
>> and performance is actually delivered by real world mobile devices in  
>> market.  This obviously needs a testing capability, preferably one  
>> drawing on multiple contributions and with shared buy-in. The test  
>> results of in market devices should be publically available.  For  
>> devices under development, tests need to be run privately. This kind of  
>> infrastructure could help mobile web developers, mobile operators, and  
>> device vendors.
>
> Understood ringmark is an input to CoreMob, not an output, however I  
> believe ringmark and its results posted to Browserscope [1] take us a  
> step forward. Ringmark's ring structure is helpful,

Well, it's a good visual communication technique. Frankly I find the
failure to run ring 1 if you don't get 100% on ring 0 to be extremely
limiting to its usefulness :(

> and the summarised graphical view plus ability to drill down to get the  
> details works for me. The ability to simply point my browser, let it run  
> and then get results is also compelling..

Yes and yes.

> Things I think would really add value
>  - ability to export/save the results in PC readable format

True - although you could do that with an extension in most browsers, it
should be done in the test.

>  - ability to find online, the test results for other devices & browser  
> (in same ringmark format)

Yes...

>  - feature-based drill down (like in r0) for r1 and r2, in addition to  
> app-genre drill down
>  - run all the rings by default, not stop on failing a ring

Absolutely

> I also hope we don't lose ringmark's r0 with any CoreMob decision to  
> shelve level zero.  It is useful to show if some browsers do not yet  
> fully implement a reasonable baseline.

Except that one reason for shelving coremob zero is the fact that we
failed to come up with agreement on what a reasonable baseline *is*.

> Then there is the topic of ringmark breadth and depth...
> It's a scary thought to hear estimates of 1Mio tests to do the job !
> Hopefully the 80:20 rule applies, but I suspect we will need to leverage  
> existing work of others, and cannot assume people will (re)write tests  
> to one specific framework.
>
> Currently ringmark [2] declares to make 382 tests covering 78 features.  
> There exists well known tools such as html5test [3] and css3test [4] and  
> others including W3C tests and browser vendor tests. I hope we can  
> leverage these.

Sure. But we should be careful. It turns out that there are browsers which
pass tests for these things, but don't actually implement the
functionality :( Which means we need to check the tests pretty carefully.

> 2) identify other existing publically available tests which may help  
> CoreMob objective
> 3) identify existing private tests in CoreMob member organisations which  
> could be contributed

Ensure all these tests are provided to the relevant W3C working groups,
(and checked by them to ensure that they test what they claim)

> 4) investigate potential to evolve ringmark to become an umbrella or  
> framework test result visualisation tool which displays results from  
> multiple sources; its own automated tests, other automated tests, manual  
> test results.

That seems like a scope creep that we don't need. Although it would be
great if someone else decided to take it on.

> 5) Boil up 1-4 and draw some conclusions and next steps

Conclusions: We don't have enough tests, they are not good enough - but
let's not let perfect be the enemy of good.
Next steps: Rinse and repeat.

> 6) recognise that test cases and notions of conformance can be "highly  
> political" - so rather than expecting consensus, drive instead for  
> transparency of results and allow room for different interpretations /  
> visualisations if necessary.

Indeed.

> At Vodafone, we are open to adopting a shared test tool and focus some  
> resources to contribute to make something better for all. I think others  
> have a similar view. However so far it has been difficult to 1)  
> understand the scope of ringmark

It seems ringmark as shipped has a particular focus driven by the
Statement of Work. But as open source, you can focus it where you want...

> and 2) identify where focussed contributions are required.

IMHO (since I am not at the face to face, I'll just say what I think it
should be concluding ;) ):

writing tests for features you care about (because you are trying to ship
stuff based on them) is probably the most important contribution you can
make.

I'd be very happy to see a fork that adds the "save a picture for me"
feature. As a bonus, make the image SVG, enable to to collect results from
different browsers using semi-transparent colours so you can overlay them
and see e.g. the common set for your browser list (the things that are
just green green green)...

document the source code and how to work with it better (this is like "fix  
bugs" - it more or less never goes away as a task, no matter how well it  
is begin done).

cheers

Chaals

-- 
Charles 'chaals' McCathieNevile  Opera Software, Standards Group
       je parle français -- hablo español -- jeg kan noen norsk
http://my.opera.com/chaals       Try Opera: http://www.opera.com

Received on Monday, 25 June 2012 07:31:09 UTC