Evolving ringmark

Beware, long email, I've been lurking for too long...

I am not in the spec detail as much as some of the group, and am unable to attend the F2F but still wanted to provide an input from a mobile operator member. Hopefully it is helpful for CoreMob testing discussions.

>From my perspective, one of the most useful deliverables of CoreMob would be to improve implementation transparency - ie what web features and performance is actually delivered by real world mobile devices in market.  This obviously needs a testing capability, preferably one drawing on multiple contributions and with shared buy-in. The test results of in market devices should be publically available.  For devices under development, tests need to be run privately. This kind of infrastructure could help mobile web developers, mobile operators, and device vendors.

Understood ringmark is an input to CoreMob, not an output, however I believe ringmark and its results posted to Browserscope [1] take us a step forward. Ringmark's ring structure is helpful, and the summarised graphical view plus ability to drill down to get the details works for me. The ability to simply point my browser, let it run and then get results is also compelling..

Things I think would really add value 
 - ability to export/save the results in PC readable format
 - ability to find online, the test results for other devices & browser (in same ringmark format) 
 - feature-based drill down (like in r0) for r1 and r2, in addition to app-genre drill down
 - run all the rings by default, not stop on failing a ring

I also hope we don't lose ringmark's r0 with any CoreMob decision to shelve level zero.  It is useful to show if some browsers do not yet fully implement a reasonable baseline.

In short, I think ringmark is a good attempt to visualise what is extremely complex to measure. Any summary inherently has weaknesses so maybe there is room to have more than one visualisation of the same base (test result) data for different audiences, and to accommodate the inevitable different views of what is most important, how to implement, and what constitutes a pass or fail for a specific feature.  The most useful visualisation will no doubt rise to the top.



Then there is the topic of ringmark breadth and depth...
It's a scary thought to hear estimates of 1Mio tests to do the job !  Hopefully the 80:20 rule applies, but I suspect we will need to leverage existing work of others, and cannot assume people will (re)write tests to one specific framework.

Currently ringmark [2] declares to make 382 tests covering 78 features. There exists well known tools such as html5test [3] and css3test [4] and others including W3C tests and browser vendor tests. I hope we can leverage these.


Some steps that may help us move forward in CoreMob
1) get an overview of ringmark priorities, test coverage, scope (reported feature presence, functionality depth, implementation consistency, performance), major gaps and roadmap
2) identify other existing publically available tests which may help CoreMob objective
3) identify existing private tests in CoreMob member organisations which could be contributed
4) investigate potential to evolve ringmark to become an umbrella or framework test result visualisation tool which displays results from multiple sources; its own automated tests, other automated tests, manual test results.
5) Boil up 1-4 and draw some conclusions and next steps
6) recognise that test cases and notions of conformance can be "highly political" - so rather than expecting consensus, drive instead for transparency of results and allow room for different interpretations / visualisations if necessary.


At Vodafone, we are open to adopting a shared test tool and focus some resources to contribute to make something better for all. I think others have a similar view. However so far it has been difficult to 1) understand the scope of ringmark and 2) identify where focussed contributions are required. 

So, ringmark is a great start, we just need a little more detail and structure so we can agree a way forward.

Interested in your comments
Gavin


[1] http://www.browserscope.org/?category=ringmark&v=top-m
[2] http://rng.io/?all
[3] http://html5test.com (awards 500 points weighted across ~250 features)
[4] http://css3test.com (runs 900 tests for ~200 features)

Received on Sunday, 24 June 2012 20:23:53 UTC