Notes on using the CSS Mobile Profile Test Suite from a mobile device

Hi,

As per my action item, I've tried to run the CSS Mobile Profile Test
Suite [1] on a set of user agents/devices; I have indeed run all the
tests of my Treo 650 with its default user agent (Blazer 4.0), as well
as having run a few tests cases on Opera Mini on that same device, on
the Netfront browser of the PSP, and on the OpenWave browser made
available on Sagem myX-5m.

The biggest problem of doing so at this time is the total lack of
usability of the test suites: the list of tests is given in a big table
that makes it pretty hard to use on most devices, and in fact mostly
impossible on the Openwave browser (which sits on a much smaller
configuration than the other ones).

Also, to go from one test to another, one need to go back and forth
between the big list of tests and each individual test: there is no
internal navigation provided.

Beyond this, the test suite heavily relies on colors, which aren't
available on all devices - the mobile web best practices specifically
recommend not relying on colors [2]. Should I have been able to run the
tests suite on the openwave browser, I couldn't have told any useful
results since it doesn't support colors on that specific device. Even on
devices/browsers that support colors, some choices of color make it hard
to distinguish whether the results matches what was expected (e.g. some
dark dark blue in 526-01 that is hard to distinguish from black).

Some tests assume a fairly wide screen, especially the ones for float:
and other layout properties, and make it hard or impossible to assess
whether the results is a pass or not.

The tests that rely on some specific fonts set up are also fragile,
given that most devices provide only a limited sets of fonts.

I also noticed some more casual bugs or lack of clarity in the
description of the expected results.

Till and Carmelo have also volunteered to provide feedback on that test
suite; I propose that once they have sent their own feedback, we should
look at ways in which these problems could be solved (if they can), and
send feedback to the CSS WG.

Dom

1. http://www.w3.org/Style/CSS/Test/Mobile/1.0/current/
2. http://www.w3.org/TR/2006/PR-mobile-bp-20061102/#iddiv3126714184

Received on Tuesday, 27 February 2007 08:55:56 UTC