W3C home > Mailing lists > Public > public-mobileok-checker@w3.org > February 2009

Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

From: Francois Daoust <fd@w3.org>
Date: Mon, 16 Feb 2009 09:47:15 +0100
Message-ID: <49992813.5050105@w3.org>
To: Yeliz Yesilada <yesilady@cs.man.ac.uk>
CC: public-mobileok-checker@w3.org, Kentarou Fukuda <KENTAROU@jp.ibm.com>
Yeliz Yesilada wrote:
> Hi Francois,
> 
> I am sorry but I am a bit confused about what you are suggesting :(

Yes, I see what you mean, I was mostly thinking aloud, but didn't make 
any choice ;)

I think I convinced myself that the first approach below is both the 
most satisfactory and the easiest to implement. So I'd go for it.


> Are we talking about the same approach? so in summary...

> 
> Francois wrote:
>> Another possibility is to have FAIL override CANNOTTELL. In other 
>> words compute the test outcome as:
>>  If any of the subtests outcome is FAIL,
>>    then test outcome is FAIL
>>  else if any of the subtests outcome is CANNOTTELL,
>>    then test outcome is CANNOTTELL
>>  else
>>    test outcome is PASS
>> ... and about the same thing at the overall outcome level.

+1 for this one. It's easy to implement and ensures the outcome value 
still means something:
  PASS: mobileOK
  CANNOTTELL: looking fine, but not everything could be checked
  FAIL: something's wrong (test may not have been run entirely)


> 
> 
> 
> or are you suggesting that we go for your original suggestion by 
> including PARTIAL results?
> 
>> In short, the possible outcomes for a subtest (the <result> element in 
>> the XML report) would be:
>>  - PASS, WARN, FAIL for subtests that can be run normally.
>>  - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can only 
>> be applied partially.
>>  - CANNOTTELL for subtests that simply can't be run.
>>
>> The possible outcomes for a test would be:
>>  - PASS, FAIL for tests that can be completely checked
>>  - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or 
>> CANNOTTELL in one of the subtests
>>  - CANNOTTELL when none of the subtests could be run (e.g. for CACHING)
>>
>> The possible overall outcomes would be:
>>  - PASS, FAIL when all tests can be completely checked (http/https case)
>>  - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or 
>> CANNOTTELL in one of the tests

-1 for that one, since it's more complex and the outcome value would 
then carry two orthogonal dimensions, which doesn't look that good.

Francois.

>>
> 
> Regards,
> Yeliz.
> 
> 
> 
> 
> On 13 Feb 2009, at 13:57, Francois Daoust wrote:
> 
>>
>> Yeliz Yesilada wrote:
>> [...]
>>>>
>>>> 3/ I think there is a useful distinction to be made between a 
>>>> subtest that can't be run because some data is missing, and a 
>>>> subtest that can't be run because it doesn't need to, i.e. if there 
>>>> are no objects in the page, the OBJECT_OR_SCRIPTS subtests de facto 
>>>> pass. The first possibility is what we're talking about. The second 
>>>> possibility may be of some use in the future (I'm not suggesting we 
>>>> implement it right now). In short, I would rather keep 
>>>> NOT_APPLICABLE to the second case, and use DATA_MISSING (I can't 
>>>> think of a better proposal, but the idea is to point out that the 
>>>> moki representation is incomplete) for checks on files.
>>> I agree. I think what Dominique suggested is a good idea: using 
>>> "cannotTell".
>>
>> Yes, good idea, thanks dom!
>>
>>
>> [...]
>>> I think we need to think "why one would like to know if a *sub-test* 
>>> passed partially or not?". For example, in our application if a 
>>> sub-test can only be checked partially, then we have to use the 
>>> Tester (URI) version to check that again so it's enough to know that 
>>> the a particular sub-test cannot be tested.
>>> I am just not sure if these partial solutions would be useful or not. 
>>> I would rather prefer to keep the approach simple:
>>> subtests
>>> =======
>>> - PASS/FAIL/WARN that can run normally
>>> - CANNOTTELL if there is missing information
>>> tests
>>> ====
>>> - PASS/FAIL/WARN that can run normally
>>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL"
>>> But do you still think it's important to have PARTIAL results?
>>
>> At the subtest level, there aren't so many subtests that are 
>> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2 and 
>> -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is simply too 
>> big, because that's a core mobile limitation. That being said, it may 
>> be added in a second time.
>>
>> At the test level, I just think that we lose the ability to quickly 
>> point out the outcome of the test. You'll basically end up with the 
>> following in each and every check run on a file document:
>> <tests outcome="CANNOTTELL">
>>  <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL">
>>   [list of FAIL/WARN/CANNOTTELL results]
>>  </test>
>>  <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL">
>>   [list of FAIL/WARN/CANNOTTELL results]
>>  </test>
>>  <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL">
>>   [list of FAIL/WARN/CANNOTTELL results]
>>  </test>
>>  [...]
>> </tests>
>>
>> Whilst totally correct, the overall outcome and the outcome of the 
>> tests mentioned above doesn't tell you whether there is a FAIL in one 
>> of the subtests or not. Sure enough, this can be sorted out by having 
>> a look at the list of <result />, but it's the same thing today: the 
>> outcome attribute is more a "visual" clue than a computational need 
>> (i.e. its value can be re-computed at any time by having a look at the 
>> list of results elements). By limiting ourselves to CANNOTTELL, we're 
>> dropping that "visual" clue: any report on a file will need to be 
>> parsed to see how many of the tests that could be run failed.
>>
>> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL 
>> pretty easily, but I guess I just like the idea to still have a notion 
>> of PASS/FAIL at the test and overall outcome level.
>>
>> Or...
>>
>> Another possibility is to have FAIL override CANNOTTELL. In other 
>> words compute the test outcome as:
>>  If any of the subtests outcome is FAIL,
>>    then test outcome is FAIL
>>  else if any of the subtests outcome is CANNOTTELL,
>>    then test outcome is CANNOTTELL
>>  else
>>    test outcome is PASS
>> ... and about the same thing at the overall outcome level.
>>
>> Tools would still have to go through the list of results to tell which 
>> tests were only partially run, but at least, looking at a CANNOTTELL 
>> would tell you that "the thing looks great so far although not 
>> everything could be checked", while FAIL would keep its "something's 
>> wrong" meaning. It does work well provided that a FAIL in a file:// 
>> case always imply a FAIL in a http:// case, otherwise we would just 
>> raise a red flag that isn't a real one. I think that's almost true, 
>> except the uncertainty about <object> elements for computing included 
>> resources when files are handled. I can live with that.
>>
>> This could even be applied to subtests, and the net result is that we 
>> don't have to define more outcome values that carry orthogonal meanings.
>>
>> What about it?
>>
>> [...]
>>
>> Francois.
>>
> 
> 
Received on Monday, 16 February 2009 08:48:01 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 16 February 2009 08:48:02 GMT