W3C home > Mailing lists > Public > public-mobileok-checker@w3.org > March 2009

Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

From: Jo Rabin <jrabin@mtld.mobi>
Date: Mon, 02 Mar 2009 21:39:05 +0000
Message-ID: <49AC51F9.9090607@mtld.mobi>
To: Yeliz Yesilada <yesilady@cs.man.ac.uk>
CC: Francois Daoust <fd@w3.org>, public-mobileok-checker@w3.org, Kentarou Fukuda <KENTAROU@jp.ibm.com>
I'm sorry to pick up on this so belatedly. I Generally agree with the 
thrust of the thread but given the specific wording in mobilOK Basic 
Tests - e.g. 1.2 Applicability, 2.2 Validity of the Tests and 2.3 
Testing outcomes - whatever is being discussed here is not mobileOK and 
we know (and love) it.

If there is scope for off-line testing then I think it might be useful 
for a WG note on the topic.

Jo

Yeliz Yesilada wrote:
> Hi Francois,
> 
> Thanks for the clarification. I also think the first approach is better. 
> If everybody agrees with this, we will go for that approach.
> 
> Yeliz.
> On 16 Feb 2009, at 08:47, Francois Daoust wrote:
> 
>> Yeliz Yesilada wrote:
>>> Hi Francois,
>>> I am sorry but I am a bit confused about what you are suggesting :(
>>
>> Yes, I see what you mean, I was mostly thinking aloud, but didn't make 
>> any choice ;)
>>
>> I think I convinced myself that the first approach below is both the 
>> most satisfactory and the easiest to implement. So I'd go for it.
>>
>>
>>> Are we talking about the same approach? so in summary...
>>
>>> Francois wrote:
>>>> Another possibility is to have FAIL override CANNOTTELL. In other 
>>>> words compute the test outcome as:
>>>>  If any of the subtests outcome is FAIL,
>>>>    then test outcome is FAIL
>>>>  else if any of the subtests outcome is CANNOTTELL,
>>>>    then test outcome is CANNOTTELL
>>>>  else
>>>>    test outcome is PASS
>>>> ... and about the same thing at the overall outcome level.
>>
>> +1 for this one. It's easy to implement and ensures the outcome value 
>> still means something:
>>  PASS: mobileOK
>>  CANNOTTELL: looking fine, but not everything could be checked
>>  FAIL: something's wrong (test may not have been run entirely)
>>
>>
>>> or are you suggesting that we go for your original suggestion by 
>>> including PARTIAL results?
>>>> In short, the possible outcomes for a subtest (the <result> element 
>>>> in the XML report) would be:
>>>>  - PASS, WARN, FAIL for subtests that can be run normally.
>>>>  - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can 
>>>> only be applied partially.
>>>>  - CANNOTTELL for subtests that simply can't be run.
>>>>
>>>> The possible outcomes for a test would be:
>>>>  - PASS, FAIL for tests that can be completely checked
>>>>  - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or 
>>>> CANNOTTELL in one of the subtests
>>>>  - CANNOTTELL when none of the subtests could be run (e.g. for CACHING)
>>>>
>>>> The possible overall outcomes would be:
>>>>  - PASS, FAIL when all tests can be completely checked (http/https 
>>>> case)
>>>>  - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or 
>>>> CANNOTTELL in one of the tests
>>
>> -1 for that one, since it's more complex and the outcome value would 
>> then carry two orthogonal dimensions, which doesn't look that good.
>>
>> Francois.
>>
>>>>
>>> Regards,
>>> Yeliz.
>>> On 13 Feb 2009, at 13:57, Francois Daoust wrote:
>>>>
>>>> Yeliz Yesilada wrote:
>>>> [...]
>>>>>>
>>>>>> 3/ I think there is a useful distinction to be made between a 
>>>>>> subtest that can't be run because some data is missing, and a 
>>>>>> subtest that can't be run because it doesn't need to, i.e. if 
>>>>>> there are no objects in the page, the OBJECT_OR_SCRIPTS subtests 
>>>>>> de facto pass. The first possibility is what we're talking about. 
>>>>>> The second possibility may be of some use in the future (I'm not 
>>>>>> suggesting we implement it right now). In short, I would rather 
>>>>>> keep NOT_APPLICABLE to the second case, and use DATA_MISSING (I 
>>>>>> can't think of a better proposal, but the idea is to point out 
>>>>>> that the moki representation is incomplete) for checks on files.
>>>>> I agree. I think what Dominique suggested is a good idea: using 
>>>>> "cannotTell".
>>>>
>>>> Yes, good idea, thanks dom!
>>>>
>>>>
>>>> [...]
>>>>> I think we need to think "why one would like to know if a 
>>>>> *sub-test* passed partially or not?". For example, in our 
>>>>> application if a sub-test can only be checked partially, then we 
>>>>> have to use the Tester (URI) version to check that again so it's 
>>>>> enough to know that the a particular sub-test cannot be tested.
>>>>> I am just not sure if these partial solutions would be useful or 
>>>>> not. I would rather prefer to keep the approach simple:
>>>>> subtests
>>>>> =======
>>>>> - PASS/FAIL/WARN that can run normally
>>>>> - CANNOTTELL if there is missing information
>>>>> tests
>>>>> ====
>>>>> - PASS/FAIL/WARN that can run normally
>>>>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL"
>>>>> But do you still think it's important to have PARTIAL results?
>>>>
>>>> At the subtest level, there aren't so many subtests that are 
>>>> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2 
>>>> and -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is 
>>>> simply too big, because that's a core mobile limitation. That being 
>>>> said, it may be added in a second time.
>>>>
>>>> At the test level, I just think that we lose the ability to quickly 
>>>> point out the outcome of the test. You'll basically end up with the 
>>>> following in each and every check run on a file document:
>>>> <tests outcome="CANNOTTELL">
>>>>  <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL">
>>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>>  </test>
>>>>  <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL">
>>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>>  </test>
>>>>  <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL">
>>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>>  </test>
>>>>  [...]
>>>> </tests>
>>>>
>>>> Whilst totally correct, the overall outcome and the outcome of the 
>>>> tests mentioned above doesn't tell you whether there is a FAIL in 
>>>> one of the subtests or not. Sure enough, this can be sorted out by 
>>>> having a look at the list of <result />, but it's the same thing 
>>>> today: the outcome attribute is more a "visual" clue than a 
>>>> computational need (i.e. its value can be re-computed at any time by 
>>>> having a look at the list of results elements). By limiting 
>>>> ourselves to CANNOTTELL, we're dropping that "visual" clue: any 
>>>> report on a file will need to be parsed to see how many of the tests 
>>>> that could be run failed.
>>>>
>>>> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL 
>>>> pretty easily, but I guess I just like the idea to still have a 
>>>> notion of PASS/FAIL at the test and overall outcome level.
>>>>
>>>> Or...
>>>>
>>>> Another possibility is to have FAIL override CANNOTTELL. In other 
>>>> words compute the test outcome as:
>>>>  If any of the subtests outcome is FAIL,
>>>>    then test outcome is FAIL
>>>>  else if any of the subtests outcome is CANNOTTELL,
>>>>    then test outcome is CANNOTTELL
>>>>  else
>>>>    test outcome is PASS
>>>> ... and about the same thing at the overall outcome level.
>>>>
>>>> Tools would still have to go through the list of results to tell 
>>>> which tests were only partially run, but at least, looking at a 
>>>> CANNOTTELL would tell you that "the thing looks great so far 
>>>> although not everything could be checked", while FAIL would keep its 
>>>> "something's wrong" meaning. It does work well provided that a FAIL 
>>>> in a file:// case always imply a FAIL in a http:// case, otherwise 
>>>> we would just raise a red flag that isn't a real one. I think that's 
>>>> almost true, except the uncertainty about <object> elements for 
>>>> computing included resources when files are handled. I can live with 
>>>> that.
>>>>
>>>> This could even be applied to subtests, and the net result is that 
>>>> we don't have to define more outcome values that carry orthogonal 
>>>> meanings.
>>>>
>>>> What about it?
>>>>
>>>> [...]
>>>>
>>>> Francois.
>>>>
>>
> 
> 
Received on Monday, 2 March 2009 21:39:51 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 2 March 2009 21:39:52 GMT