Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

Hi Francois,

Thanks for the clarification. I also think the first approach is  
better. If everybody agrees with this, we will go for that approach.

Yeliz.
On 16 Feb 2009, at 08:47, Francois Daoust wrote:

> Yeliz Yesilada wrote:
>> Hi Francois,
>> I am sorry but I am a bit confused about what you are suggesting :(
>
> Yes, I see what you mean, I was mostly thinking aloud, but didn't  
> make any choice ;)
>
> I think I convinced myself that the first approach below is both  
> the most satisfactory and the easiest to implement. So I'd go for it.
>
>
>> Are we talking about the same approach? so in summary...
>
>> Francois wrote:
>>> Another possibility is to have FAIL override CANNOTTELL. In other  
>>> words compute the test outcome as:
>>>  If any of the subtests outcome is FAIL,
>>>    then test outcome is FAIL
>>>  else if any of the subtests outcome is CANNOTTELL,
>>>    then test outcome is CANNOTTELL
>>>  else
>>>    test outcome is PASS
>>> ... and about the same thing at the overall outcome level.
>
> +1 for this one. It's easy to implement and ensures the outcome  
> value still means something:
>  PASS: mobileOK
>  CANNOTTELL: looking fine, but not everything could be checked
>  FAIL: something's wrong (test may not have been run entirely)
>
>
>> or are you suggesting that we go for your original suggestion by  
>> including PARTIAL results?
>>> In short, the possible outcomes for a subtest (the <result>  
>>> element in the XML report) would be:
>>>  - PASS, WARN, FAIL for subtests that can be run normally.
>>>  - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can  
>>> only be applied partially.
>>>  - CANNOTTELL for subtests that simply can't be run.
>>>
>>> The possible outcomes for a test would be:
>>>  - PASS, FAIL for tests that can be completely checked
>>>  - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or  
>>> CANNOTTELL in one of the subtests
>>>  - CANNOTTELL when none of the subtests could be run (e.g. for  
>>> CACHING)
>>>
>>> The possible overall outcomes would be:
>>>  - PASS, FAIL when all tests can be completely checked (http/ 
>>> https case)
>>>  - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or  
>>> CANNOTTELL in one of the tests
>
> -1 for that one, since it's more complex and the outcome value  
> would then carry two orthogonal dimensions, which doesn't look that  
> good.
>
> Francois.
>
>>>
>> Regards,
>> Yeliz.
>> On 13 Feb 2009, at 13:57, Francois Daoust wrote:
>>>
>>> Yeliz Yesilada wrote:
>>> [...]
>>>>>
>>>>> 3/ I think there is a useful distinction to be made between a  
>>>>> subtest that can't be run because some data is missing, and a  
>>>>> subtest that can't be run because it doesn't need to, i.e. if  
>>>>> there are no objects in the page, the OBJECT_OR_SCRIPTS  
>>>>> subtests de facto pass. The first possibility is what we're  
>>>>> talking about. The second possibility may be of some use in the  
>>>>> future (I'm not suggesting we implement it right now). In  
>>>>> short, I would rather keep NOT_APPLICABLE to the second case,  
>>>>> and use DATA_MISSING (I can't think of a better proposal, but  
>>>>> the idea is to point out that the moki representation is  
>>>>> incomplete) for checks on files.
>>>> I agree. I think what Dominique suggested is a good idea: using  
>>>> "cannotTell".
>>>
>>> Yes, good idea, thanks dom!
>>>
>>>
>>> [...]
>>>> I think we need to think "why one would like to know if a *sub- 
>>>> test* passed partially or not?". For example, in our application  
>>>> if a sub-test can only be checked partially, then we have to use  
>>>> the Tester (URI) version to check that again so it's enough to  
>>>> know that the a particular sub-test cannot be tested.
>>>> I am just not sure if these partial solutions would be useful or  
>>>> not. I would rather prefer to keep the approach simple:
>>>> subtests
>>>> =======
>>>> - PASS/FAIL/WARN that can run normally
>>>> - CANNOTTELL if there is missing information
>>>> tests
>>>> ====
>>>> - PASS/FAIL/WARN that can run normally
>>>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL"
>>>> But do you still think it's important to have PARTIAL results?
>>>
>>> At the subtest level, there aren't so many subtests that are  
>>> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2  
>>> and -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is  
>>> simply too big, because that's a core mobile limitation. That  
>>> being said, it may be added in a second time.
>>>
>>> At the test level, I just think that we lose the ability to  
>>> quickly point out the outcome of the test. You'll basically end  
>>> up with the following in each and every check run on a file  
>>> document:
>>> <tests outcome="CANNOTTELL">
>>>  <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL">
>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>  </test>
>>>  <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL">
>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>  </test>
>>>  <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL">
>>>   [list of FAIL/WARN/CANNOTTELL results]
>>>  </test>
>>>  [...]
>>> </tests>
>>>
>>> Whilst totally correct, the overall outcome and the outcome of  
>>> the tests mentioned above doesn't tell you whether there is a  
>>> FAIL in one of the subtests or not. Sure enough, this can be  
>>> sorted out by having a look at the list of <result />, but it's  
>>> the same thing today: the outcome attribute is more a "visual"  
>>> clue than a computational need (i.e. its value can be re-computed  
>>> at any time by having a look at the list of results elements). By  
>>> limiting ourselves to CANNOTTELL, we're dropping that "visual"  
>>> clue: any report on a file will need to be parsed to see how many  
>>> of the tests that could be run failed.
>>>
>>> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL  
>>> pretty easily, but I guess I just like the idea to still have a  
>>> notion of PASS/FAIL at the test and overall outcome level.
>>>
>>> Or...
>>>
>>> Another possibility is to have FAIL override CANNOTTELL. In other  
>>> words compute the test outcome as:
>>>  If any of the subtests outcome is FAIL,
>>>    then test outcome is FAIL
>>>  else if any of the subtests outcome is CANNOTTELL,
>>>    then test outcome is CANNOTTELL
>>>  else
>>>    test outcome is PASS
>>> ... and about the same thing at the overall outcome level.
>>>
>>> Tools would still have to go through the list of results to tell  
>>> which tests were only partially run, but at least, looking at a  
>>> CANNOTTELL would tell you that "the thing looks great so far  
>>> although not everything could be checked", while FAIL would keep  
>>> its "something's wrong" meaning. It does work well provided that  
>>> a FAIL in a file:// case always imply a FAIL in a http:// case,  
>>> otherwise we would just raise a red flag that isn't a real one. I  
>>> think that's almost true, except the uncertainty about <object>  
>>> elements for computing included resources when files are handled.  
>>> I can live with that.
>>>
>>> This could even be applied to subtests, and the net result is  
>>> that we don't have to define more outcome values that carry  
>>> orthogonal meanings.
>>>
>>> What about it?
>>>
>>> [...]
>>>
>>> Francois.
>>>
>

Received on Monday, 16 February 2009 09:23:57 UTC