Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

Hi Francois,

I am sorry but I am a bit confused about what you are suggesting : 
( Are we talking about the same approach? so in summary...

Francois wrote:
> Another possibility is to have FAIL override CANNOTTELL. In other  
> words compute the test outcome as:
>  If any of the subtests outcome is FAIL,
>    then test outcome is FAIL
>  else if any of the subtests outcome is CANNOTTELL,
>    then test outcome is CANNOTTELL
>  else
>    test outcome is PASS
> ... and about the same thing at the overall outcome level.



or are you suggesting that we go for your original suggestion by  
including PARTIAL results?

> In short, the possible outcomes for a subtest (the <result> element  
> in the XML report) would be:
>  - PASS, WARN, FAIL for subtests that can be run normally.
>  - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can  
> only be applied partially.
>  - CANNOTTELL for subtests that simply can't be run.
>
> The possible outcomes for a test would be:
>  - PASS, FAIL for tests that can be completely checked
>  - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or  
> CANNOTTELL in one of the subtests
>  - CANNOTTELL when none of the subtests could be run (e.g. for  
> CACHING)
>
> The possible overall outcomes would be:
>  - PASS, FAIL when all tests can be completely checked (http/https  
> case)
>  - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or  
> CANNOTTELL in one of the tests
>

Regards,
Yeliz.




On 13 Feb 2009, at 13:57, Francois Daoust wrote:

>
> Yeliz Yesilada wrote:
> [...]
>>>
>>> 3/ I think there is a useful distinction to be made between a  
>>> subtest that can't be run because some data is missing, and a  
>>> subtest that can't be run because it doesn't need to, i.e. if  
>>> there are no objects in the page, the OBJECT_OR_SCRIPTS subtests  
>>> de facto pass. The first possibility is what we're talking about.  
>>> The second possibility may be of some use in the future (I'm not  
>>> suggesting we implement it right now). In short, I would rather  
>>> keep NOT_APPLICABLE to the second case, and use DATA_MISSING (I  
>>> can't think of a better proposal, but the idea is to point out  
>>> that the moki representation is incomplete) for checks on files.
>> I agree. I think what Dominique suggested is a good idea: using  
>> "cannotTell".
>
> Yes, good idea, thanks dom!
>
>
> [...]
>> I think we need to think "why one would like to know if a *sub- 
>> test* passed partially or not?". For example, in our application  
>> if a sub-test can only be checked partially, then we have to use  
>> the Tester (URI) version to check that again so it's enough to  
>> know that the a particular sub-test cannot be tested.
>> I am just not sure if these partial solutions would be useful or  
>> not. I would rather prefer to keep the approach simple:
>> subtests
>> =======
>> - PASS/FAIL/WARN that can run normally
>> - CANNOTTELL if there is missing information
>> tests
>> ====
>> - PASS/FAIL/WARN that can run normally
>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL"
>> But do you still think it's important to have PARTIAL results?
>
> At the subtest level, there aren't so many subtests that are  
> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2  
> and -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is  
> simply too big, because that's a core mobile limitation. That being  
> said, it may be added in a second time.
>
> At the test level, I just think that we lose the ability to quickly  
> point out the outcome of the test. You'll basically end up with the  
> following in each and every check run on a file document:
> <tests outcome="CANNOTTELL">
>  <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL">
>   [list of FAIL/WARN/CANNOTTELL results]
>  </test>
>  <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL">
>   [list of FAIL/WARN/CANNOTTELL results]
>  </test>
>  <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL">
>   [list of FAIL/WARN/CANNOTTELL results]
>  </test>
>  [...]
> </tests>
>
> Whilst totally correct, the overall outcome and the outcome of the  
> tests mentioned above doesn't tell you whether there is a FAIL in  
> one of the subtests or not. Sure enough, this can be sorted out by  
> having a look at the list of <result />, but it's the same thing  
> today: the outcome attribute is more a "visual" clue than a  
> computational need (i.e. its value can be re-computed at any time  
> by having a look at the list of results elements). By limiting  
> ourselves to CANNOTTELL, we're dropping that "visual" clue: any  
> report on a file will need to be parsed to see how many of the  
> tests that could be run failed.
>
> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL  
> pretty easily, but I guess I just like the idea to still have a  
> notion of PASS/FAIL at the test and overall outcome level.
>
> Or...
>
> Another possibility is to have FAIL override CANNOTTELL. In other  
> words compute the test outcome as:
>  If any of the subtests outcome is FAIL,
>    then test outcome is FAIL
>  else if any of the subtests outcome is CANNOTTELL,
>    then test outcome is CANNOTTELL
>  else
>    test outcome is PASS
> ... and about the same thing at the overall outcome level.
>
> Tools would still have to go through the list of results to tell  
> which tests were only partially run, but at least, looking at a  
> CANNOTTELL would tell you that "the thing looks great so far  
> although not everything could be checked", while FAIL would keep  
> its "something's wrong" meaning. It does work well provided that a  
> FAIL in a file:// case always imply a FAIL in a http:// case,  
> otherwise we would just raise a red flag that isn't a real one. I  
> think that's almost true, except the uncertainty about <object>  
> elements for computing included resources when files are handled. I  
> can live with that.
>
> This could even be applied to subtests, and the net result is that  
> we don't have to define more outcome values that carry orthogonal  
> meanings.
>
> What about it?
>
> [...]
>
> Francois.
>

Received on Monday, 16 February 2009 08:11:52 UTC