Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

Yeliz Yesilada wrote:
[...]
>>
>> 3/ I think there is a useful distinction to be made between a subtest 
>> that can't be run because some data is missing, and a subtest that 
>> can't be run because it doesn't need to, i.e. if there are no objects 
>> in the page, the OBJECT_OR_SCRIPTS subtests de facto pass. The first 
>> possibility is what we're talking about. The second possibility may be 
>> of some use in the future (I'm not suggesting we implement it right 
>> now). In short, I would rather keep NOT_APPLICABLE to the second case, 
>> and use DATA_MISSING (I can't think of a better proposal, but the idea 
>> is to point out that the moki representation is incomplete) for checks 
>> on files.
> 
> I agree. I think what Dominique suggested is a good idea: using 
> "cannotTell".

Yes, good idea, thanks dom!


> 
[...]
> I think we need to think "why one would like to know if a *sub-test* 
> passed partially or not?". For example, in our application if a sub-test 
> can only be checked partially, then we have to use the Tester (URI) 
> version to check that again so it's enough to know that the a particular 
> sub-test cannot be tested.
> 
> I am just not sure if these partial solutions would be useful or not. I 
> would rather prefer to keep the approach simple:
> 
> subtests
> =======
> - PASS/FAIL/WARN that can run normally
> - CANNOTTELL if there is missing information
> 
> tests
> ====
> - PASS/FAIL/WARN that can run normally
> - CANNOTTELL if any of the sub-tests return "CANNOTTELL"
> 
> But do you still think it's important to have PARTIAL results?

At the subtest level, there aren't so many subtests that are concerned. 
I think it would be useful to run EXTERNAL_RESOURCES-2 and -3 or 
PAGE_SIZE_LIMIT-2 to alert authors that their page is simply too big, 
because that's a core mobile limitation. That being said, it may be 
added in a second time.

At the test level, I just think that we lose the ability to quickly 
point out the outcome of the test. You'll basically end up with the 
following in each and every check run on a file document:
<tests outcome="CANNOTTELL">
  <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL">
   [list of FAIL/WARN/CANNOTTELL results]
  </test>
  <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL">
   [list of FAIL/WARN/CANNOTTELL results]
  </test>
  <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL">
   [list of FAIL/WARN/CANNOTTELL results]
  </test>
  [...]
</tests>

Whilst totally correct, the overall outcome and the outcome of the tests 
mentioned above doesn't tell you whether there is a FAIL in one of the 
subtests or not. Sure enough, this can be sorted out by having a look at 
the list of <result />, but it's the same thing today: the outcome 
attribute is more a "visual" clue than a computational need (i.e. its 
value can be re-computed at any time by having a look at the list of 
results elements). By limiting ourselves to CANNOTTELL, we're dropping 
that "visual" clue: any report on a file will need to be parsed to see 
how many of the tests that could be run failed.

Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL pretty 
easily, but I guess I just like the idea to still have a notion of 
PASS/FAIL at the test and overall outcome level.

Or...

Another possibility is to have FAIL override CANNOTTELL. In other words 
compute the test outcome as:
  If any of the subtests outcome is FAIL,
    then test outcome is FAIL
  else if any of the subtests outcome is CANNOTTELL,
    then test outcome is CANNOTTELL
  else
    test outcome is PASS
... and about the same thing at the overall outcome level.

Tools would still have to go through the list of results to tell which 
tests were only partially run, but at least, looking at a CANNOTTELL 
would tell you that "the thing looks great so far although not 
everything could be checked", while FAIL would keep its "something's 
wrong" meaning. It does work well provided that a FAIL in a file:// case 
always imply a FAIL in a http:// case, otherwise we would just raise a 
red flag that isn't a real one. I think that's almost true, except the 
uncertainty about <object> elements for computing included resources 
when files are handled. I can live with that.

This could even be applied to subtests, and the net result is that we 
don't have to define more outcome values that carry orthogonal meanings.

What about it?

[...]

Francois.

Received on Friday, 13 February 2009 13:58:35 UTC