Re: Subtests that apply to HTTP header fields [was: Re: mobileOK validation logic - jar file?]

Hi Francois,

Please see my comments below.`

On 12 Feb 2009, at 17:43, Francois Daoust wrote:

> 1/ The URI of the file resources in the moki would start with  
> "file://", that's probably enough to identify the file vs. http/ 
> https case but we may need a more dedicated attribute to ease  
> checks in XSLT stylesheets. I suggest we try without for the time  
> being.

OK. We can try and if gets complicated, we can look into adding an  
attribute.

> 2/ Not to have to create a separate set of XSLT stylesheets for  
> files, we might need to produce a fake HTTPResponse in the moki  
> representation anyway, that would actually be the result of the  
> file retrieval.

Good idea.

>
> 3/ I think there is a useful distinction to be made between a  
> subtest that can't be run because some data is missing, and a  
> subtest that can't be run because it doesn't need to, i.e. if there  
> are no objects in the page, the OBJECT_OR_SCRIPTS subtests de facto  
> pass. The first possibility is what we're talking about. The second  
> possibility may be of some use in the future (I'm not suggesting we  
> implement it right now). In short, I would rather keep  
> NOT_APPLICABLE to the second case, and use DATA_MISSING (I can't  
> think of a better proposal, but the idea is to point out that the  
> moki representation is incomplete) for checks on files.

I agree. I think what Dominique suggested is a good idea: using  
"cannotTell".

> 4/ You seem to imply that we'll just ignore entire tests as soon as  
> one of the subtest cannot be run. It works with caching, but some  
> other subtests could be quite useful (e.g.  
> CHARACTER_ENCODING_SUPPORT-5 that checks whether the document is  
> valid UTF-8 would be useful even though  
> CHARACTER_ENCODING_SUPPORT-1 cannot be checked). Some other  
> subtests could be useful as well even though they can only apply  
> partially (e.g. EXTERNAL_RESOURCES-2 is still worth raising when  
> the document targets too many external resources, even though the  
> total is not 100% correct without HTTP redirections).
>
> In short, the possible outcomes for a subtest (the <result> element  
> in the XML report) would be:
>  - PASS, WARN, FAIL for subtests that can be run normally.
>  - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can  
> only be applied partially.
>  - DATA_MISSING for subtests that simply can't be run.
> I don't think that creates a real complexity or requires a lot more  
> changes, because we'll need to update the corresponding XSLT  
> stylesheets and handle these cases anyway.
>
> (Note I had suggested to use a different attribute to flag the  
> results as "partial", but I think that your approach requires fewer  
> changes and actually helps "seeing" that the outcome is not a  
> definitive response).
>
> The possible outcomes for a test would be:
>  - PASS, FAIL for tests that can be completely checked
>  - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or  
> DATA_MISSING in one of the subtests
>  - DATA_MISSING when none of the subtests could be run (e.g. for  
> CACHING)
>
> The possible overall outcomes would be:
>  - PASS, FAIL when all tests can be completely checked (http/https  
> case)
>  - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or  
> DATA_MISSING in one of the tests
>  [- DATA_MISSING is not going to ever occur, since it would mean  
> none of the tests could actually be run at all]

I think we need to think "why one would like to know if a *sub-test*  
passed partially or not?". For example, in our application if a sub- 
test can only be checked partially, then we have to use the Tester  
(URI) version to check that again so it's enough to know that the a  
particular sub-test cannot be tested.

I am just not sure if these partial solutions would be useful or not.  
I would rather prefer to keep the approach simple:

subtests
=======
- PASS/FAIL/WARN that can run normally
- CANNOTTELL if there is missing information

tests
====
- PASS/FAIL/WARN that can run normally
- CANNOTTELL if any of the sub-tests return "CANNOTTELL"

But do you still think it's important to have PARTIAL results?

> 5/ There is one specific case we need to handle carefully: when the  
> primary document is using an HTTP/HTTPS scheme, then we need to  
> return an error when one the linked resources is using the file  
> scheme, whereas when the primary document is using the FILE scheme,  
> we should process other file resources.

OK.

>
> If we do that correctly, the final mobileOK report returned when  
> the Checker is given a real http/https URI will be no different to  
> the one that is returned today and that is good! (i.e. no  
> PARTIAL_blah or DATA_MISSING in the http/https case).

I agree. It's good to have something that is consistent with the  
current approach.

Regards,
Yeliz.

Received on Friday, 13 February 2009 10:46:08 UTC