- From: Jo Rabin <jrabin@mtld.mobi>
- Date: Mon, 02 Mar 2009 21:39:05 +0000
- To: Yeliz Yesilada <yesilady@cs.man.ac.uk>
- CC: Francois Daoust <fd@w3.org>, public-mobileok-checker@w3.org, Kentarou Fukuda <KENTAROU@jp.ibm.com>
I'm sorry to pick up on this so belatedly. I Generally agree with the thrust of the thread but given the specific wording in mobilOK Basic Tests - e.g. 1.2 Applicability, 2.2 Validity of the Tests and 2.3 Testing outcomes - whatever is being discussed here is not mobileOK and we know (and love) it. If there is scope for off-line testing then I think it might be useful for a WG note on the topic. Jo Yeliz Yesilada wrote: > Hi Francois, > > Thanks for the clarification. I also think the first approach is better. > If everybody agrees with this, we will go for that approach. > > Yeliz. > On 16 Feb 2009, at 08:47, Francois Daoust wrote: > >> Yeliz Yesilada wrote: >>> Hi Francois, >>> I am sorry but I am a bit confused about what you are suggesting :( >> >> Yes, I see what you mean, I was mostly thinking aloud, but didn't make >> any choice ;) >> >> I think I convinced myself that the first approach below is both the >> most satisfactory and the easiest to implement. So I'd go for it. >> >> >>> Are we talking about the same approach? so in summary... >> >>> Francois wrote: >>>> Another possibility is to have FAIL override CANNOTTELL. In other >>>> words compute the test outcome as: >>>> If any of the subtests outcome is FAIL, >>>> then test outcome is FAIL >>>> else if any of the subtests outcome is CANNOTTELL, >>>> then test outcome is CANNOTTELL >>>> else >>>> test outcome is PASS >>>> ... and about the same thing at the overall outcome level. >> >> +1 for this one. It's easy to implement and ensures the outcome value >> still means something: >> PASS: mobileOK >> CANNOTTELL: looking fine, but not everything could be checked >> FAIL: something's wrong (test may not have been run entirely) >> >> >>> or are you suggesting that we go for your original suggestion by >>> including PARTIAL results? >>>> In short, the possible outcomes for a subtest (the <result> element >>>> in the XML report) would be: >>>> - PASS, WARN, FAIL for subtests that can be run normally. >>>> - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can >>>> only be applied partially. >>>> - CANNOTTELL for subtests that simply can't be run. >>>> >>>> The possible outcomes for a test would be: >>>> - PASS, FAIL for tests that can be completely checked >>>> - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or >>>> CANNOTTELL in one of the subtests >>>> - CANNOTTELL when none of the subtests could be run (e.g. for CACHING) >>>> >>>> The possible overall outcomes would be: >>>> - PASS, FAIL when all tests can be completely checked (http/https >>>> case) >>>> - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or >>>> CANNOTTELL in one of the tests >> >> -1 for that one, since it's more complex and the outcome value would >> then carry two orthogonal dimensions, which doesn't look that good. >> >> Francois. >> >>>> >>> Regards, >>> Yeliz. >>> On 13 Feb 2009, at 13:57, Francois Daoust wrote: >>>> >>>> Yeliz Yesilada wrote: >>>> [...] >>>>>> >>>>>> 3/ I think there is a useful distinction to be made between a >>>>>> subtest that can't be run because some data is missing, and a >>>>>> subtest that can't be run because it doesn't need to, i.e. if >>>>>> there are no objects in the page, the OBJECT_OR_SCRIPTS subtests >>>>>> de facto pass. The first possibility is what we're talking about. >>>>>> The second possibility may be of some use in the future (I'm not >>>>>> suggesting we implement it right now). In short, I would rather >>>>>> keep NOT_APPLICABLE to the second case, and use DATA_MISSING (I >>>>>> can't think of a better proposal, but the idea is to point out >>>>>> that the moki representation is incomplete) for checks on files. >>>>> I agree. I think what Dominique suggested is a good idea: using >>>>> "cannotTell". >>>> >>>> Yes, good idea, thanks dom! >>>> >>>> >>>> [...] >>>>> I think we need to think "why one would like to know if a >>>>> *sub-test* passed partially or not?". For example, in our >>>>> application if a sub-test can only be checked partially, then we >>>>> have to use the Tester (URI) version to check that again so it's >>>>> enough to know that the a particular sub-test cannot be tested. >>>>> I am just not sure if these partial solutions would be useful or >>>>> not. I would rather prefer to keep the approach simple: >>>>> subtests >>>>> ======= >>>>> - PASS/FAIL/WARN that can run normally >>>>> - CANNOTTELL if there is missing information >>>>> tests >>>>> ==== >>>>> - PASS/FAIL/WARN that can run normally >>>>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL" >>>>> But do you still think it's important to have PARTIAL results? >>>> >>>> At the subtest level, there aren't so many subtests that are >>>> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2 >>>> and -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is >>>> simply too big, because that's a core mobile limitation. That being >>>> said, it may be added in a second time. >>>> >>>> At the test level, I just think that we lose the ability to quickly >>>> point out the outcome of the test. You'll basically end up with the >>>> following in each and every check run on a file document: >>>> <tests outcome="CANNOTTELL"> >>>> <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL"> >>>> [list of FAIL/WARN/CANNOTTELL results] >>>> </test> >>>> <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL"> >>>> [list of FAIL/WARN/CANNOTTELL results] >>>> </test> >>>> <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL"> >>>> [list of FAIL/WARN/CANNOTTELL results] >>>> </test> >>>> [...] >>>> </tests> >>>> >>>> Whilst totally correct, the overall outcome and the outcome of the >>>> tests mentioned above doesn't tell you whether there is a FAIL in >>>> one of the subtests or not. Sure enough, this can be sorted out by >>>> having a look at the list of <result />, but it's the same thing >>>> today: the outcome attribute is more a "visual" clue than a >>>> computational need (i.e. its value can be re-computed at any time by >>>> having a look at the list of results elements). By limiting >>>> ourselves to CANNOTTELL, we're dropping that "visual" clue: any >>>> report on a file will need to be parsed to see how many of the tests >>>> that could be run failed. >>>> >>>> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL >>>> pretty easily, but I guess I just like the idea to still have a >>>> notion of PASS/FAIL at the test and overall outcome level. >>>> >>>> Or... >>>> >>>> Another possibility is to have FAIL override CANNOTTELL. In other >>>> words compute the test outcome as: >>>> If any of the subtests outcome is FAIL, >>>> then test outcome is FAIL >>>> else if any of the subtests outcome is CANNOTTELL, >>>> then test outcome is CANNOTTELL >>>> else >>>> test outcome is PASS >>>> ... and about the same thing at the overall outcome level. >>>> >>>> Tools would still have to go through the list of results to tell >>>> which tests were only partially run, but at least, looking at a >>>> CANNOTTELL would tell you that "the thing looks great so far >>>> although not everything could be checked", while FAIL would keep its >>>> "something's wrong" meaning. It does work well provided that a FAIL >>>> in a file:// case always imply a FAIL in a http:// case, otherwise >>>> we would just raise a red flag that isn't a real one. I think that's >>>> almost true, except the uncertainty about <object> elements for >>>> computing included resources when files are handled. I can live with >>>> that. >>>> >>>> This could even be applied to subtests, and the net result is that >>>> we don't have to define more outcome values that carry orthogonal >>>> meanings. >>>> >>>> What about it? >>>> >>>> [...] >>>> >>>> Francois. >>>> >> > >
Received on Monday, 2 March 2009 21:39:51 UTC