- From: Francois Daoust <fd@w3.org>
- Date: Mon, 16 Feb 2009 09:47:15 +0100
- To: Yeliz Yesilada <yesilady@cs.man.ac.uk>
- CC: public-mobileok-checker@w3.org, Kentarou Fukuda <KENTAROU@jp.ibm.com>
Yeliz Yesilada wrote: > Hi Francois, > > I am sorry but I am a bit confused about what you are suggesting :( Yes, I see what you mean, I was mostly thinking aloud, but didn't make any choice ;) I think I convinced myself that the first approach below is both the most satisfactory and the easiest to implement. So I'd go for it. > Are we talking about the same approach? so in summary... > > Francois wrote: >> Another possibility is to have FAIL override CANNOTTELL. In other >> words compute the test outcome as: >> If any of the subtests outcome is FAIL, >> then test outcome is FAIL >> else if any of the subtests outcome is CANNOTTELL, >> then test outcome is CANNOTTELL >> else >> test outcome is PASS >> ... and about the same thing at the overall outcome level. +1 for this one. It's easy to implement and ensures the outcome value still means something: PASS: mobileOK CANNOTTELL: looking fine, but not everything could be checked FAIL: something's wrong (test may not have been run entirely) > > > > or are you suggesting that we go for your original suggestion by > including PARTIAL results? > >> In short, the possible outcomes for a subtest (the <result> element in >> the XML report) would be: >> - PASS, WARN, FAIL for subtests that can be run normally. >> - PARTIAL_PASS, PARTIAL_WARN, PARTIAL_FAIL for subtests that can only >> be applied partially. >> - CANNOTTELL for subtests that simply can't be run. >> >> The possible outcomes for a test would be: >> - PASS, FAIL for tests that can be completely checked >> - PARTIAL_PASS, PARTIAL_FAIL when there is a PARTIAL_* and/or >> CANNOTTELL in one of the subtests >> - CANNOTTELL when none of the subtests could be run (e.g. for CACHING) >> >> The possible overall outcomes would be: >> - PASS, FAIL when all tests can be completely checked (http/https case) >> - PARTIAL_PASS, PARTIAL_FAIL where there is a PARTIAL_* and/or >> CANNOTTELL in one of the tests -1 for that one, since it's more complex and the outcome value would then carry two orthogonal dimensions, which doesn't look that good. Francois. >> > > Regards, > Yeliz. > > > > > On 13 Feb 2009, at 13:57, Francois Daoust wrote: > >> >> Yeliz Yesilada wrote: >> [...] >>>> >>>> 3/ I think there is a useful distinction to be made between a >>>> subtest that can't be run because some data is missing, and a >>>> subtest that can't be run because it doesn't need to, i.e. if there >>>> are no objects in the page, the OBJECT_OR_SCRIPTS subtests de facto >>>> pass. The first possibility is what we're talking about. The second >>>> possibility may be of some use in the future (I'm not suggesting we >>>> implement it right now). In short, I would rather keep >>>> NOT_APPLICABLE to the second case, and use DATA_MISSING (I can't >>>> think of a better proposal, but the idea is to point out that the >>>> moki representation is incomplete) for checks on files. >>> I agree. I think what Dominique suggested is a good idea: using >>> "cannotTell". >> >> Yes, good idea, thanks dom! >> >> >> [...] >>> I think we need to think "why one would like to know if a *sub-test* >>> passed partially or not?". For example, in our application if a >>> sub-test can only be checked partially, then we have to use the >>> Tester (URI) version to check that again so it's enough to know that >>> the a particular sub-test cannot be tested. >>> I am just not sure if these partial solutions would be useful or not. >>> I would rather prefer to keep the approach simple: >>> subtests >>> ======= >>> - PASS/FAIL/WARN that can run normally >>> - CANNOTTELL if there is missing information >>> tests >>> ==== >>> - PASS/FAIL/WARN that can run normally >>> - CANNOTTELL if any of the sub-tests return "CANNOTTELL" >>> But do you still think it's important to have PARTIAL results? >> >> At the subtest level, there aren't so many subtests that are >> concerned. I think it would be useful to run EXTERNAL_RESOURCES-2 and >> -3 or PAGE_SIZE_LIMIT-2 to alert authors that their page is simply too >> big, because that's a core mobile limitation. That being said, it may >> be added in a second time. >> >> At the test level, I just think that we lose the ability to quickly >> point out the outcome of the test. You'll basically end up with the >> following in each and every check run on a file document: >> <tests outcome="CANNOTTELL"> >> <test name="CHARACTER_ENCODING_SUPPORT" outcome="CANNOTTELL"> >> [list of FAIL/WARN/CANNOTTELL results] >> </test> >> <test name="CONTENT_FORMAT_SUPPORT" outcome="CANNOTTELL"> >> [list of FAIL/WARN/CANNOTTELL results] >> </test> >> <test name="OBJECTS_OR_SCRIPT" outcome="CANNOTTELL"> >> [list of FAIL/WARN/CANNOTTELL results] >> </test> >> [...] >> </tests> >> >> Whilst totally correct, the overall outcome and the outcome of the >> tests mentioned above doesn't tell you whether there is a FAIL in one >> of the subtests or not. Sure enough, this can be sorted out by having >> a look at the list of <result />, but it's the same thing today: the >> outcome attribute is more a "visual" clue than a computational need >> (i.e. its value can be re-computed at any time by having a look at the >> list of results elements). By limiting ourselves to CANNOTTELL, we're >> dropping that "visual" clue: any report on a file will need to be >> parsed to see how many of the tests that could be run failed. >> >> Any tool can compute the corresponding PARTIAL_PASS/PARTIAL_FAIL >> pretty easily, but I guess I just like the idea to still have a notion >> of PASS/FAIL at the test and overall outcome level. >> >> Or... >> >> Another possibility is to have FAIL override CANNOTTELL. In other >> words compute the test outcome as: >> If any of the subtests outcome is FAIL, >> then test outcome is FAIL >> else if any of the subtests outcome is CANNOTTELL, >> then test outcome is CANNOTTELL >> else >> test outcome is PASS >> ... and about the same thing at the overall outcome level. >> >> Tools would still have to go through the list of results to tell which >> tests were only partially run, but at least, looking at a CANNOTTELL >> would tell you that "the thing looks great so far although not >> everything could be checked", while FAIL would keep its "something's >> wrong" meaning. It does work well provided that a FAIL in a file:// >> case always imply a FAIL in a http:// case, otherwise we would just >> raise a red flag that isn't a real one. I think that's almost true, >> except the uncertainty about <object> elements for computing included >> resources when files are handled. I can live with that. >> >> This could even be applied to subtests, and the net result is that we >> don't have to define more outcome values that carry orthogonal meanings. >> >> What about it? >> >> [...] >> >> Francois. >> > >
Received on Monday, 16 February 2009 08:48:01 UTC