Re: [testtwf-shenzhen] Need your input: Writing Webapps tests at TestTWF

On 06/09/13 22:14, Rebecca Hauck wrote:
>
>
> On 9/6/13 7:48 AM, "Ms2ger" <ms2ger@gmail.com> wrote:
>
>> On 09/06/2013 03:54 PM, Arthur Barstow wrote:
>>> * File API - this spec should soon enter Last Call Working Draft. It has
>>> some submissions
>>>
>>> <http://w3c-test.org/web-platform-tests/master/FileAPI/tests/submissions/
>>>>
>>> (PRs #53 and #54), at least six PRs from TTWF-Shanghai and PubStatus
>>> says it is "~20% complete". KrisK is the spec's Test Facilitator so it
>>> would be good to get his input re the priority tasks and plans for this
>>> test suite. For example, it's not clear if the existing coverage is
>>> reasonably good, where are the holes, etc.
>>
>> I don't know if anyone has a good idea of what coverage we have right
>> now, and what the outstanding PRs will add, so I suspect that writing
>> new tests right now is more likely to duplicate effort than to add
>> useful coverage. If someone wanted to spend the time to figure that out,
>> I think that would be more useful.
>
> Can each of the Test Facilitators provide a coverage assessment for their
> specs?  Any volunteers? The coverage tool [1] would be useful here as a
> starting point. Tobie has report hosted somewhere on w3.org, but I'm
> pretty sure the JSON data behind it is static and needs to be refreshed.
> Tobie, if you have anything to say here, chime in. Like maybe should be
> host the coverage reports on testthewebforward.org now?

FWIW I think that we are at a stage where the only reasonable answers to 
"do we have enough coverage for this spec" are "no" and "don't know". We 
just don't have the experience or methodology at the moment to be at all 
definite with this kind of question. Even where we think we have good 
coverage based on counting tests, or matching tests to normative 
statements, it is not uncommon to find bugs that only occur when some 
combination of conditions are met, or when a feature interacts with some 
other part of the platform, or where the spec has changed since the 
tests were written.

Received on Sunday, 8 September 2013 09:49:47 UTC