Re: [widget-digsig] Test assertions

On Tue, Sep 29, 2009 at 9:51 AM, Dominique Hazael-Massieux <dom@w3.org> wrote:
> Hi Marcos,
>
> As Kai alluded to in his report [1], we had a chance to look at Widgets
> Digital Signature last week to see what would be required to create test
> cases for that specification.
>
> As part of that exploratory work, we started two documents similar to
> the ones that were developed for P&C:
>  * a test suite edition of the spec, at:
> http://dev.w3.org/2006/waf/widgets-digsig/Overview_TSE.html

Great.

> It marks up 17 test assertions for user agents
>  * a test plan document where these test assertions appear,
> automatically extracted:
> http://dev.w3.org/2006/waf/widgets-digsig/tests/

Fantastic.

> We discussed (but haven't documented yet) that the test cases for DigSig
> would be of two main types:
>  * the ones testing the proper parsing of the signatures files, similar
> in the work done for config.xml in P&C
>  * the ones that focus on the actual hash/signature validation
> algorithms
>
> Kai took an action item [3] to start working on tests cases; that said,
> as I was the one working on marking up test assertions in the
> non-official test-suite-edition of DigSig, I noticed that DigSig seems
> much less testing-ready than P&C is (thanks to the huge efforts you've
> put in the TSE for that spec).

Yes, we've learned some hard lessons from P&C. It was a HUGE mistake
to go to CR without a test suite. It won't happen again for any future
Widget spec (well, not without a formal objection from me).

> For instance, DigSig considers signature files as class of products,
> where as these aspects would be better considered under either the
> generic user agent or the conformance checker angle; as a result, many
> of the MUST in the specs can't easily be linked to a test case in the
> current state of the spec - I only marked up the 17 ones that were
> fairly clearly testable.

Right. We need to convert those to statements of fact.

> Are you considering putting the same kind of work in DigSig as you did
> in P&C to ease the testing phase?

Yes, I assumed I would have to. I want to finish P&C first, however.
It's super close to being done.

> Could you look into the existing 17
> assertions as a starting point to see if they reflect realistically the
> expected behavior of a user agent?

Yes. I will do that. But I need a few days.

> Should you start working on a TSE for digSig, it would be great if you
> could keep the same test assertions ids I've started to use (although
> given their small number at this time, it wouldn't be a big deal if you
> choose not to); note that I opted to use two-letters longs ids (e.g.
> ta-aa, ta-ab), rather than the 8-random-letters-long ones you picked for
> P&C that made up for interesting discussions last week :) [2]

Again, more lessons learned - won't happen again. I will restrict to 2
characters, that's still a massive address space. Apologies again, mia
culpa.





-- 
Marcos Caceres
http://datadriven.com.au

Received on Tuesday, 29 September 2009 10:41:56 UTC