Re: Presentation: Multi-spec test suites

At 04:37 PM 3/1/2005 -0500, Patrick Curran wrote:
>Here's my first cut at tomorrow's presentation. I only have 5-7 minutes, 
>so it has to be pretty short.
>
>Whaddya think?

Good and thorough.  Probably the main danger is too much for 5-7 
minutes.  But if you skip through it lightly, then the detailed slides are 
available for people to look at later.  I like that approach.

I didn't have time for careful critical reading, but will try to do so later.

-Lofton.




>
>Background
>
>    * Modularity of specifications is a Good Thing
>    * Specs can be re-used (eg, XSLT and XForms use XPath as their 
> expression language)
>    * There are issues (for spec authors)
>        * Component specs may advance at different rates
>        * Need to be explicit about versions of referenced specs
>        * Clearly state what is required and what is optional
>            * Excessive optionality can reduce interoperability
>    * Can we also create modular, reusable test suites?
>
>
>
>The Complexity Continuum
>
>    * Simple: one spec in a single document (no external references)
>        * Develop tests for what's specified in the document
>    * More complex: one spec in multiple documents (no external references)
>        * Develop tests for what's specified in the documents
>        * Beware...
>            * Advance the docs together where inter-dependencies exist
>            * Don't simultaneously support/promote different versions of 
> one doc

I'm unsure exactly what this means.  How would you support different 
versions of a document?  So Module B v1.0 is in /TR/ and
>    * Create an umbrella spec that explicitly states the versions of 
> component spec

The implication here is that umbrella must publish/republish every time a 
component is published or republished.  Although that is sensible (else 
there really isn't an umbrella spec), I expect some might object.
>    * Complex: your spec references an "external" spec (created by another 
> group)
>        * You still need an umbrella spec
>        * Can you assume that the included functionality is tested (or out 
> of scope), or must you test it?
>        * If you must test it, should you test By reference, or test By 
> value?
>            * "Pass my tests and pass their tests too"
>                * Assumes there is an explicit, versioned test suite you 
> can point to
>            * Or (more likely) incorporate external tests into your test 
> suite
>                * Hopefully the tests were developed with re-use in mind
>
>
>
>Developing tests for re-use
>
>    * Create a test suite and not simply a random collection of tests
>        * Test, package, document, version, deal with errata
>    * State explicitly what versions of what specs the test suite addresses
>    * Indicate what portion of the spec each test addresses
>        * Do these tests apply to me? (I may not have implemented optional 
> features)
>    * Provide data indicating how thoroughly the tests cover the spec
>        * How much confidence should implementors have if they "pass"?
>        * Do I need to augment these tests with others?
>    * Explain how to execute the tests and how to create a test harness
>        * Provide appropriate metadata
>        * Define expected results
>    * Report results in a consistent and useful format
>        * How should I interpret the results? Did I pass?
>        * If I didn't, what went wrong?
>    * Explain how to challenge the validity of a test or submit a bug report
>
>
>
>Multiple specifications and the Java Platform
>
>    * J2SE Platform (umbrella) Spec references multiple standalone specs
>        * J2SE 6.0 spec has just been published at 
> http://jcp.org/en/jsr/detail?id=270
>    * Some of these (not all) have an independent existence - and their 
> own test suites
>        * XML Digital Signature
>        * JDBC (database connectivity)
>        * JAXB (XML binding)
>        * JAX-RPC (SOAP)
>        * JAXP (XML parsing)
>    * Several of these in turn reference other specs (often from W3C)
>    * Subsetting and supersetting are strongly discouraged to preserve 
> interoperability
>        * Must (usually) implement all component specs
>        * Must (usually) implement the same version of all component specs
>
>
>
>Multiple test suites and the Java Platform
>
>    * All Sun-developed test suites use the same test harness
>    * Some differences nevertheless exist
>        * Different build mechanisms, test metadata formats
>        * Different execution environments (J2EE, J2SE, J2ME)
>    * Currently we test By Value
>        * Incorporating tests is (relatively) simple, since we develop to 
> (relatively) common standards
>    * Considering testing By Reference due to rapidly increasing size of 
> the platform test suites
>        * Need to improve the user-experience when multiple test suites 
> are to be executed
>    * We also incorporate W3C tests (eg, for XML Core, Schema)
>        * We'd really like to us By Reference for W3C specs
>            * Plug in any parser you want, so long as it passes version 
> x.y of the W3C parser test suite, located <here>
>