Re: Testable assertion tagging for W3C specifications

Alex Rousskov <rousskov@measurement-factory.com> wrote:
> In other words, "WHAT we are really pointing at" should be outside of
> the pointing (addressing) scheme both because we cannot predict all
> things that people will point at AND because there is no practical
> need in making such predictions. The smarts should be in the test
> suite or elsewhere; the addressing scheme should be simple.

There is probably need for both.  Of course a spec author can't cover all
the test cases or "complete thoughts" with markup.  On the other hand, I
distrust external "xpointers" as fragile, and they are certainly not part
of the semantic content expressed by the original author.  The abstract XML
content can and should have certain areas flagged as noteable for testing
and as distinct thoughts.  It is just another form of communication of the
author's original intent.  It should be able to help spec authors think in
terms of testing structure, just as they often have to think in terms of
presentation structure, and make it easier to create test suites.

Make no mistake.  Creating a complete test suite for a spec is hard, and
takes a long time (the test suite for XSLT is still not cooked, and there's
a good chance it won't be by the time XSLT 2.0 is released).  The earlier
this process begins, the faster it can be completed (I hope).  And the time
to start thinking about testing is when design is being wrought.  Just like
good software program should include testing as one of the design factors,
so should a software language specification.

While some seem to think that some added structure for use by testers would
be a burden for spec writers, I tend to think of it as a helpful part of
the specification skeleton, that will help to hold it up over the years.

These are just tools that are being suggested.  At least from my viewpoint,
their use by spec authors are optional, and any presented design should be
very lightweight.  But it would be nice if a well thought out scheme could
be designed and experimented with, and, if it works, evolved over the
coming years.

-scott




                                                                                                                                             
                      Alex Rousskov                                                                                                          
                      <rousskov@measurement-f        To:       David Marston/Cambridge/IBM <david_marston@us.ibm.com>                        
                      actory.com>                    cc:       www-qa@w3.org, Scott Boag/Cambridge/IBM <scott_boag@us.ibm.com>               
                                                     Subject:  Re: Testable assertion tagging for W3C specifications                         
                      05/29/2002 06:09 PM                                                                                                    
                                                                                                                                             
                                                                                                                                             




On Wed, 29 May 2002, David Marston/Cambridge/IBM wrote:

> I want it to address them in a way that recognizes that they *are*
> productions, assertions, or whatever.

Why? What is the rationale for recognizing one "whatever" from another
"whatever" and, hence, limiting the number of "whatever"s and the
relationships among them? Can we abstract the model enough so that we
do not depend on today's "whatever" definition? I think we can.

> Based on your verbiage above, I gather that you are referring to a
> text range, delimited at both start and end.

Yes, a single "address" could be a text range or an XML fragment.

> Now, let's look at typical normative source under the current
> practice. Quoting section 7.7 of XSLT:
>
> <li>[...]</li>
>
> You have to go outside the enclosing <ul> and read prose to
> realize that all the above is only applicable *if* there is no
> "value" attribute set, so no sentence in the above can be quoted
> standalone.

Sure. A single test case (or whoever is quoting the specs), can use a
set or sequence of addresses to highlight/extract relevant pieces. I
think I mentioned that possibility on the list. Whether a set or a
sequence of addresses is itself an address is an open question (and
not a terribly difficult/important one, IMO).

> But you can't even quote sentences without resorting to
> substring() and relying on non-varying character counts. What you
> can isolate is a whole text node or a <code> element, so you can
> get ungainly sequences with dangling sentence fragments. For
> example, to specify the interaction of level="any" and specified
> from, without just quoting the entire <p>, you get: " axes). If
> the <code>from</code> attribute is specified, then only nodes
> after the first node before the current node that match the
> <code>from</code> pattern are considered." The above is 5
> sequential descendants of the <p>: text()[5] through text()[7].

A good addressing scheme should allow for quoting of arbitrary
sequential text. Thus, it would not leave " axes)." string in the
above example. It would address exactly what you want.

> As I mentioned earlier in this thread, this is about more than
> just a one-way link. We should try to identify the passage as
> something higher-level than just text()[5] through text()[7]. We
> should be able to say that we are pointing at the sentence that
> describes the interaction of two attributes.

A good design would separate the two aspects: extraction and
rendering/presenting/processing in a meaningful way. For example, when
a user clicks on a test case description, the description may include
some readable narrative _and_ precise quote(s) from the Recommendation
that, together, give user a good idea on what is being tested (among
other details).

In other words, "WHAT we are really pointing at" should be outside of
the pointing (addressing) scheme both because we cannot predict all
things that people will point at AND because there is no practical
need in making such predictions. The smarts should be in the test
suite or elsewhere; the addressing scheme should be simple.

You are essentially arguing that we need not simply pointers/addresses
but pointers with types. Introducing "pointer types" is only useful if
you start manipulating with pointers of different types (i.e., doing
address arithmetic for pieces of a Recommendation) -- hardly on the
horizon for most test suites!

> I mentioned before about discerning that two tests address the
> same interaction of attributes. If that is too adventurous for
> now, would you at least agree that we ought to be able to take a
> collection of tests, follow their citation links, and see how much
> of the verbiage is covered by the tests? (Picture a Recommendation
> with all "covered" sentences having a light green background. You
> could eye-scan the Rec, applying human intelligence, to see which
> of the uncovered portions look like testable sentences.)

Yes, absolutely! Moreover, my team is working on that kind of rendering
of RFC 2616 for our test suite. As far as I can see, the proposed
simple addressing approach is sufficient to build a Coverage Map given
a set of addresses derived from a set of test cases.

> > My understanding is that no tag taxonomy is needed to address pieces
> > of text or XML. We are not talking about the process of automatically
> > locating assertions in a random Recommendation, are we? While fun,
> > that task seems to have little utility.
>
> This is the pivot of our disagreement. To me, "addressing pieces of
> text" is the activity that has little utility.

I have formulated my requirements, which were derived from my test
suite needs and my understanding of other's needs. I think that if I
can address any arbitrary piece of a Rec, I can do everything I need.
I can specify more detailed requirements (e.g., persistency of
addresses), but I do not want to waste your time until we agree on the
principle.

What are your requirements? If we start from requirements, we can come
to an agreement much sooner. So far, a simple "piece extraction"
scheme seems to cover all your explicit needs.

> We *may* want to locate assertions in any Rec, but (as you say) we
> are more likely to want to locate assertions in designated Recs.
> It's more than just test suites in the overall plan, though. We
> already have some Recs citing other Recs down to the level of
> individual productions, which is beneficial. Recs should be
> citable at the granularity of any sentence, equation, or similar
> construct that normatively expresses a complete thought.

Exactly. I think that the proposed approach can be used to implement
all of the above. If you can address any arbitrary piece of a
Recommendation, then you should be able to address any complete
thought, provided the Recommendation has complete thoughts.

On the other hand, once you start introducing artificial barriers
such as hard-coded tags/types, you will run into problems of citing
thoughts that cross those artificial barriers (e.g., a paragraph can
contain many interleaved thoughts in an unstructured format)!

Alex.

Received on Wednesday, 29 May 2002 22:03:44 UTC