Feeding the dogs

'But the woman came and did him homage, saying, "Lord, help me."  He
said in reply, "It is not right to take the food of the children and
throw it to the dogs."  She said, "Please, Lord, for even the dogs eat
the scraps that fall from the table of their masters."'  (Mt 15:25-27)

A recent incident involving a new open source implementation of
XSL-FO raised, for me, a number of issues about the relationship
between the various tiers of developers in open source
implementations.  I detail this event only to draw attention to the
general principles.

One of the editors recently announced, on behalf of the major
corporation by which he is employed, a new implementation with the
code slated for publication.  What raised a few eyebrows was that
there was in progress on SourceForge another implementation realised
in a similar software environment, and to that point, there had been
no hint of the existence of the corporate project.  In the
conversations that followed, the editor explained that the lack of
communication between him and the SourceForge developers was mandated
by procedures within his corporation.  This raises a number of
interesting issues about the relationship between those corporations
which are active in open source development, the high profile open
source foundations like Apache, and the large group of committed open
source developers.

However, it was the fact that corporate implementor is also one of the
editors of the specification that intrigued me.  I have no idea what
life is like for implementors of other W3C Recommendations, but down
here in XSL-FO land it's kinda tough.  The spec is complex and, quite
understandably, has required many clarifications on quite significant
points, not to mention relatively minor points.  Implementing this
spec was always going to be difficult.  I cannot speak for other
implementors. but for me personally, it has turned out to be extremely
difficult.

It may be an unusual view, but in my experience, successful
implementation requires that a model of the specification and its
resolution be built up at many levels, from the most general and
abstract to the most particular.  Some have it that the levels can be
constructed in isolation, but if that were the case, there would be no
requirement for implementation support before a W3C Recommendation
were accepted.  In fact the models at different levels interact in
shockingly intimate and often unpredictable ways.  The better
understood the basic processes being modelled, the more likely that
such interaction and the design groping, probing, trial and
backtracking that underlies all software development will become
merely the ancestral memories of 'patterns' and other well-tested
habits of software thought.  For whatever reasons, that is not the
case with XSL-FO.

Because such is not the case, design sketches have been very fragile
and highly susceptible to discoveries made during the detailed design
of fragments.  I have always found in such circumstances that the
models I construct to guide the implementation must be as
comprehensive as I can possibly make them.  One has to be aware of
every wrinkle in the development of the models if one hopes to keep
control of the design, but awareness often comes only after traversing
a number of blind alleys.  Once all of this has been done, the result
will hopefully look clean, modular, robust and obvious.  The process
of getting there will be anything but.

I am trying to paint a picture of the peculiar agony of difficult
software design, as I have experienced it.  My experience may be
peculiar in the extreme, though I doubt that.  Think of riding a
unicycle on a high-wire between the twin towers, clasping a copy of
the spec and making notes while trying to keep track of seven
different references with the fingers of one hand.  To this obviously
unstable configuration, add a number of ambiguities in the spec, and
watch what happens.

I would have thought all of this more or less self-evident, as I would
have thought obvious the impact on effectiveness and motivation of
developers of the presence of long-unresolved questions about the
spec.  It seems, however, that I may have been too sanguine in this
assessment.   The editors do not seem to appreciate it.

That is too bald a statement.  At least one of the of the editors, to
my knowledge, has expressed a concern about the speed of resolution of
developers' enquiries.  For all I know, this concern is shared by all
of the editors.  Nonetheless, the problem remains.  It can take a
number of months to get responses to questions.  That's not the worst
of it, because often enough the response will raise further questions.
What to do?  When one is on the design unicycle, the only way to stay
upright is to keep moving.  After a few of these experiences, one
gives up on the nineteenth century pace of conversations with the
editors, and takes a punt on the interpretation of certain design
features.  At least one can proceed.

If every implementor faced the same situation, there would at least be
a rueful solidarity among all of them.  This is not the case for those
implementors who are also editors.  All other things being equal
(which they are not) the simple fact that an implementor-editor is
able to argue for an interpretation directly to the other editors
endows an enormous advantage.  In addition, of course, the
implementor-editor has access to the internal channels of
communication among the editors, and amongst the subgroup of
implementors.  This access bestows the multiple advantage that those
involved in implementation can kick the issues around in at least email
time, can draw other editors into their deliberations, and can make
joint representations in behalf of their interpretations to all of the
other editors.

I have no idea whether these forms of discussion actually occur, of
course.  If they do, they represent an impost on all other
implementors.  If they do not, I propose that they should be created,
with one significant difference: the implementation channel should be
open to all implementors.  The rationale is simple, and extends the
rationale of requiring implementations before promoting PRs.
Implementors create the testing lab and reality check for proposals,
and perform shakedown testing for intelligibility, winkling out all of
the ways in which a statement or set of statements can be read.  The
way to maximise the efficacy of the process is to maximise the
interactive input by maximising the number of implementors involved.

By this means, other implementors have the opportunity, both to hear
the rationale of those who are closest to the process, and to argue
with them for other interpretations and modifications.  This combined
implementation experience then comes to the full formal meetings of
the editors though the editor-implementors.  Outside implementors,
noting the opinions and arguments of the editors on the list, can then
make informed interim decisions about their own designs.  It goes
without saying that the editors remain free to take whatever decisions
they choose in the various publication milestones of the spec.

Of course there would be issues about how to moderate such a group,
how to determine membership, etc.  My own view is that a group
considering technical issues of the specification in such detail would
be self-moderating, and that it would be sufficient, at least at
first, to create an open list.  The presupposition here, of course, is
that the editors, and more particularly the implementor-editors, are
interested in free and open discussion of these issues; that no-one
has an axe to grind.

Which brings me back to the beginning.  We have seen that corporate
self-interest interferes with the normal courtesies of developer
communication in the open source environment.  The larger question is
whether corporate representatives on editorial boards such as this are
free to seek the common good in the development of standards for the
X-Web.

Peter

====
This message is also being posted separately to the xsl-editors@w3.org list.

-- 
Peter B. West  pbwest@powerup.com.au  http://www.powerup.com.au/~pbwest/
"Lord, to whom shall we go?"

Received on Thursday, 16 January 2003 06:26:36 UTC