RE: SAG-FO-02 follow-up: Durations

This is a response to Jim Melton's response to Software AG's proposal to
eliminate many of the functions and operators on durations, relying instead
on numeric arithmetic.

JM> I think it's fair to summarize Mike's proposal in [1] thus: eliminate
all uses of the xdt:yearMonthDuration and xdt:dayTimeDuration types, as well
as the types themselves, and replace them with operations involving only the
existing xs:duration type.  There are, of course, a number of details in the
proposal in support of the high-level goal. 

JM> Now, my principle reason for objecting to this direction is based on the
complexity that Mike rightfully deplores.  He proposes that we simplify the
specifications for various documents in the XQuery/XPath suite and the lives
if the WG members working on those documents.  Unfortunately, my two decades
of experience in this area has convinced me that this has the unfortunate
side effect of saddling the users (the query writers, that is) with solving
the problems that Mike observes to be so complicated in the XQuery/XPath
documents.

MK> I don't think this is true. I think that very many of the functions and
operators we are providing are straight duplicates of what you could do more
simply with regular arithmetic.

Certainly, adding, multiplying, comparing, and sorting durations expressed
as a number of seconds or as a number of months is not difficult. By asking
users to do these operations in the same way as they manipulate lengths,
weights, or numbers of paperclips, we are making their lives easier, not
harder.

The only things that are special and difficult about durations are (a) the
interaction between month-durations and second-durations (which is a problem
we have washed our hands of), and (b) computations that combine dates (and
times) with durations, where the Software AG proposal retains the
functionality of the current specification.

JM> I am unwilling to require tens of thousands of query writers to figure
out how to deal with an issue that was deemed unsolvable by the 30 or 40
people in the world who are most expert in the subject of XQuery and XPath.
If we, the members of the XQuery WG, have so much difficulty getting this
right, how many thousands of errors, subtle and devious, will be produced by
query authors who are trying to solve their business problems but are forced
to solve data type problems instead?

MK> I would like to see concrete examples. I think that getting rid of all
these functions makes users' lives easier. For example, it's not at all
obvious to the casual reader that the operation to multiply a duration by a
number is not what they need when calculating how much to pay someone who
has worked for four hours at a rate of $10 per hour. In fact, I think it's
quite hard for that user to wade through our list of dozens of functions and
discover that none of them meets this need. If the duration was represented
by a number, like every other quantity, then the difficulty would not arise.

JM> In many ways, Mike's proposal is attractive.  On the surface, it appears
to simplify a lot of things.  For example, it removes a large number of
functions from the F&O specification [3] (as well as removing a number of
lines from various tables).  As somebody employed by an XQuery implementor,
I am concerned (as is, of course, Mike) with having such a large number of
functions to specify, implement, test, and document.  However, I much more
concerned with having to deal with what will be (in my opinion) a
significant number of support calls related to user-written bugs that might
be avoided by supporting durations natively in XQuery. 

JM> It is true that an *implementation* of XQuery might choose to represent
a value of type xdt:yearMonthDuration as an integer number of months and a
value of type xdt:dayTimeDuration as a decimal number of seconds.  And it is
also true that an *implementation* of XQuery (and of XML Schema) might
choose to represent a value of type xs:duration as a 2-tuple containing the
number of months and the number of seconds.  But I disagree that such a
representation is the right paradigm to present to our users.  In fact, the
XML Schema WG is believed to be considering the xdt:yearMonthDuration and
xdt:dayTimeDuration types for inclusion in a future version of the XML
Schema Recommendation.  To me, that suggests that data type experts are
concerned with the practical use of xs:duration.  (Full disclosure: I have
argued that the XML Schema WG adopt the xdt: types related to durations for
reasons closely related to my positions in this message.)

JM> I believe that the reality of the situation is that durations are a
useful type, one whose values have particular semantics that should be
observed on their own, without being manipulated in some other surrogate
form.  Yes, of course it is possible to use numerics as a surrogate for
durations --- and for dateTime value as well.  It's possible to do so with
character strings, too.  (The proof is that all computers do exactly this
internally...the one, two, or more 8-bit bytes used to represent a character
is nothing but a binary number.)  But characters and character strings are
sufficiently useful that nobody seriously proposes that they be handled as
numbers.  Of course, character strings are pretty well understood by most
people (although Unicode has offered some challenges to some of those
people), while durations are inherently more complex.

MK> So why don't we have special data types for lengths, weights, voltages,
and temperatures? None of this reasoning convinces me that durations are any
different from other units of measure, or that there is a good case for
treating them differently. If we did really clever things with "complex"
durations (those that mix months and seconds), then there might be some
argument. But we've decided not to handle complex durations, and only handle
those that map trivially to numbers: which means, as I see it, that our data
types are not adding any value. 

JM> Let's look at a detail or two.  In [1], Mike uses this example: 
A dateTime with lexical representation 1999-05-31T13:20:00-05:00
has a value represented by the tuple (1999-05-31T18:20:00Z, -18000)

JM> I consider myself fairly adept at mental arithmetic, but I have to pause
for awhile to figure out that "-18000" means "Eastern Standard Time" in the
USA, Canada, and probably other locations --- in other words, that it means
"-05:00", or "5 timezones to the west of UTC".  In fairness, Mike has not
proposed that the lexical representation of the time zone component of an
xs:dateTime be changed.  But he does propose that the *value space*
representation be changed to a number of seconds, and it is the value space
representation that application/query writers have to manipulate.

MK> Well, there is at least one country in the world that still uses
non-metric units for lengths. If you're going to manipulate lengths in feet
and inches, then you're going to have to get used to doing a lot of
arithmetic to convert between different representations of a length. I ask
the question again: why are durations different? And if you don't like using
constants like -18000, it's easy to write them as variables ($TZ:EDT, for
example) or as expressions (for example: tz(-5)). 

JM> Mike also proposes that: 
The elapsed time between two dates, times, or dateTimes is
generally handled as a number of seconds, expressed as an xs:double. Some
functions are also provided that manipulate a duration as an integer
number of months. All arithmetic, comparison, and sorting of durations is
achieved by expressing the duration as an xs:integer number of months
plus an xs:double number of seconds, and then manipulating these values
using conventional numeric arithmetic.

JM> As I said above, this is a perfectly reasonable mechanism for
implementations to use internally.  But I believe that it is generally
awkward and confusing to have humans manipulate this notation for a data
type that inherently has structure to it.  Quick now, what is the difference
between 11:00 and 17:00?  Six hours, right?  Sure, that's the same as 21600
seconds, but why is that important in this particular calculation?

MK> Well, having calculated the difference, what are you going to do with
it? If you want to display it as "six hours", then I would suggest that
translating "PT6H" to "six hours" is likely to be just as difficult as
translating "21600". If you want to do anything with the result other than
display it (for example, to calculate the average speed over your journey,
or to plot it on a graph in SVG), then the value 21600 is much more useful
to you than the value PT6H.

JM> Worse, imagine having to write a query that asks "What is the sum of 3
hours 45 minutes and 4 hours 15 minutes?"  Should a query writer have to
first translate each of the values into a numeric value, then perform the
addition, then transform the result back into a duration?  Or would the
query writer be better off with merely adding the two values together?
Consider the following examples (which I have sincerely attempted to write
fairly).  Assume that an XQuery variable $dur1 contains the first value, 3
hours 45 minutes, and another variable $dur2 contains the second value, 4
hours 45 minutes.  (In the existing F&O specification, the types of $dur1
and $dur2 would be xdt:dayTimeDuration; under Mike's proposal, both would
have the type xs:duration.)

(1) fn:make-duration ( 0, ( fn:get-seconds-from-duration ( $dur1 ) +
fn:get-seconds-from-duration ( $dur2 ) ) )

(2) $dur1 + $dur2

Given that choice, I think that most query authors would prefer the
simplicity (and, probably less important, brevity) of example (2).

MK> Firstly, you are assuming that the data starts of as a duration. My
expectation is that usually, it won't. For a start, I think that it's much
more common in persistent data to hold absolute dates/times than to hold
durations; and in many fields of application, where people do hold
durations, I think they will already be held as numbers. XBRL, for example,
always holds duration information as a start date and end date; while I have
seen XML files that compare the performance of different software products
under different operational conditions, and they invariably represent
elapsed times as numbers. They typically start life as numbers, and I think
most people would need a compelling reason to translate the number 132.0578
into the duration PT132.0578S; the fact that the duration data type exists
isn't of itself a reason for using it. 

I agree that your example 2 is easier, and my advice would be that to
achieve this simplicity, all you need to do is represent your durations as
numbers, just as you would represent lengths or monetary amounts.  

JM> Again, ask the question "What time will it be 4 hours 30 minutes after
8:00?".  In this case, $time1 has type xs:time in both examples and contains
the time 08:00.  $dur1 has either type xs:duration or type
xdt:dayTimeDuration, both representing 4 hours 30 minutes. 

(3) fn:add-seconds-to-time ( $time1, fn:get-seconds-from-duration ( $dur1 )
)

(4) $time1 + $dur1

Again, I feel that the second example would generally be preferred by most
query authors.

MK> I agree that there is a need for special data types to represent dates
and (probably) times, and that this creates a need for special functions to
do arithmetic on dates. In this case I am not at all convinced that
overloading the "+" operator is a good idea. There are two reasons why (4)
looks more appealing than (3). One reason is that you have chosen to start
with a duration represented as an xs:duration, whereas I think it will be
equally common (whether the Software AG proposal is accepted or not) to
start with a duration represented as a number. (I don't think I have every
used an interval data type in SQL; I have always used numbers.) The second
reason is that our operators are generally one character while our function
names can be 28 characters or more. If we started with a numeric duration,
and used shorter function names, then these two examples might read as:

(3') later($time1, $dur1)

(4') $time1 + seconds($dur1)

where I think there is no practical difference in usability.


JM> Now, in [2], Jeni Tennison has correctly observed that F&O fails to
provide for the obvious operations of 

division of an xdt:yearMonthDuration by another
xdt:yearMonthDuration and division of an xdt:dayTimeDuration by another
xdt:dayTimeDuration, as well as subtraction of two dates from each other
to get a xdt:yearMonthDuration (rather than a xdt:dayTimeDuration)? 
That omission is certainly something that could be corrected (even though
I note in passing that the SQL standard does not provide for division of
one duration/INTERVAL by another).

MK> In my view there's an infinite number of functions on durations that can
only be achieved by converting them to numbers. Patching up the gaps is
pointless, there will always be more. 


JM> But I am not so comfortable with Mike's need to compute average speeds
by dividing a number by a duration --- even though that is made easier by
his proposal and the equivalent using existing F&O capabilities is rather
tedious. By the way, I think the two approaches to solving this problem
would look something like this.  In these examples, $dur1 is either an
xdt:dayTimeDuration or an xs:duration, and $dist is an xs:double. 

(5) $dist div fn:get-seconds-from-duration ( $dur1 )

(6) $dist div ( ( ( fn:get-days-from-dayTimeDuration ( $dur1 ) * 24 +
                    fn:get-hours-from-dayTimeDuration ( $dur1 ) ) * 60 +
                    fn:get-minutes-from-dayTimeDuration ( $dur1 ) ) * 60 +
                    fn:get-seconds-from-dayTimeDuration ( $dur1 ) ) * 60

It's very clear that Mike's alternative is much shorter and easier to
understand.  However, one must ask whether this is a typical question that
will be asked using durations.  In some business, it will be; in others, it
will not be.  I suggest that, in businesses where such a question is common,
it is easy enough to write a user-defined function
(my:convert-dayTimeDuration-to-seconds, for example) to be used by every
query needing this sort of computation, which would reduce example (6) to:

(7) $dist div my:convert-dayTimeDuration-to-seconds ( $dur1 )

And (7) is not significantly different from (5).

MK> I actually started with a real-life problem of comparing performance
measurements on software products, and decided to translate it to something
that looked less parochial.

JM> We come now to my final source of discomfort, and this is one that I do
not believe can be papered over by workarounds.  In Mike's proposal, he
deletes the two newly-defined subtypes of xs:duration, xdt:yearMonthDuration
and xdt:dayTimeDuration.  A prime reason for the creation of those two
subtypes is that each of them is "totally ordered", meaning that one can
compare two values of one type and unambiguously determine whether the first
is greater than the second, or less than it, or equal to it.  By contrast,
one cannot do that with values of the xs:duration type in the general case.
I believe that the same deficiency would result from Mike's proposal,
leaving the user to deal with the consequences. 

MK> Yes. Our two data types do nothing to solve the problem of managing
complex durations. They try to wish the problem away. One of the fundamental
insights that came from my colleagues in Darmstadt was that the
justification for duration data types is because durations are difficult,
but we have restricted ourselves to handling only those durations that are
easy, in which case the justification for having special data types
disappears.

JM> Consider two values of type xs:duration under Mike's proposal.  One of
these, $dur1, is (effectively) constructed from fn:make-duration(2, 0) and
the other, $dur2, from fn:make-duration(1, 30*24*60*60).  What is the result
of comparing $dur1 and $dur2?  It is not possible to determine this
unambiguously.  The reason is simple: We do not know how many days there are
in a month!  Some months have 31 days, others have either 28 or 29 days
(depending on the year involved).  Therefore, the expression "30*24*60*60"
may or may not equal 1 month.  Consequently, the type xs:duration is *not*
totally ordered.  That means, of course, that some pairs of values, such as
fn:make-duration(2, 0) and $dur2, from fn:make-duration(1, 1*24*60*60), can
be compared without ambiguity (2 months is greater than 1 month and 1 day),
while others cannot. 

The existing F&O specification that introduces and uses the two xdt: types
avoid the mixing of year-month durations and day-time durations specifically
because of this problem.  In Mike's proposal, he suggests (in changes to
Annex C.5) that one could define comparison of two values of type
xs:duration by comparing their months values and also comparing their
seconds values and require that both must be equal in order for the result
of the comparison to indicate equality. 

But he admits that greater than and less than comparisons are more
problematic because of the partially-ordered nature of xs:duration.  He
suggests a pragmatic solution that uses 
365.242199 as the average
number of days in a year.  That works well when the values involves
are intended for statistical use (e.g., over thousands, probably even
hundreds, of years), but they don't work nearly as well for computations
involving one year or two years or 1 month!

MK> The key point of my proposal in this area is that mixed durations are
indeed problematic, and that our two new data types do nothing to help the
user in handling them, and because they don't help with the problem, they
serve no useful purpose. By contrast, if complex durations are represented
as a pair of numbers, there are various algorithms the user could employ to
make them tractable, some of which might meet the needs of some
applications. If you've got a pair of numbers, the problem is probably
easier to deal with than if you've got a (xdt:yearMonthDuration,
xdt:dayMonthDuration) pair.


JM> In short (in case I have not been clear), I cannot support, and will
oppose, acceptance of this proposal on the grounds that it quite often makes
the application/query authors' lives more difficult, raises the probability
of errors that will result in support costs borne by vendors, and loses the
safety provided by the xdt: types, all in the goal of simplifying the lives
of the authors of the XQuery/XPath suite of documents. 


MK> I emphatically do NOT try to justify the proposal on the basis that it
makes the lives of implementors and specifiers easier. The key point about
the proposal is that it makes the language significantly smaller without
removing any useful functionality, and that this actually makes the users'
lives easier.

Thanks for giving the proposal such careful attention!

Michael Kay