Re: XProc Usability (was Re: New to Xproc Question : conditionnal "output port" definition?)

On Oct 12, 2011, at 10:29 AM, Norman Walsh wrote:
> Zearin <zearin@gonk.net> writes:
>> <opinion feedback="wanted" feedback-types="flames comments questions suggestions misc">
>>    The barrier of entry to XProc newcomers is needlessly high. I
>>    think of it sort of like W3C’s XSD spec: it’s capable of
>>    achieving powerful goals—but it bends over backwards so much to
>>    accommodate implementors that the end users spend all their time
>>    trying to learn all the “inconvenient corners” of XProc instead
>>    of making progress on their projects.
> 
> Could you be more specific?


Sure.  

Geert’s earlier reply was a good start.  I’ll start with that:


On Oct 12, 2011, at 10:48 AM, Geert Josten wrote:
> Just jumping in and out in the middle, regarding compactness..

> 

> It would already help a lot if the specs could be tuned such that you can rely more on default behavior, or could write things with less characters:

Yes.

> - You would not need the p:with-option stuff if attribute value templates were supported from the start.

YES!!!

Having to use p:with-option is so unbelievably annoying!  It’s annoying to write.  It’s annoying to read.  

How would you feel if you were forced to pass all non-static arguments through an extra function in your programming language?  For example, let me simulate an example in Java:

//  Example #1
// (Apologies for any syntax errors—I only know the very very basics of Java.)
String staticArgument = "non-dynamic string";
aFunctionBeingCalled( staticArgument, with-argument(aDymanicArgument()), with-argument(anotherDynamicArgument())  );

Now, compare that to this:

//  Example #2
String staticArgument = "non-dynamic string";
aFunctionBeingCalled( staticArgument, aDynamicArgument(), anotherDynamicArgument());

Tell me: Do you honestly prefer the first example to the second?

What if you had no choice but to type all that extra crap…EVERY…SINGLE…TIME…you called a function that had arguments evaluated at runtime?  

And the more dynamic I want my pipeline to be, the worse it gets.

What’s more—the first example is still significantly leaner than the XProc equivalent.  My example only added 1 extra level to the code (the with-argument() wrapper function).  In XProc, each dynamic argument adds so much more extra:

A new child element (<p:with-option />)
Specify which option (@name)
Pass the value (@select, OR yet ANOTHER child element—most of which also have their own attributes that need to be typed)

I mean, whenever I hear someone complain that XML is too verbose, I ask them what editor they code in.  I mean, it’s 2011—editors can do most of that automatic typing for you!

…But then, there’s <p:with-option />.  An extra child element, plus attributes, plus the potential for still more child elements of its own…just because a step’s argument is evaluated at runtime?  Come on!  

Sigh.  

I know there was a reason behind this.  I know it exists for the implementors.  

But ultimately, the implementors are not the end-users of XProc.  We are.  

It’s as though everyone agreed we needed a macroscopic XML tool, but the moment the W3C started working on it, they forgot all about the end users and only listened to implementors.


> - Things like generating empty input taking up 5 lines (p:identity with p:input with p:empty inside), can't that be done shorter?


YES!!!  

Seriously!

Would it have caused World War Ⅲ to do this?

<p:input port="source" empty="true" />


Or this?


<p:input port="source" select="empty()" />


I don’t care which.  It’s the equivalent of passing null to a function in other languages.  It shouldn’t be any more complicated than that.


> - Maybe something specific to XMLCalabash, but why can't I just do p:variable with a p:pipe connecting to parameters port? It complains about context, forcing me to pipe the parameters through a p:parameters first, and rerouting input around the p:parameters as well.


Still more of the same.  Designed for implementors.  

We end users pay the price.


> - Lot's of extensions don't produce output, but do take input without doing much with that (most notably the file io extensions for example). Why not just let them do identity transform as well, making it much easier to do a large sequence of such steps..


OMG, yes!  

I have never had so much extra work just to store something in the middle of a pipeline.  

…

I’m so sorry, Norm.  

It’s just…XProc makes me angry.  

(Please don’t hurt me!  ☹  I feel like I have just betrayed a childhood hero…I just never imagined something I looked forward to so much would turn out to be this frustrating to use.)

…

So far I’ve just agreed with the points from Geert’s earlier post, but there’s still more on the XProcVNext page on the wiki.  

I don’t understand 100% of the suggestions there, but the rest of them—most of them—make complete sense to me.  I agree with them completely.


> Simpler tools, or just different tools? Can you enumerate lessons
> learned along the way?


Good question.  Sometimes those two things are indeed confused.  

But, I do actually mean simpler:

XMLSH

some XProc + some other scripts (bash, Python, XMLSH, or 3rd party utilities I download)
I started to write an XProc pipeline, but some would-be steps proved too annoying to write.  Sometimes I can’t figure out what’s wrong, and sometimes I just don’t feel like being forced into the extreme excess verbosity (described above).

Sometimes I wanted to automate something with a script, but ran into an endless stream of trivial “nitpick” errors that I gave up, and resort to a combination of search-and-replace and some of Oxygen’s refactoring functionality instead.  Even if I have to type a keyboard shortcut 100 times to it—it’s preferable to staring at an XProc pipeline and “edit-run-error-edit” over and over again.  


Regarding “lessons learned along the way”—I don’t remember everything, because over time I’ve used XProc less and less, and used XMLSH more (though I still use both)…

But off the top of my head, some things are:


Processing <p:http-request /> results is hard
This isn’t XProc’s fault—just an unfortunate reality of dealing with tag-soup HTML.  (And Tidy is useless if you want X/HTML5. ☹)

Doing anything with the filesystem in XProc is an exercise in masochism.  Having to reconnect an input in the middle of a pipeline whenever I use a <p:store /> makes me want to break things.  
There have been several times I actually preferred run the pipeline, waiting for Oxygen’s “results” view, and copy-paste the results into a new document.  
The EXProc steps admittedly help a bit.  Why they weren’t included in the core spec is beyond me.  
On the other hand, I kinda wish the EXProc filesystem steps were also available as functions.  That way I could just use them as functions in p:inputs and p:outputs (or use them with p:options in my own steps).

XProc does not lend itself well to “learning by exploration”.  (This is XMLSH’s greatest strength, in my opinion.)  In XProc, I spend too much time trying to understand errors, figuring out how to work around stuff like <p:store />, looking up what steps and options do, and so on.  Before long I’ve forgotten what I started the pipeline for, and instead I’m trying to remember all these other things in order to get it to work.


> Would a non-XML syntax for XProc help?

[…]

> "Compact" syntax:
> 
> pipeline {
>  serialization "result" with method="xhtml", indent="true"
>  xinclude
>  xslt {
>    input "stylesheet" {
>      document "dbslides.xsl"
>    }
>  }
> }
> 
> What struck me most after writing it is that it's not really much more
> compact than the XML syntax. Jeni Tennison took up the challenge to
> produce a more natural compact syntax. I'm having trouble getting
> JavaCC to swallow it, but I'll get there eventually, I expect.


How interesting!  

Actually, the compact syntax makes some of the required-verbosity a bit less frustrating.  (But I do mean a bit less.  I’d rather be able to just pass in $variable wherever I needed it.)

Hmm.  You say the compact syntax isn’t really much more “compact”…I’m not sure I agree.  If you’re talking about total lines of code, I’d agree.  But your example is both less typing and much easier to read.  

On the other hand…I’m not sure how to express what I’m feeling.  I’ve designed XML grammars, but I’ve never designed a language syntax before.  My mind keeps flipping between different syntaxes—your example above, an equivalent script in XMLSH, and a mix of syntactic choices from other languages.  (By “other languages,” I’m specifically thinking of Python, Ruby, and XPath—all of which I think are powerful, easy to read, easy to write, and just “feel” elegant when I use them.)




> I've certainly written stylesheets that produce stylesheets.

I know! ☺ 

I looked at the DocBook XSL that comes with Oxygen.  I saw the comments stating “auto-generated—do not edit!” in stylesheet after stylesheet.  

I bow to your superior skill.  (With envy.  ☺)


> And I'm
> one of the outliers who's unconvinced that moving from an XML
> pattern-based syntax to a string syntax for XPath (back way before
> XSLT 1.0 came out) was a good thing. Water under the bridge.


Oh, wow!  The thought of that gives me shudders…

I’m curious—why do you feel this way?  

XPath is a very big part of what makes many flavors XML so attractive to me.  Definitely one of my favorite things in the XML universe.  It makes SQL seem clumsy and oafish by comparison. :)


>>        What are the pros and cons of designing a language to work
>>        with XML using an XML syntax vs. a non-XML syntax?
> That's a good question. I'm toying with the idea of writing about it
> for XML Prague 2012. You know, in my copiuos free time.


I don’t have 1337 programming skills, Norm.  But I’ve offered to help with some grunt-work on your websites/documentation before.  I figured you were too busy to respond, but if you reading this, the offer’s still open.  (E-mail me off-list if you’re interested.)

I enjoyed your talk from XML Prague 2011.  If you decide to discuss this for 2012, I’d be very interested.


>>    Is there any official work (as in W3C) on improving XProc at the moment?
> No. The XProc WG is not chartered to do new work. We're going to talk
> informally about next steps at the upcoming f2f in CA. It's a bit of a
> chicken and egg problem. Trying to get the W3C Membership and our
> member companies to support work on V.next would be aided by greater
> adoption of V1.0.


What a Catch-22!  If I want XProc to get better, I have to use the version that makes me want to cry.  

I feel like…

♫ Everything I know is wrong!  
Up is down,  Black is white, and Short is long… ♬


Sigh…What a day.


—Zearin (Tony)