Re: Calabash: cache the compiled pipeline

Edwin de Jong <e.dejong@exxellence.nl> writes:
> At first I thought the "com.xmlcalabash.runtime.XPipeline" instance 
> would suffice, but unfortunately, it uses the XProcRuntime instance for 
> things like stacks, which makes it impossible to concurrently execute 
> multiple pipelines with a shared Runtime. Using synchronized access to 
> such a pipeline would work. In my setup, I've created an 
> XProcRuntimePool, filled with XProcRuntimes (flyweight pattern), each 
> runtime having references to a set of cached XPipeline's.

Hmm. Interesting. I hadn't intended to make that so difficult. I'm
building XML Calabash V2 in my spare time with an eye towards
streaming, threading, and better memory utilization. So far so good,
which probably means I just haven't gotten to the really hard parts
yet.

Assuming I wanted to make caching compiled pipeline easier in V2, what
would you suggest? 

> 1. Parallel execution of one pipeline by multiple threads

Yeah. I'm not sure I want to promise to make that work, but I guess it
depends how far I need to go.

My new approach is to build a nice, clean data structure of the
pipeline document, then from that build a graph[1] that I can analyze
to determine what parts I can stream, what parts might run in separate
threads, etc.

If I kept the result of the graph analysis around, but built some sort
of a new execution model from that each time, then I can imagine
caching the pipeline model and the graph analysis. As long as each
thread builds the execution model each time, I think it might work to
run the same pipeline in separate threads. Not clear if that's
sufficient caching for you guys or not.

(It's not clear yet if I will need the pipeline model after I've built
the graph or not, probably for things like p:inline, but maybe not. I
dunno yet.)

                                        Be seeing you,
                                          norm

[1] http://norman.walsh.name/2010/11/11/xprocpipes

-- 
Norman Walsh
Lead Engineer
MarkLogic Corporation
www.marklogic.com