Re: Memory usage

This is really interesting and clearly the frame of reference that
developers have not yet had time to optimize important.

With the case of a for loop that reads input files, processes and
stores output for each file and creates a result set of the files,
would an implementation be able to easily determine that memory is not
required in all cases given for example that a developer could be
storing data and/or merging it?

Would it be easy until such time that optimizations are completed to
create a <reclaim memory> step of some kind which perhaps reclaims the
memory of a given port and sinks? <p:sinkMem>

  <p:input port="source">
  <p:output port="result" sequence="true"/>


  <p:for-each name="forEachFile">

          <p:xslt version="1.0" name="LoadData">

          <p:xslt version="1.0" name="Processing">

           <p:store name="store">

              <p:input port="source">
                <p:pipe step="store" port="result"/>


  <p:documentation>Wrap result XML </p:documentation>
  <p:wrap-sequence wrapper="forEachFile"/>


On Mon, May 10, 2010 at 9:16 AM,  <> wrote:
>> That's pretty well as far as I had worked out - whilst the memory
> problem
>> I've been seeing is *irritating*, it can't really be described as
> simply
>> wrong (complexly wrong perhaps) because there is nothing simple that
> can
>> be used to determine if a file loaded using p:load can be discarded.
> After the static analysis phase, the XProc processor has a pretty good
> picture of what the connections in the pipeline look like and when/where
> results of XProc steps are used (if they are used at all). That
> knowledge alone can be used for various memory optimizations. For
> instance, in a pipeline like this one:
> <p:pipeline>
>  <step1/>
>  <step2/>
> </p:pipeline>
> the result of step1 can almost certainly be discarded after step2 is has
> finished because there is no other step that refers to the result of
> step1.
> Another thing is scoping of steps. By wrapping a step in, for example, a
> p:group, you can very easily restrict the visibility of the results
> produced by the steps in the sub-pipeline:
> ...
> <p:group>
>  <step1/>
>  <step2/>
> </p:group>
> ...
> In the above example, the results produced by step1 can be discarded
> once p:group has finished because they will be in an inaccessible scope.
> And so on. There are many optimizations that XProc processors can do,
> but I think the implementers are just entering this stage after having
> implemented the standard. For instance, EMC's Calumet (which I am
> involved with) does not yet detect when the result of a step is not used
> any more, but it does release the documents when they become out of
> scope.
> Regards,
> Vojtech
> --
> Vojtech Toman
> Principal Software Engineer
> EMC Corporation


An informal recording with one mic under a tree leads to some pretty
sweet acoustic sounds.

Received on Tuesday, 11 May 2010 09:32:01 UTC