W3C home > Mailing lists > Public > xproc-dev@w3.org > May 2010

Memory usage

From: Nic Gibson <nicg@corbas.net>
Date: Fri, 7 May 2010 10:22:21 +0000
To: XProc Dev <xproc-dev@w3.org>
Message-ID: <20100507102221.GA12899@magpie.corbas.net>
We're seeing an XProc script through Calabash that shows increasing memory 
usage over time. I suspect that this is to be expected under the circumstances 
but I wanted to check and see if anyone can suggest a mitigating action.

The script takes and XML file containing (basically) a list of file
URLs. Each of these URLs is a directory on the local filesystem. All XML
files in each directory are read using p:load then transformed using
several XSLT pipelines. The whole script is basically two big nested 
p:for-each loops (one to read directories and a nested one to read
and process the files found)

As this runs the memory usage goes up for each file loaded and, eventually,
the jvm kills the process with a heap exhaustion error.

I suspect that there is nothing in the script above that might indicate
to calabash that any file can be discarded so each one is held in memory until 
the end of the script. Is that likely? I'm not exactly a skilled Java
programmer so I'm not in a position to read the code.

Can anyone see any sensible approach that might allow us to run this
script over several thousand XML file when it currently dies after around
nine?

cheers

nic
Received on Friday, 7 May 2010 10:26:31 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Friday, 7 May 2010 10:26:32 GMT