Re: mod_perl memory usage

Hi Ville, Gerald, all.


On Aug 2, 2007, at 08:46 , Gerald Oskoboiny wrote:
> mod_perl may have been a red herring; I was blaming it for the  
> massive apache2 process sizes (200 MB+ on jesssica) because I  
> thought that was something you guys had changed recently (though I  
> don't even know if that's true) and I had never seen apache  
> processes that big.
> [...]
> But even after we pruned that and other stuff last night the apache2  
> process sizes are 120 MB, and the 'check' processes are 80-90 MB, so  
> maybe we would be OK with mod_perl after all.

I did some benchmarking of our different validator versions, and threw  
in mod_perl2 in the mix as well, and found some very impressive results.

benchmark was done with ab -n500 -c2 -t30 http://localhost/wmvs/HEAD_modperl2/check?uri= 
…
and taking Time per request [ms] (mean, across all concurrent  
requests) as a metric.


Document            Validator   time (ms)

html2.0             0.7.4       583
html2.0             0.8.0       879
html2.0             0.8.2       1002
html2.0             HEAD        1025
html2.0             HEAD (mp2)  49

http://qa-dev.w3.org/wmvs/HEAD/dev/tests/html20.html

xhtml 1.0           0.7.4       600
xhtml 1.0           0.8.0       2288
xhtml 1.0           0.8.2       1029
xhtml 1.0           HEAD        1046
xhtml 1.0           HEAD (mp2)  53

http://qa-dev.w3.org/wmvs/HEAD/dev/tests/xhtml1-appc-emptycontent.html

xhtml+svg+mathml    0.7.4       1090
xhtml+svg+mathml    0.8.0       3955
xhtml+svg+mathml    0.8.2       1580
xhtml+svg+mathml    HEAD        1693
xhtml+svg+mathml    HEAD (mp2)  559

http://www.w3.org/TR/XHTMLplusMathMLplusSVG/sample.xhtml

Memory usage:
With mod_perl2 apache processes are 52M virt, 31m RES.
With check as CGI, apache2 is 25M virt, + check 25M
This, however, was done on qa-dev, a 32 bits machine. It will be more  
than doubled on the 64 bits production servers.

* why is 0.8.0 VERY slow?
   - running HTML::Tidy even when not necessary.
   - xml well-formedness check is costly
   - according to Bjoern, xmlns checking is *very* costly too
* why are 0.8.2, HEAD slow?
   - HTML::Tidy issue fixed, but additional checks
   - xml-wf and xmlns checking still costly
* why is mod-perl so fast?
   - beats me why there is an order of magnitude in difference...
   - looks like forking check was very costly after all
* why so long for xhtml+svg+mathml?
   - unlike the other test docs, it haas to be retrieved on www.w3.org

On Aug 2, 2007, at 16:02 , Ville Skyttä wrote:
> You're probably right, however at least the httpd.conf snippets in  
> CVS make
> the validator run with ModPerl::Registry which does a bunch of other  
> things
> in addition to setting up a preloaded, shared perl interpreter.  IIRC
> ModPerl::PerlRun does the shared interpreter (+ some precompiled  
> shared
> standard perl modules?) part only and could also be worth a try.

Benchmarks above were done with ModPerl::Registry.
However, using PerlResponseHandler   ModPerl::PerlRun
seems to be fairly equivalent to ModPerl::Registry in terms of speed  
and memory usage. I haven't had time to look into the difference in  
depth but this may be interesting data.

On Aug 2, 2007, at 08:46 , Gerald Oskoboiny wrote:
> The biggest problem now seems to be a few URIs that consistently eat  
> up many minutes of CPU time; currently on jessica there are several  
> processes that have consumed 20+ cpu minutes each:
>
>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   31664 www-data  25   0 88632  36m  10m R   29  0.9  22:14.48 check
>   32073 www-data  25   0 97820  36m  10m R   29  0.9  21:09.62 check
>   31862 www-data  25   0  131m  72m  10m R   27  1.8  21:26.18 check
>   31732 www-data  25   0 97824  36m  10m R   24  0.9  22:10.81 check

This was a problem with HTML::Tidy (being used even when not  
necessary) and some documents. Now that Tidy is only used on demand, I  
think the problem is 99% solved, and hopefully ulimit should help keep  
things in check.

Another issue I remember is that when I switched on mod-perl, the  
mechanism to brush off the (many) bogus requests to /check/referer  
without a Referer did not work. Something to do with the apache  
config, nothing we can't handle.

I would like, now that both jessica and lovejoy have been upgraded and  
that the Tidy issue has been fixed, to give mod-perl another chance.  
For now I will use the community leverage and ask people who have  
installed their own validator to test mod-perl2 and report.

Any thought on this plan?

Thanks,
-- 
olivier

Received on Monday, 4 February 2008 08:35:18 UTC