W3C home > Mailing lists > Public > public-qa-dev@w3.org > February 2008

Re: mod_perl memory usage

From: olivier Thereaux <ot@w3.org>
Date: Mon, 4 Feb 2008 17:35:02 +0900
Cc: QA-dev <public-qa-dev@w3.org>
Message-Id: <6099006F-AC48-4A9E-8426-C9C4CF4FB30B@w3.org>
To: Ville Skyttä <ville.skytta@iki.fi>, Gerald Oskoboiny <gerald@w3.org>

Hi Ville, Gerald, all.

On Aug 2, 2007, at 08:46 , Gerald Oskoboiny wrote:
> mod_perl may have been a red herring; I was blaming it for the  
> massive apache2 process sizes (200 MB+ on jesssica) because I  
> thought that was something you guys had changed recently (though I  
> don't even know if that's true) and I had never seen apache  
> processes that big.
> [...]
> But even after we pruned that and other stuff last night the apache2  
> process sizes are 120 MB, and the 'check' processes are 80-90 MB, so  
> maybe we would be OK with mod_perl after all.

I did some benchmarking of our different validator versions, and threw  
in mod_perl2 in the mix as well, and found some very impressive results.

benchmark was done with ab -n500 -c2 -t30 http://localhost/wmvs/HEAD_modperl2/check?uri= 
and taking Time per request [ms] (mean, across all concurrent  
requests) as a metric.

Document            Validator   time (ms)

html2.0             0.7.4       583
html2.0             0.8.0       879
html2.0             0.8.2       1002
html2.0             HEAD        1025
html2.0             HEAD (mp2)  49


xhtml 1.0           0.7.4       600
xhtml 1.0           0.8.0       2288
xhtml 1.0           0.8.2       1029
xhtml 1.0           HEAD        1046
xhtml 1.0           HEAD (mp2)  53


xhtml+svg+mathml    0.7.4       1090
xhtml+svg+mathml    0.8.0       3955
xhtml+svg+mathml    0.8.2       1580
xhtml+svg+mathml    HEAD        1693
xhtml+svg+mathml    HEAD (mp2)  559


Memory usage:
With mod_perl2 apache processes are 52M virt, 31m RES.
With check as CGI, apache2 is 25M virt, + check 25M
This, however, was done on qa-dev, a 32 bits machine. It will be more  
than doubled on the 64 bits production servers.

* why is 0.8.0 VERY slow?
   - running HTML::Tidy even when not necessary.
   - xml well-formedness check is costly
   - according to Bjoern, xmlns checking is *very* costly too
* why are 0.8.2, HEAD slow?
   - HTML::Tidy issue fixed, but additional checks
   - xml-wf and xmlns checking still costly
* why is mod-perl so fast?
   - beats me why there is an order of magnitude in difference...
   - looks like forking check was very costly after all
* why so long for xhtml+svg+mathml?
   - unlike the other test docs, it haas to be retrieved on www.w3.org

On Aug 2, 2007, at 16:02 , Ville Skyttä wrote:
> You're probably right, however at least the httpd.conf snippets in  
> CVS make
> the validator run with ModPerl::Registry which does a bunch of other  
> things
> in addition to setting up a preloaded, shared perl interpreter.  IIRC
> ModPerl::PerlRun does the shared interpreter (+ some precompiled  
> shared
> standard perl modules?) part only and could also be worth a try.

Benchmarks above were done with ModPerl::Registry.
However, using PerlResponseHandler   ModPerl::PerlRun
seems to be fairly equivalent to ModPerl::Registry in terms of speed  
and memory usage. I haven't had time to look into the difference in  
depth but this may be interesting data.

On Aug 2, 2007, at 08:46 , Gerald Oskoboiny wrote:
> The biggest problem now seems to be a few URIs that consistently eat  
> up many minutes of CPU time; currently on jessica there are several  
> processes that have consumed 20+ cpu minutes each:
>   31664 www-data  25   0 88632  36m  10m R   29  0.9  22:14.48 check
>   32073 www-data  25   0 97820  36m  10m R   29  0.9  21:09.62 check
>   31862 www-data  25   0  131m  72m  10m R   27  1.8  21:26.18 check
>   31732 www-data  25   0 97824  36m  10m R   24  0.9  22:10.81 check

This was a problem with HTML::Tidy (being used even when not  
necessary) and some documents. Now that Tidy is only used on demand, I  
think the problem is 99% solved, and hopefully ulimit should help keep  
things in check.

Another issue I remember is that when I switched on mod-perl, the  
mechanism to brush off the (many) bogus requests to /check/referer  
without a Referer did not work. Something to do with the apache  
config, nothing we can't handle.

I would like, now that both jessica and lovejoy have been upgraded and  
that the Tidy issue has been fixed, to give mod-perl another chance.  
For now I will use the community leverage and ask people who have  
installed their own validator to test mod-perl2 and report.

Any thought on this plan?

Received on Monday, 4 February 2008 08:35:18 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:36:27 UTC