- From: Arthur Smith <apsmith@aps.org>
- Date: Fri, 29 Jul 2005 15:31:34 -0400
- To: www-lib@w3.org
- CC: Tanmay Patwardhan <tpthesis@hotmail.com>, Paul Dlug <paul@ridge.aps.org>
- Message-ID: <42EA8416.8010700@aps.org>
As had been found by Tanmay Patwardhan <tpthesis@hotmail.com
<mailto:tpthesis@hotmail.com?Subject=Re:%20Slowing%20down%20in%20POST%20request-response&In-Reply-To=%3CLaw14-F21JSw9dOXwHy00045e04@hotmail.com%3E&References=%3CLaw14-F21JSw9dOXwHy00045e04@hotmail.com%3E>>,
on the list about 2 years ago, repeated calls to any HTProfile
client-creation routine (HTProfile_newNoCacheClient,
HTProfile_newClient, HTProfile_newRobot, etc) result in the
request-response loop running slower and slower.
This is rather a problem because clients like xmlrpc-c only interface
with the w3c libraries through the HTProfile subroutines, so repeated
client-creation requests there result in slower and slower responses
(after several thousand requests on a new-ish Solaris machine, we saw
delays of 10 seconds or more).
The problem, as was pointed out on this list back in 2003, is that these
subroutines were written with the thought they would only be called once
per application. But the specific problem is that each time they are
called, they "re-initialize" things like the list of transport
protocols, BEFORE and AFTER filters, MIME types, icons, etc. But the
"re-initializing" did not clear out old values, it just added default
ones on, piling them up so after several thousand calls, the BEFORE
filter list had several thousand instances of HTRuleFilter, etc.
This turns out to be reasonably simple to fix - it would be even easier
if HTList managed objects with consistently named keys, but as it is,
there is no reason why HTNet_addBefore, for instance, can't just remove
any previous instance of a filter before adding it in again
(HTNet_deleteBefore, after all, assumes there's only one instance of a
filter with a given name). Similarly for all the other cases where a
list of objects is managed in this sort of way in the code - there's no
good reason to have more than one object with the same label in these lists.
A patch to fix these is attached - obtained by CVS diff -c of my version
relative to the most recent CVS version from the repository. It greatly
speeds up repeated profile creations, and I don't see any significant
downside. I hope to hear this has been applied in the repository version.
Arthur Smith (apsmith@aps.org)
631-591-4072
Index: Library/src/HTAlert.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTAlert.c,v
retrieving revision 2.53
diff -c -r2.53 HTAlert.c
*** Library/src/HTAlert.c 22 Feb 1999 22:10:10 -0000 2.53
--- Library/src/HTAlert.c 29 Jul 2005 13:29:36 -0000
***************
*** 169,174 ****
--- 169,175 ----
PUBLIC BOOL HTAlert_add (HTAlertCallback * cbf, HTAlertOpcode opcode)
{
if (!HTMessages) HTMessages = HTList_new();
+ else HTAlert_delete(cbf); /* Remove duplicates */
return HTAlertCall_add(HTMessages, cbf, opcode);
}
Index: Library/src/HTHeader.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTHeader.c,v
retrieving revision 2.13
diff -c -r2.13 HTHeader.c
*** Library/src/HTHeader.c 7 Feb 1999 18:20:31 -0000 2.13
--- Library/src/HTHeader.c 29 Jul 2005 13:29:36 -0000
***************
*** 69,74 ****
--- 69,75 ----
HTParserCallback * callback)
{
if (!ParseSet) ParseSet = HTMIMEParseSet_new(MIME_HASH_SIZE);
+ else HTHeader_deleteParser(token); /* Remove duplicates */
return (HTMIMEParseSet_add(ParseSet, token, case_sensitive, callback) != NULL);
}
Index: Library/src/HTInit.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTInit.c,v
retrieving revision 2.100
diff -c -r2.100 HTInit.c
*** Library/src/HTInit.c 18 Dec 2000 17:00:56 -0000 2.100
--- Library/src/HTInit.c 29 Jul 2005 13:29:37 -0000
***************
*** 345,350 ****
--- 345,351 ----
{
const char * prefix = url_prefix ? url_prefix : ICON_LOCATION;
+ HTIcon_deleteAll(); /* Start fresh */
HTIcon_addBlank("blank.xbm", prefix, NULL);
HTIcon_addDir("directory.xbm", prefix, "DIR");
HTIcon_addParent("back.xbm", prefix, "UP");
Index: Library/src/HTNet.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTNet.c,v
retrieving revision 2.106
diff -c -r2.106 HTNet.c
*** Library/src/HTNet.c 4 Jul 2000 15:26:25 -0000 2.106
--- Library/src/HTNet.c 29 Jul 2005 13:29:39 -0000
***************
*** 350,355 ****
--- 350,356 ----
void * param, HTFilterOrder order)
{
if (!HTBefore) HTBefore = HTList_new();
+ else HTNet_deleteBefore(before); /* Ensure not listed twice */
return HTNetCall_addBefore(HTBefore, before, tmplate, param, order);
}
***************
*** 392,397 ****
--- 393,399 ----
void * param, int status, HTFilterOrder order)
{
if (!HTAfter) HTAfter = HTList_new();
+ else HTNet_deleteAfter(after); /* Ensure not listed twice */
return HTNetCall_addAfter(HTAfter, after, tmplate, param, status, order);
}
Index: Library/src/HTProfil.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTProfil.c,v
retrieving revision 2.30
diff -c -r2.30 HTProfil.c
*** Library/src/HTProfil.c 14 Aug 2001 16:15:50 -0000 2.30
--- Library/src/HTProfil.c 29 Jul 2005 13:29:40 -0000
***************
*** 58,66 ****
HTAlertInit();
HTAlert_setInteractive(YES);
! if (!converters) converters = HTList_new();
! if (!transfer_encodings) transfer_encodings = HTList_new();
! if (!content_encodings) content_encodings = HTList_new();
/* Register the default set of transport protocols */
HTTransportInit();
--- 58,91 ----
HTAlertInit();
HTAlert_setInteractive(YES);
! if (!converters) {
! converters = HTList_new();
! /* Register the default set of converters */
! HTConverterInit(converters);
! /* Register the default libwww HTML parser */
! if (HTMLParser) HTMLInit(converters);
! /* Set the converters as global converters for all requests */
! HTFormat_setConversion(converters);
! }
!
! if (!transfer_encodings) {
! transfer_encodings = HTList_new();
! /* Register the default set of transfer encoders and decoders */
! HTTransferEncoderInit(transfer_encodings);
! HTFormat_setTransferCoding(transfer_encodings);
! }
!
! if (!content_encodings) {
! content_encodings = HTList_new();
! /* Register the default set of content encoders and decoders */
! HTContentEncoderInit(content_encodings);
! if (HTList_count(content_encodings) > 0)
! HTFormat_setContentCoding(content_encodings);
! else {
! HTList_delete(content_encodings);
! content_encodings = NULL;
! }
! }
/* Register the default set of transport protocols */
HTTransportInit();
***************
*** 89,116 ****
/* Get any proxy or gateway environment variables */
HTProxy_getEnvVar();
- /* Register the default set of converters */
- HTConverterInit(converters);
-
- /* Register the default libwww HTML parser */
- if (HTMLParser) HTMLInit(converters);
-
- /* Set the convertes as global converters for all requests */
- HTFormat_setConversion(converters);
-
- /* Register the default set of transfer encoders and decoders */
- HTTransferEncoderInit(transfer_encodings);
- HTFormat_setTransferCoding(transfer_encodings);
-
- /* Register the default set of content encoders and decoders */
- HTContentEncoderInit(content_encodings);
- if (HTList_count(content_encodings) > 0)
- HTFormat_setContentCoding(content_encodings);
- else {
- HTList_delete(content_encodings);
- content_encodings = NULL;
- }
-
/* Register the default set of MIME header parsers */
HTMIMEInit();
--- 114,119 ----
***************
*** 176,184 ****
/* If the Library is not already initialized then do it */
if (!HTLib_isInitialized()) HTLibInit(AppName, AppVersion);
! if (!converters) converters = HTList_new();
! if (!transfer_encodings) transfer_encodings = HTList_new();
! if (!content_encodings) content_encodings = HTList_new();
/* Register the default set of transport protocols */
HTTransportInit();
--- 179,209 ----
/* If the Library is not already initialized then do it */
if (!HTLib_isInitialized()) HTLibInit(AppName, AppVersion);
! if (!converters) {
! converters = HTList_new();
! /* Register the default set of converters including the HTML parser */
! HTConverterInit(converters);
! HTMLInit(converters);
! /* Set the converters as global converters for all requests */
! HTFormat_setConversion(converters);
! }
! if (!transfer_encodings) {
! transfer_encodings = HTList_new();
! /* Register the default set of transfer encoders and decoders */
! HTTransferEncoderInit(transfer_encodings);
! HTFormat_setTransferCoding(transfer_encodings);
! }
! if (!content_encodings) {
! content_encodings = HTList_new();
! /* Register the default set of content encoders and decoders */
! HTContentEncoderInit(content_encodings);
! if (HTList_count(content_encodings) > 0)
! HTFormat_setContentCoding(content_encodings);
! else {
! HTList_delete(content_encodings);
! content_encodings = NULL;
! }
! }
/* Register the default set of transport protocols */
HTTransportInit();
***************
*** 200,225 ****
/* Get any proxy or gateway environment variables */
HTProxy_getEnvVar();
- /* Register the default set of converters including the HTML parser */
- HTConverterInit(converters);
- HTMLInit(converters);
-
- /* Set the convertes as global converters for all requests */
- HTFormat_setConversion(converters);
-
- /* Register the default set of transfer encoders and decoders */
- HTTransferEncoderInit(transfer_encodings);
- HTFormat_setTransferCoding(transfer_encodings);
-
- /* Register the default set of content encoders and decoders */
- HTContentEncoderInit(content_encodings);
- if (HTList_count(content_encodings) > 0)
- HTFormat_setContentCoding(content_encodings);
- else {
- HTList_delete(content_encodings);
- content_encodings = NULL;
- }
-
/* Register the default set of MIME header parsers */
HTMIMEInit();
--- 225,230 ----
Index: Library/src/HTProt.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTProt.c,v
retrieving revision 2.21
diff -c -r2.21 HTProt.c
*** Library/src/HTProt.c 4 Apr 1999 00:09:14 -0000 2.21
--- Library/src/HTProt.c 29 Jul 2005 13:29:41 -0000
***************
*** 63,69 ****
--- 63,71 ----
newProt->preemptive = preemptive;
newProt->client = client;
newProt->server = server;
+
if (!protocols) protocols = HTList_new();
+ else HTProtocol_delete(name); /* Ensure not listed twice */
HTTRACE(CORE_TRACE, "Protocol.... Adding `%s'\n" _ name);
return HTList_addObject(protocols, (void *) newProt);
}
Index: Library/src/HTTrans.c
===================================================================
RCS file: /sources/public/libwww/Library/src/HTTrans.c,v
retrieving revision 2.8
diff -c -r2.8 HTTrans.c
*** Library/src/HTTrans.c 4 Apr 1999 00:09:14 -0000 2.8
--- Library/src/HTTrans.c 29 Jul 2005 13:29:41 -0000
***************
*** 42,47 ****
--- 42,48 ----
tp->input_new = get_input;
tp->output_new = get_output;
if (!transports) transports = HTList_new();
+ else HTTransport_delete(name); /* Ensure not listed twice */
HTTRACE(CORE_TRACE, "Transport... Adding `%s'\n" _ name);
return HTList_addObject(transports, (void *) tp);
} else HTTRACE(CORE_TRACE, "Transport... Can't add this...\n");
Received on Sunday, 31 July 2005 07:22:13 UTC