Re: is libwww thread safe?

way back when, I did a pthreads implementation of libwww (on the old
4.x code base). 

Good news: this wasn't too hard; I basically just tossed around a
bunch of local locks on things.

Bad news: Performance bites. I mean, really, really bites. Like Mike
Tyson who hasn't eaten for days. 

At issue: libwww uses a asynchronous I/O based on the select() system
call. It's really not threads, all it is is the same stuff everyone
used back in the 70s to do routers and things like that. Just lots of
FSMs. :)

This is actually an amazingly efficient way to do things. What
happened when I put threads around it is that every SINGLE connection had
this HUGE amount of baggage. Select() is a pretty hefty call, and to
do it when there's only one connection outstanding is bad; to call one 
select() for every single connection is really bad. That's what the
threads ended up doing. Oops.

Also: just for fun, most OSes have a user thread limit, typically
around 512 or 1024. So, if you use one thread per connection, and you
do a lot of connections, you run out of threads _really_ quickly. You
also can't fork any processes while you're at the thread limit, which
brings everything to an ungraceful halt really quickly.

If you're able to do a thread-based implementation around the 
select() mechanism (say for a multiprocessor, where each processor
gets a thread), that would be way cool. But it's a harder issue,
because you need to have multiple select() calls doing stuff. However, 
the performance gains should be there.

Cheers,
-e

Received on Monday, 23 March 1998 17:30:40 UTC