Re: multiple requests
> Sacha writes:
> > How can I send off multiple requests to various hosts?
> > Will these be handled 'intelligently'? I know next to nothing
> > about how all the network stuff works, but I should think
> > processing all the requests to one host in one go would be
> > more efficient than reconnecting N times.
> > Does the library do this for me or do I have to program it
> > explicitly?
> This is handled in such a way that it is completely transparent to the
> application. The application operates in "requests" which is creating a
> request object and passing it to libwww. A request may be to the same host
> or to multiple hosts - it doesn't matter. The net manager (HTNet.c) handles
> all connections including if the remote server is capable of using
> persistent connections.
> Therefore, as an application you can safely issue new requests at any point
> in time. If the number of requests is bigger than the number of open sockets
> then the requests are queued up in the HTNet manager and started when a
> socket is available.
Ok... now the question is, how can I send off a batch of requests at the
same time? I mean, how can I tell the library:
o here is a bunch of URLs (requests)
o go get them
o let me know when you're done
I looked at the robot, which seems to be the example most likely to fetch
multiple documents. Broadly speaking, it seems to:
o start at one URL (HTLoadAnchor in main)
o go into the event loop
o parse incoming data (is this what HText functions do?)
o send off a new request when it sees a url in the incoming data
o on termination of a request check whether any other requests are
Now let's assume I have a list of requests to send off in one batch. Does this
mean that I would have to:
1. provide a function for the event loop to select whenever it is
idle (how would I do this?), in which a request is taken off the
list and sent (let's call the function 'dispatch')
2. pick one request, make it pre-emptive, and send it
3. go into the event loop
4. when the list of requests to send is empty, un-register the
'dispatch' function so that the event loop no longer selects it
Would I have to do step 2 at all?
Is this the right/only/best way to do it?
Or is there an HTLoad_X function somewhere which takes a list of requests?