- From: Pim Lemmens <wsinpim@win.tue.nl>
- Date: Thu, 4 Apr 1996 16:20:20 +0200 (MET DST)
- To: www-lib@w3.org
Hello, Currently I am trying to add robot exclusion support to our fish search program. In order to find out what objects are not accessible to robots at a given server, the program should load the 'robots.txt' file from the server root directory and interprete it. I have incorporated the interpreter in a (unstructured) stream that is supposed to receive its input in "text/plain" format. A robot exclusion stream object is created by HTStream* RobExStream_new(char* robot, Queue* NoGoList); that reads an ASCII stream and produces a list of forbidden addresses. It all has been tested OK, but I can't succeed to make it work in an actual application, using the www library. What is wrong with the following: ..... HTRequest_setAnchor(request, HTAnchor_findAddress(URL)); HTRequest_setOutputFormat(request, WWW_PLAINTEXT); HTRequest_setOutputStream(request, RobExStream_new("fish-search", &((*SE)->NoGoList))); ..... dest = (HTAnchor*)HTRequest_anchor(request); ..... if (!HTLoadAnchor(dest, request)) /* request not accepted */ status = ExceptionHandler(request); else status = HT_OK; ..... Stream tracing produces the following: ........... HTNet_new... starting request 7bd30 (retry=2) with net object 7c460 StreamStack. Constructing stream stack for text/html to text/plain HTNetDelete. Object and call callback functions HTNetDelete. 7c528 not registered! HTNet_delete Remove net object 7c528 HTNet_delete closing 5 HTNetDelete. Object and call callback functions HTNet_delete Remove net object 7c460 ........... Any idea where that unregistered object comes from? Greetings, -- Pim Lemmens Mathematics & Computing Science wsinpim@win.tue.nl Eindhoven University of Technology http://www.win.tue.nl/win/cs/is/wsinpim/ P.O. Box 513 tel. (((*31)(0)40)247)3755 5600 MB Eindhoven Netherlands
Received on Thursday, 4 April 1996 09:20:31 UTC