Robot application profile; should recognize 301 automatic redirect?

From http://www.w3.org/Library/src/HTProfil.html#Robot:

Robot Application Profile 

The robot profile contains much of the same functionality as the
client, but it does contain less filters. For example, robots are
normally not interested in performing automatic redirections or access
authentication, and hence this is not part of the robot profile. 

---

Given that a typical behavior of web servers on this request:

	http://host.domain/directory

is a 301 response:

<TITLE>301 Moved Permanently</TITLE>
</HEAD><BODY>
<H1>Moved Permanently</H1>
The document has moved <A HREF="http://host.domain/directory/">here</A>.<P>

Wouldn't we want a robot to follow this automatic redirect?

I modified my own HTProfil.c to use the redirect filter:

HTProfil.c:150

    /* Register some default set of BEFORE and AFTER filters */
    HTNet_addBefore(HTRuleFilter, NULL, NULL, HT_FILTER_MIDDLE); 
    HTNet_addBefore(HTProxyFilter, NULL, NULL, HT_FILTER_MIDDLE); 
    HTNet_addAfter(HTInfoFilter, NULL, NULL, HT_ALL, HT_FILTER_LATE); 

    HTNet_addAfter(HTRedirectFilter,
		   "http://*",	
		   NULL, 
		   HT_PERM_REDIRECT, HT_FILTER_MIDDLE);

Mike


---
Mike Berry, ISC Network Services
Information Systems and Computing, University of Pennsylvania
Voice:(215)/898-1264   Fax:(215)/898-9348     E-mail:<berrym@isc.upenn.edu>

Received on Wednesday, 5 August 1998 17:08:17 UTC