Re: Indexing extension

Do any access control systems recognize the From: field?  It
was my impression that they pretty much ignored it, as it was
user settable.  If they did layer it on top of the dns-reported
domain, this would work for situations where access restrictions
were based on hostname or domain; it would not, unfortunately
work in situations where it was IP based.

On another note, several people have pointed out the existence
of the Robot exclusion standard, and have suggested using a robots.txt
at sites or in hierarchies that should not be indexed.  This is
a fine temporary suggestion, but I think it is a bit inelegant, as
it requires the maintainers of those pages to keep two different
forms of access control--one for humans and one for local robots.

Right now, servers will tell browsers which don't have access to a
resource that access control exists for that resource (in the form
of an error code), but they will not tell browsers that do have
access that access control exists.  For the purpose of local indexing,
I believe that adding some way to request that information would 
be a valuable information.  That way it doesn't get sent all the time
when the browsers aren't interested.  

Perhaps a Pragma method of "request restrictions" would be the best
idea; it would allow the server to determine whether to send a description
of the restrictions to the browser (useful if the indexer wishes to
use some logic to determine whether to index) or a simple "restrictions
exist" reply.

What do people think of using that Pragma method as a solution?

				Regards,
					Ted Hardie
					NAIC



> 
> I think the "right" solution is to have some kind of indication of
> the identity of the person or process the access is performed on behalf of.
> For instance:
> 
> - A cache server should indicate who does access (and what set of others
>   it will grant access to without asking permission)
> - An index server should indicate the set of people it does access for
> 
> And so on. Of course, the queried server should respond according to
> min(trust in accessing server, trust in claimed user).
> 
> I would think that the HTTP level is the right level to attack this problem.
> What about overloading the From: field with the value "anybody@anywhere"?
> Just a random suggestion......
> 
>              Harald A
> 

Received on Wednesday, 31 May 1995 10:43:17 UTC