I, robot

Hey all,

I may have missed it in the archives, but I didn't see any topics on which
sections of a document a robot or spider or other Web traveller might examine
to determine content information.

I suppose that this sort of information could be contained in the <META> tag,
like

<META HTTP-EQUIV="Keywords" CONTENT="Audio/Video Equipment">
<META HTTP-EQUIV="Keywords" CONTENT="Laserdiscs sold here!">

but is this indeed what robots look for besides (I guess) the first few lines
of the document itself?

If there isn't a specific indexing method that is standard for all search
engines/robots/crawlers, shouldn't there be one?


thanks,


steve


_____________________________________________________________________________
Can't think of anything right now, other than the traditional e-mail address:
steve@mordred.gatech.edu

Received on Monday, 11 December 1995 15:18:26 UTC