Re: Keeping crawlers up-to-date

Hi,

My only concern about this is that you need to limit the number of
> items in the feed. If you have a sudden burst of activity and the
> crawler just ping the feed at regular intervals, it may miss some
> updates. However, even with 1M updates in a day, with a feed capped to
> 100 items would just need the crawlers to ping the feed about every
> hour and a half. So that's not too bad.
> (Just noticed that Soren's proposal includes pagination of feeds,
> which might solve that problem).


I need to have a look at Soren's proposal, but that does sound like it'd
help.

IMO using a RESTful solution that has URIs for specific resources seems to
play nicer with web architecture and the general idea of linked data :)

Cheers,

L.


-- 
Leigh Dodds
Programme Manager, Talis Platform
Talis
leigh.dodds@talis.com
http://www.talis.com