W3C home > Mailing lists > Public > public-lod@w3.org > April 2009

Re: Keeping crawlers up-to-date

From: Leigh Dodds <leigh.dodds@talis.com>
Date: Tue, 28 Apr 2009 18:28:33 +0100
Message-ID: <f323a4470904281028w5d729ad1t6f7f451ea0350cf@mail.gmail.com>
To: Yves Raimond <yves.raimond@gmail.com>
Cc: Linking Open Data <public-lod@w3.org>, Nicholas J Humfrey <njh@aelius.com>, Patrick Sinclair <metade@gmail.com>

My only concern about this is that you need to limit the number of
> items in the feed. If you have a sudden burst of activity and the
> crawler just ping the feed at regular intervals, it may miss some
> updates. However, even with 1M updates in a day, with a feed capped to
> 100 items would just need the crawlers to ping the feed about every
> hour and a half. So that's not too bad.
> (Just noticed that Soren's proposal includes pagination of feeds,
> which might solve that problem).

I need to have a look at Soren's proposal, but that does sound like it'd

IMO using a RESTful solution that has URIs for specific resources seems to
play nicer with web architecture and the general idea of linked data :)



Leigh Dodds
Programme Manager, Talis Platform
Received on Tuesday, 28 April 2009 17:29:18 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 7 January 2015 15:15:56 UTC