W3C home > Mailing lists > Public > public-tdmrep@w3.org > February 2021

Re: page about robots.txt

From: Brendan Quinn <brendan@cluefulmedia.com>
Date: Thu, 25 Feb 2021 16:02:08 +0200
Message-ID: <CAMvELkeaLVKGXaJPBAkmVTS4y3F4kQgJR9VV-tpbiKAovz-JsQ@mail.gmail.com>
To: laurent.lemeur@edrlab.org
Cc: public-tdmrep@w3.org
Thanks Laurent, that looks good.

It's probably worth mentioning that there are some provider-specific
extensions to robots.txt used in the wild, eg sitemap: used by "Google,
Bing,and other major search engines".


I guess we should also document the .well-known folder, with spec here:
https://tools.ietf.org/html/rfc8615 and the quite extensive "well-known URI
repository" at

Also see IAB's ads.txt initiative: https://iabtechlab.com/ads-txt/

Sorry I missed the call on Tuesday. I hope it was fruitful.

Best regards,


On Thu, 25 Feb 2021 at 15:29, Laurent Le Meur <laurent.lemeur@edrlab.org>

> Dear participants,
> I have added a page to the Github repo, which tries to summarize what is
> robots.txt and how it is used. Robots.txt has been described by Ivan Herman
> as a possible source of inspiration during our last call.
> https://github.com/w3c/tdm-reservation-protocol/blob/main/docs/robots.md
> Best regards
> Laurent Le Meur
Received on Thursday, 25 February 2021 14:02:34 UTC

This archive was generated by hypermail 2.4.0 : Thursday, 25 February 2021 14:02:35 UTC