Update robots.txt standards Community Group Launched

With your support, the Update robots.txt standards Community Group has been launched:
  http://www.w3.org/community/robotstxt/

This group was originally proposed on 2023-10-02
by Hans Petter Blindheim. The following people supported its
creation: 
      Hans Petter Blindheim
      Brijesh Gohil
      Robin Berjon
      Max Gendler
      Hallison Brancalhão
      Gustavo Henrique Quinalha
  
To join the group, please use:
 http://www.w3.org/community/robotstxt/join

Please note that supporting a group is different from joining
a group. Supporters must also enroll if they wish to participate.

--------------------
Robots.txt is currently based on opting out of what you do not want your
website to be a part of. 

This is hard to maintain (almost a full time job right now) if you do
not wish for your websites content to be applied for e.g. training AI,
be a part of market research (e.g. price robots), to be a part of
non-search engine databases and more.

This proposal is to update what type of instructions robots.txt should
support to rather be treated as an opt-in, where you can give
instructions based on intent of robots rather than a wildcard or in
granular detail.

Example: 
Agent-group: searchengines

Applies to all robots that seeks to update, process or maintain websites
for search engine databases. Does not grant permission to apply scraped
data for AI purposes (this should have its own Agent-group).

Also, the absence of instructions should be treated as not having opted
in, and for robots working on behalf of AI, there might need to be
additional instructions (e.g. max-snippet and if you require a citation
if your content is applied to provide an answer).
--------------------

Thank you,

W3C Community Development Team

Received on Friday, 2 February 2024 16:34:21 UTC