RE: [CT] Using robots.txt to flag an adapting site

First, apologies to Jo for not posting my original message to the
appropriate public list.

For the benefit of the public record, I append below the original
message that started the thread. As indicated in my message, the
metadata held in the robots.txt file applies to the server. It is
"site-wide". I also agree with the suggestion in the subsequent thread
that POWDER could be equally useful in this case. My motivation for
mentioning robots.txt was merely to enrich the pool of possible
solutions.

Furthermore, for cases where neither robots.txt, POWDER or any other
mechanism was present, I suggested a strategy that could be employed by
proxies to identify and record for themselves if they were dealing with
adapting servers.

The original text follows:
===========================================



I would like to throw into the pot an idea I mentioned back in June [1],
which is that of using the robots.txt file to flag to a proxy that the
server is an adapting server. The robots.txt is extensible [2] so there
should be no problem adding a custom extension to indicate that a site
is adapting, mobile-specific, takes-all-comers, demands desktop etc. The
default would be the "long tail" position: this site was designed with
the assumption that a big clunky PC would be the client.

I think this could help the search engines and proxy solutions.

Meanwhile, proxies could check that a complete site is adapting by
probing with a simple simulated browser request to the home page (for
any site it has never seen before). If the proxy remembers the kind of
response it got, it could "do the right thing" more often than not.

Thoughts anyone?

---Rotan.


[1] http://lists.w3.org/Archives/Public/public-ddwg/2007Jun/0001.html
[2] http://www.robotstxt.org/wc/norobots-rfc.html   (section 3.2)

Received on Saturday, 29 September 2007 15:02:23 UTC