RE: [CT] Using robots.txt to flag an adapting site

Hey, if you feel it necessary to apologise, what should Luca feel!

Thanks again for the contribution, we'd like to see more of you in CT

Cheers
Jo


> -----Original Message-----
> From: public-bpwg-ct-request@w3.org
[mailto:public-bpwg-ct-request@w3.org]
> On Behalf Of Rotan Hanrahan
> Sent: 29 September 2007 16:02
> To: public-bpwg-ct@w3.org
> Subject: RE: [CT] Using robots.txt to flag an adapting site
> 
> 
> First, apologies to Jo for not posting my original message to the
> appropriate public list.
> 
> For the benefit of the public record, I append below the original
> message that started the thread. As indicated in my message, the
> metadata held in the robots.txt file applies to the server. It is
> "site-wide". I also agree with the suggestion in the subsequent thread
> that POWDER could be equally useful in this case. My motivation for
> mentioning robots.txt was merely to enrich the pool of possible
> solutions.
> 
> Furthermore, for cases where neither robots.txt, POWDER or any other
> mechanism was present, I suggested a strategy that could be employed
by
> proxies to identify and record for themselves if they were dealing
with
> adapting servers.
> 
> The original text follows:
> ===========================================
> 
> 
> 
> I would like to throw into the pot an idea I mentioned back in June
[1],
> which is that of using the robots.txt file to flag to a proxy that the
> server is an adapting server. The robots.txt is extensible [2] so
there
> should be no problem adding a custom extension to indicate that a site
> is adapting, mobile-specific, takes-all-comers, demands desktop etc.
The
> default would be the "long tail" position: this site was designed with
> the assumption that a big clunky PC would be the client.
> 
> I think this could help the search engines and proxy solutions.
> 
> Meanwhile, proxies could check that a complete site is adapting by
> probing with a simple simulated browser request to the home page (for
> any site it has never seen before). If the proxy remembers the kind of
> response it got, it could "do the right thing" more often than not.
> 
> Thoughts anyone?
> 
> ---Rotan.
> 
> 
> [1] http://lists.w3.org/Archives/Public/public-ddwg/2007Jun/0001.html
> [2] http://www.robotstxt.org/wc/norobots-rfc.html   (section 3.2)

Received on Saturday, 29 September 2007 20:51:47 UTC