W3C home > Mailing lists > Public > public-bpwg-ct@w3.org > September 2007

RE: [CT] Using robots.txt to flag an adapting site

From: Jo Rabin <jrabin@mtld.mobi>
Date: Sat, 29 Sep 2007 21:51:21 +0100
Message-ID: <C8FFD98530207F40BD8D2CAD608B50B4732B8B@mtldsvr01.DotMobi.local>
To: "Rotan Hanrahan" <rotan.hanrahan@mobileaware.com>, <public-bpwg-ct@w3.org>

Hey, if you feel it necessary to apologise, what should Luca feel!

Thanks again for the contribution, we'd like to see more of you in CT

Cheers
Jo


> -----Original Message-----
> From: public-bpwg-ct-request@w3.org
[mailto:public-bpwg-ct-request@w3.org]
> On Behalf Of Rotan Hanrahan
> Sent: 29 September 2007 16:02
> To: public-bpwg-ct@w3.org
> Subject: RE: [CT] Using robots.txt to flag an adapting site
> 
> 
> First, apologies to Jo for not posting my original message to the
> appropriate public list.
> 
> For the benefit of the public record, I append below the original
> message that started the thread. As indicated in my message, the
> metadata held in the robots.txt file applies to the server. It is
> "site-wide". I also agree with the suggestion in the subsequent thread
> that POWDER could be equally useful in this case. My motivation for
> mentioning robots.txt was merely to enrich the pool of possible
> solutions.
> 
> Furthermore, for cases where neither robots.txt, POWDER or any other
> mechanism was present, I suggested a strategy that could be employed
by
> proxies to identify and record for themselves if they were dealing
with
> adapting servers.
> 
> The original text follows:
> ===========================================
> 
> 
> 
> I would like to throw into the pot an idea I mentioned back in June
[1],
> which is that of using the robots.txt file to flag to a proxy that the
> server is an adapting server. The robots.txt is extensible [2] so
there
> should be no problem adding a custom extension to indicate that a site
> is adapting, mobile-specific, takes-all-comers, demands desktop etc.
The
> default would be the "long tail" position: this site was designed with
> the assumption that a big clunky PC would be the client.
> 
> I think this could help the search engines and proxy solutions.
> 
> Meanwhile, proxies could check that a complete site is adapting by
> probing with a simple simulated browser request to the home page (for
> any site it has never seen before). If the proxy remembers the kind of
> response it got, it could "do the right thing" more often than not.
> 
> Thoughts anyone?
> 
> ---Rotan.
> 
> 
> [1] http://lists.w3.org/Archives/Public/public-ddwg/2007Jun/0001.html
> [2] http://www.robotstxt.org/wc/norobots-rfc.html   (section 3.2)
Received on Saturday, 29 September 2007 20:51:47 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 8 January 2008 14:10:36 GMT