Re: Retry-After header on 20X response -- HTTP/1.1 spec extension?

I would like to propose a change to the HTTP specification, to permit the
Retry-After header to be used on a 20x HTTP response.

I don't expect browsers to respect this.  My target is cooperative automated
harvesting agents, such as
http://www.openarchives.org/OAI/openarchivesprotocol.htm<http://www.openarchives.org/OAI/openarchivesprotocol.html>

The scenario here are robots that harvest huge data sets, but are
deliberately rate limited.

If I can legally return Retry-After with the 20x result, it will cut the
total requests in half.  Clients can ask for data, and know immediately how
long to wait before they ask again.  Only a client that violates the timeout
would ever see a 503.

The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is for
503 and 3xx return codes only.

*Under the proposal*
Client request data
Server sends 200 response, with Retry-After 5 seconds
Client waits 5 seconds and requests data
Server sends 200 response, with Retry-After 3 seconds

*Steps required under the current specification:*
Request data
200 response
Request data
503 response, Retry-After 5 seconds
(Wait 5 seconds) Request data
200 response
Request data
503 response, Retry-After 5 seconds





On Wed, Apr 15, 2009 at 12:01 AM, Mark Nottingham <mnot@mnot.net> wrote:

> Hi Bryce,
>
> What's the advantage of using Retry-After over Cache-Control here?
>
> Cheers,
>
>
>
> On 05/01/2009, at 2:46 PM, Bryce Nesbitt wrote:
>
>  Dear http-wg members.
>>
>> Where would I go to propose a specification change to HTTP such as the one
>> below (allowing optional Retry-After headers in a 20X response)?  This is a
>> backwards compatible change, and need not have any browser support to be
>> valuable to cooperating automated harvesting robots (e.g.
>> http://www.openarchives.org/OAI/openarchivesprotocol.html ).
>>
>> On Mon, Dec 22, 2008 at 8:51 PM, Bryce Nesbitt <bnesbitt@bepress.com>
>> wrote:
>> Dear Working Group Folks,
>>
>> I am not a member of the working group.  But I have recently been tempted
>> to "stretch" the HTTP spec, and I'm writing to inquire if what I'm doing is
>> reasonable enough to eventually fold into the spec.
>>
>> Basically I'm sending a Retry-After header on a 20x HTTP response.
>>
>> I'm working with a "throttled" data service which rate limits connections.
>>   Clients are harvesting a huge volumes of data over time.  Presently
>> clients get some data with a 200 result, ask again right away and get a 503
>> response, then wait out the proper Retry-After time.
>>
>> If I can return Retry-After with the 20x result, it will cut the total
>> requests in half.  Clients can ask for data, and know immediately how long
>> to wait before they ask again.  Only a client that violates the timeout
>> would ever see a 503.
>>
>> The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is
>> for 503 and 3xx return codes only. Your thoughts?  Where would I go to
>> suggest an expansion of the Retry-After header, to be inclusive of 20x
>> results?  Is this a reasonable extension in your view?
>>
>>
>
>
>
>
>
> Dear Working Group Folks,
>
>
>> I am not a member of the working group.  But I have recently been tempted
>> to "stretch" the HTTP spec, and I'm writing to inquire if what I'm doing is
>> reasonable enough to eventually fold into the spec.
>
>
>> Basically I'm sending a Retry-After header on a 20x HTTP response.
>
>
> I'm working with a "throttled" data service which rate limits
> connections.   Clients are harvesting a huge volumes of data over time.
> Presently clients get some data with a 200 result, ask again right away and
> get a 503 response, then wait out the proper Retry-After time.
>
>
> If I can return Retry-After with the 20x result, it will cut the total
> requests in half.  Clients can ask for data, and know immediately how long
> to wait before they ask again.  Only a client that violates the timeout
> would ever see a 503.
>
>
> The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is
> for 503 and 3xx return codes only. Your thoughts?  Where would I go to
> suggest an expansion of the Retry-After header, to be inclusive of 20x
> results?  Is this a reasonable extension in your view?
>
>  --
> Mark Nottingham     http://www.mnot.net/
>
>
>


-- 
Bryce Nesbitt
The Berkeley Electronic Press
bepress: sustainable scholarly publishing