RE: Retry safety of HTTP requests

@Mark
Thanks for your comments. I'm more interested to have a "I really mean this is safe" for POST-like methods and dealing with methods that aren't already safe.

The mention of CORS was not to be used as a motivation for this but to highlight real world problems that people have. The more appropriate motivation is that "I'm in a bad network, I have to retry this POST, is it safe to do that?". It's unfortunate that got mixed up in the email ordering.

Patrick brought up a point that this could be handled in the application layer itself, and Amos brought up good points that this could be handled by the browser itself, and HTTP doesn't need to know about this. These are totally legitimate, and that's a layer to handle it. 

However, I think this could be useful to deal with in HTTP because people would end up accessing the same resources over mobile apps or browsers and it's probably useful to have a common meaning, especially if this could be used to express things like security properties of the request to the transport for TLS1.3.

@Patrick I was thinking of writing a draft and sending it to the HTTP workshop that's coming up. Maybe that's a good venue to discuss this.

Subodh
________________________________________
From: Mark Nottingham [mnot@mnot.net]
Sent: Tuesday, March 22, 2016 5:33 PM
To: Subodh Iyengar
Cc: ietf-http-wg@w3.org
Subject: Re: Retry safety of HTTP requests

Hi Subodh,

My .02 - It sounds like what you're asking for is a "I really mean this is safe" flag for GET.

Historically, this kind of mechanism doesn't work out very well, because if people misunderstand the original semantics (in this case, the safety and idempotency of GET), they're just as likely to misunderstand/misuse the "I really mean it" flag. Then, you need a "I really, REALLY mean it" flag, and so on.

Furthermore, motivating this with the use case that people are trying to work around security mechanisms like CORS isn't likely to work; it's be better to fix any problems in CORS than create workarounds in other specs*.

Explicitly allowing retries of POST (or POST-like methods) is something that might get more traction. Recently we've seen a proposal for a safe, idempotent SEARCH method; would that meet your needs? <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dsnell-2Dsearch-2Dmethod&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=iWWfZHGVJvuoN_CUfirnYHIs5eLjiJ3VRBBfYgC2j4I&e= >.

Another possibility would be a pattern of use that assured that non-idempotent requests weren't able to be retried; e.g., <https://github.com/mnot/I-D/blob/gh-pages/Abandoned/http-poe/draft-nottingham-http-poe-00.txt> (I abandoned this draft because it's not really a new protocol element, it's just a pattern of use.).

Cheers,

* CORS is now maintained as part of Fetch <https://urldefense.proofpoint.com/v2/url?u=https-3A__fetch.spec.whatwg.org&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=T0ixsfqeSSHggCjL88RL8WpxmPPAHB0IioI2X8yAFts&e= >, and Anne is very receptive to proposed improvements; I think a lot of people recognise its shortcomings.

P.S. Our "snow bunny" has been banned from the list; apologies for the noise.


> On 22 Mar 2016, at 7:27 PM, Subodh Iyengar <subodh@fb.com> wrote:
>
> Thanks for the comments Amos.
>
>> Application layer mechanisms are outside the scope of HTTP.
> I was presenting an example use case of retry safety by an application. The mechanism of indicating this is definitely outside of the scope of HTTP.
>
>> If the TLS/1.3 layer is going to perform packet retry itself in the
>> presence of 0-RTT flight data, then it needs to ensure that opaque data
>> is only ever delivered to a recipient once regardless of how many times
>> its actually sent on-wire. To be a "reliable transport" it needs to
>> guarantee that property.
>
> I should have added more context here.
> The property of retry safety for TLS is not for reliability here, but for security of 0-RTT data.
> In TLS1.3, the 0-RTT payload is not protected against replay attacks.
> Only idempotent or retry safe data should be sent in the 0-RTT payload. This would enable
> even requests which do not have idempotence or safe semantics to be used in 0-RTT data if
> the application takes care of the idempotency. The exact mechanism for the application taking care
> of this are outside the scope of HTTP.
>
>> Do not confuse transport layer frame/message with transfer layer
> frame/message.
> I'm not. The idea here was to provide a mechanism to indicate to HTTP that this request is retry safe, and then the HTTP layer can propagate this information to the TLS client, hence it is free to send its data in the 0-RTT payload.
>
>> What? thats false.
>
> I was mistaken and you're correct on this. A lot of applications though practically do send non-idempotent GET requests. I can understand why the WG considers them non-compliant. However these considerations do not take into account restrictions of app developers for example security features like CORS, and I can imagine many applications that might use the methods like POST, GET. and HEAD to get around CORS restrictions to build legitimate features.
>
>>> It tells the HTTP layer by using the correct RESTful method for the
>> semantic operation that message seeks to perform:
>> - a fetch (GET)
>> - a store (PUT)
>> - a removal (DELETE)
>> - a submission (POST)
>> - a state query (HEAD)
>> - a service capabilities query (OPTIONS, TRACE)
>
> Even if we used the correct semantics for each operation, an application could want POST semantics yet want to allow the request to be retried.
>
>> TLS is just a transport for HTTP. There is no reason (nor should there
>> be) for any difference in semantics of the HTTP messages than over
>> TLS/1.2 or TCP plain-text transports.
>
> For the above mentioned reasons, the semantics of the HTTP messages are important for TLS1.3.
>
> I agree *how* is difficult for HTTP to define, however HTTP can for example impose limits on retries, for example, you shouldn't retry a message whose response you've even partially delivered to the client. HTTP might also be able to define certain best practices for browsers that perform retries, and applications that ask for retries, and provide mechanisms for browsers to indicate to servers how many retries have been performed so far similar to :scheme and :path.
>
> Subodh
> ________________________________________
> From: Amos Jeffries [squid3@treenet.co.nz]
> Sent: Tuesday, March 22, 2016 12:32 AM
> To: ietf-http-wg@w3.org
> Subject: Re: Retry safety of HTTP requests
>
> On 22/03/2016 6:55 p.m., Subodh Iyengar wrote:
>> HTTP/1.1 and the current HTTP/2 spec don't define the concept of
>> retry safety of requests, i.e. when are requests are safe to retry
>> and what are the limits on how retries can be performed by the lower
>> layers.
>
>
> RFC 7230 section 6.3.1 titled "Retrying Requests"
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__tools.ietf.org_html_rfc7230-23section-2D6.3.1&d=CwIC-g&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=dqzc-LMd_ldrS1bDvcdcNQqv8Bqf6kkvaOyNBWSMeqg&s=OC5WwQavs-zI-QCuNCeV99nz6sTtsOcG23ffZlJH8yg&e= >
>
> The criteria for *when* are outlined.
>
> The *how* is not possible to be defined at the HTTP layer IMO. There are
> just so many possible ways. Up to and including things like RFC 1149
> Avian Carrier protocol as fallback.
>
> So the "how" is application and indeed message specific and should be
> defined at those upper layers. HTTP already provides a number of
> mechanisms such as; GET, HEAD methods and/or ETag, If-*, Range headers,
> etc. which can be used by the application layer to check the state
> before its chosen method of retry (if wanted).
>
>
>>
>> I believe this is a new item and I'm trying to gauge people's
>> interest in this topic, and whether such an idea has been explored
>> before in the httpwg.
>>
>> I think defining retry safety will become an important problem in the
>> near future. I can see several use cases if we have a proper
>> definition of retry safety in HTTP:
>>
>> 1) Usually an application like a piece of javascript will create a
>> HTTP request and send it to a browser or mobile app which executes
>> it. The browser or a mobile app might want to retry the request. In
>> unreliable networks, retrying requests is important for reliability.
>> The application could decide that it would create a non idempotent
>> request like a POST, and have an application layer mechanism to
>> enforce idempotence.
>
> Application layer mechanisms are outside the scope of HTTP.
>
> HTTP is not even the only protocol such messages can be sent over. There
> is HTTP (1.x, 2), WebSockets, SPDY, QUICK, COAP(S),  and others all in
> use various places.
>
>> It could then tell the browser that the request
>> was retry safe because it was enforcing idempotence, and let the
>> browser perform optimizations to retry it depending on network
>> conditions, including caching the request offline, retrying when the
>> network comes back, sending the request across 2 connections on
>> different interfaces, etc.
>
> It tells the HTTP layer by using the correct RESTful method for the
> semantic operation that message seeks to perform:
> - a fetch (GET)
> - a store (PUT)
> - a removal (DELETE)
> - a submission (POST)
> - a state query (HEAD)
> - a service capabilities query (OPTIONS, TRACE)
>
> All HTTP software understands those semantics regardless of what other
> extensions it may or may not support. Or how old it is.
>
>>
>> 2) Some transport protocols which treat retry safe requests
>> differently from non retry safe requests. For example in TLS 1.3,
>> idempotent requests may be sent in a 0-RTT flight, which reduces the
>> latency for the request. An application might desire a non idempotent
>> request which is retry safe be sent in this 0-RTT flight.
>
> Do not confuse transport layer frame/message with transfer layer
> frame/message.
>
> The semantics of the transfer layer (HTTP) message and even its framing
> boundaries are opaque to the transport layer (TLS).
>
> If the TLS/1.3 layer is going to perform packet retry itself in the
> presence of 0-RTT flight data, then it needs to ensure that opaque data
> is only ever delivered to a recipient once regardless of how many times
> its actually sent on-wire. To be a "reliable transport" it needs to
> guarantee that property.
>
> I find it hard to believe that the TLS WG would allow TLS/1.3 to become
> an unreliable transport layer.
>
>
>>
>> 3) A non-application server such as a load balancer might also desire
>> to know whether or not the client expressed that this request is
>> retry safe so that it may retry this request if the backend fails
>> over offering more reliability to clients.
>>
>> Currently, the only semantics for an application to express retry
>> safety is via idempotency (the request method). The status quo shows
>> that even this is not expressive enough, since even though GETs are
>> not technically idempotent, some browsers will retry them anyway.
>
> What? thats false.
>
> RFC 7231 section 4.2.2:
> "PUT, DELETE, and safe request methods are idempotent".
>
> RFC 7231 section 4.2.1:
>  "GET, HEAD, OPTIONS, and TRACE methods are defined to be safe".
>
> thusly "idempotent" == PUT, DELETE, GET, HEAD, OPTIONS, TRACE
>
> plus any other extension method defined as being idempotent or safe.
>
>
> Note that certain response status codes also indicate explicitly that
> nothing reached the origin. So their request is now retriable for
> exactly +1 try despite its natural idempotency.
>
> HTTP/2 extended that with GOAWAY to mark a range of pipelined request
> streams.
>
>
> IIRC we have a WG consensus that applications using methods counter to
> their documented idempotency are not HTTP compliant. Such software is
> explicitly left to face the consequences of its non-compliance when the
> HTTP layer "unexpectedly" treats the message according to correct semantics.
> Your application(s) treating GET as non-idempotent is a case for that.
>
>
>>
>> Having an document which would define a new property like
>> retry-safety would help apps take advantage of reliability features
>> of browsers for requests other than GETs and also prepare HTTP to be
>> able to use TLS1.3 effectively.
>
> TLS is just a transport for HTTP. There is no reason (nor should there
> be) for any difference in semantics of the HTTP messages than over
> TLS/1.2 or TCP plain-text transports.
>
>>
>> In our Facebook mobile apps we annotate all the requests that our app
>> makes as retry safe or not and that helps us decide how retry
>> policies should behave for each request. It's been very useful for
>> request reliability.
>
> Why not use the REST semantics of HTTP itself as the signal?
>
>>
>> Would the HTTP-wg be interested in a document describing retry-safety
>> and its limitations?
>
> I'm certainly interested to find out why HTTP semantics are suddenly so
> insufficient that its needed. IME Facebook was turning into one of the
> better HTTP participants before it abandoned for HTTPS-world.
>
> Amos
>
>

--
Mark Nottingham   https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mnot.net_&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=mt5I39QkpkVJ75Kv2PIImSivFtEsbIwH9hHPI2zXY_M&e=

Received on Wednesday, 23 March 2016 01:04:09 UTC