Re: HTTP Partial POST Replay

Hi Alan,

If I assume the use case as you describe, where the RTT between the proxy
and the server is tiny, and that the transactions are small, doesn't that
mean it would be better to just let any existing requests complete before
shutting down?  It seems like a better strategy would be to eliminate new
requests to the new instance (e.g. changing the routing at the proxy) and
then wait until all existing (short) requests are completed.  There are
some corner cases, but that seems more straightforward.  In the restart
case, you could have the new instance on the server take over accepting new
requests while the old instance stays active to resolve those that are open.

Matt


On Mon, Jul 1, 2019 at 12:18 PM Alan Frindell <afrind@fb.com> wrote:

> Thanks for taking the time to read and comment.  My responses are inline.
>
>     Overall impression is that this is a overly complex and resource
>     expensive replacement for the very simple 307 status mechanism. There
> is
>     no harm in telling a client it has to retry its reuqest.
>
> I disagree that there is no harm in asking the client to retry the
> request.  In our case, the intermediary (load balancing reverse proxy), is
> < 1 millisecond away from the server, but hundreds of milliseconds away
> from the client.  Further, the client may be charged for bandwidth, and the
> client may have already spent a fair amount of time/bandwidth transmitting
> the request.  I agree with you that for a generic intermediary or forward
> proxy, particularly one very close to the client, the feature doesn't make
> as much sense.  I can clarify that in the draft.
>
> Also, I should mention in the draft that the server can 307 rather than
> abort requests it cannot complete at shutdown time.
>
>     * The stated case of "server wants to shutdown" does not correlate well
>     with the fact that to use this mechanism the server has to store and
>     re-deliver all the initial request bandwidth back to the intermediary.
>
> Again, because the server and intermediary are close together, replaying
> the partial request doesn't prevent the server from shutting down.  The
> type of server and size of a typical request can also make a difference: we
> use this primary for dynamic web traffic (as opposed to something like
> video upload).  These dynamic requests tend not to be huge, and the web
> server was already holding the entire POST body in memory before
> dispatching it to the handler.
>
>     * That re-sending is where the bandwidth issue is coming from. The
>     initial request uses N bytes to arrive from the client, if M of those
>     are a) delivered to each of D servers and b) received back from the
>     initial D-1 servers, and c) deliver to the second server. That makes a
>     total bandwidth consumption of (N + (D-1)*M).
>       Whereas with 307 only consumes (N + M).
>
> It seems you are only counting proxy to server bandwidth.  Assuming the
> intermediary cannot buffer the body, and a 307 triggers the client to
> resend the entire request, that will consume 2*N + (N + M) or (3*N + M),
> compared to N + (N + (D-1)*M) or (2*N + (D-1)*M).
>
>     Also keep in mind that even a blind intermediary just pushing data to a
>     single server is handling twice the traffic that server does. That is
>     the minimum best-case situation. With multiple servers and/or clients
>     the difference increases rapidly to orders of magnitude more resource
>     consumption. That is the existing situation, before this feature even
>     starts to force more resource consumption.
>
> We find that only a fraction of requests in progress at shutdown time are
> POST requests, and of those, an even smaller fraction have incomplete
> bodies say 1 second after shutdown is initiated.  The number of requests
> that get redirected in this manner from single server in our deployment
> (which spans hundreds of thousands of servers) is relatively small.
> Though, we do find that a tiny number get quite unlucky during a web tier
> restart and get replayed a few times.
>
>     * All this re-sending of data could delay the server shutdown an
>     unreasonable amount of time. Turning what would be a few seconds into
>     minutes or even hours. Depending on the intermediary load being
>     reasonable is not a good idea.
>
> I agree with you that in a completely generic and blind case, this is a
> problem.  In our deployment, we find this feature does not impact shutdown
> time of the webserver, which is 10-15 seconds.
>
>     * Every millisecond of delay added by the re-receive and re-send of
> data
>     makes it more likely the client will terminate early. If that happens
>     all this time, bandwidth, memory, and CPU cycles spent are completely
>     wasted.
>
> This is precisely the reason I feel this is a valuable feature.  If the
> client must resend the entire request it's likely to take even longer.
>
>     Consider the case of a system which is undergoing a DoS at the
>     public-facing interface of the intermediary. Enacting this feature is a
>     huge resource expenditure for an already highly loaded intermediary.
>
> If a system is undergoing DoS, I expect it to be able to jettison less
> critical functions to save resources.  Such an intermediary could always
> convert a partial POST replay into a 307 or 500, and reset the upstream
> request.  I can add this to the security considerations.
>
>     * Section 2.1 says "The server MUST have prior knowledge"
>
>     Yet no mechanism(s) are even hinted at how a server may acquire such
>     knowledge. Defining a specific negotiation signal would be far better
>     for this and avoid a huge headache with implementations choosing
>     different signals and mechanisms for negotiating that knowledge.
>
> I agree.  I will elaborate in the draft on feature negotiation.  We set
> this up out of band by configuring the proxies and servers to use this
> feature.  H2 or H3 SETTINGS are appropriate to announce the support for the
> feature, but H1 would most likely need to add a header to each request
> indicating support.
>
>     * The Echo- or Pseudo-Echo mechanism is very clunky. I believe it to be
>     unlikely that any intermediary implementing this feature is unable to
>     simply store the initial request headers for re-use as needed.
>
> That is valuable feedback.  Our implementation stores the initial request
> headers, but I didn't know if that was a valid assumption, so I added the
> Echo- mechanism to the draft.  I can remove it entirely and make it a
> requirement that the request be stored to use the feature.
>
> Thanks
>
> -Alan
>
>