Re: HTTP Partial POST Replay

Thanks for your feedback, comments inline

> If I assume the use case as you describe, where the RTT between the proxy and the server is tiny, and that the transactions are small, doesn't that mean it would be better to just let any existing requests complete before shutting down?

I suppose I’m targeting something of a goldilocks request size – too large to wait (eg: it will take longer than 10-15 seconds to complete), but not so large that retransmitting it to the proxy and back to another server wastes too much bandwidth.  Many of these requests are hundreds of KBs to low tens of MBs, coming from slower clients.  The worst case scenario of course is a large request from a slow client that begins immediately before shutdown.  One can minimize the number of requests that are replayed by delaying the replay until ~1s remains before hard shutdown.

> It seems like a better strategy would be to eliminate new requests to the new instance (e.g. changing the routing at the proxy) and then wait until all existing (short) requests are completed.  There are some corner cases, but that seems more straightforward.

Yes, we do route new requests around a draining webserver, but we cannot wait more than 10-15s before shutting down the old instance.  This constraint comes from the fact that we have a very large number of webservers, and our continuous deployment schedule means we must be able to restart the entire fleet within X minutes.  We found that across our infra we were still failing enough slow requests that we needed to implement something to address it.

>  In the restart case, you could have the new instance on the server take over accepting new requests while the old instance stays active to resolve those that are open.

I agree that this is a good solution.  In our case, however, this is not feasible because the memory requirements of the webserver are such that we cannot run two instances on a single host.  I tried to address this in the Existing Solutions part of the draft.

Thanks

-Alan


On Mon, Jul 1, 2019 at 12:18 PM Alan Frindell <afrind@fb.com<mailto:afrind@fb.com>> wrote:
Thanks for taking the time to read and comment.  My responses are inline.

    Overall impression is that this is a overly complex and resource
    expensive replacement for the very simple 307 status mechanism. There is
    no harm in telling a client it has to retry its reuqest.

I disagree that there is no harm in asking the client to retry the request.  In our case, the intermediary (load balancing reverse proxy), is < 1 millisecond away from the server, but hundreds of milliseconds away from the client.  Further, the client may be charged for bandwidth, and the client may have already spent a fair amount of time/bandwidth transmitting the request.  I agree with you that for a generic intermediary or forward proxy, particularly one very close to the client, the feature doesn't make as much sense.  I can clarify that in the draft.

Also, I should mention in the draft that the server can 307 rather than abort requests it cannot complete at shutdown time.

    * The stated case of "server wants to shutdown" does not correlate well
    with the fact that to use this mechanism the server has to store and
    re-deliver all the initial request bandwidth back to the intermediary.

Again, because the server and intermediary are close together, replaying the partial request doesn't prevent the server from shutting down.  The type of server and size of a typical request can also make a difference: we use this primary for dynamic web traffic (as opposed to something like video upload).  These dynamic requests tend not to be huge, and the web server was already holding the entire POST body in memory before dispatching it to the handler.

    * That re-sending is where the bandwidth issue is coming from. The
    initial request uses N bytes to arrive from the client, if M of those
    are a) delivered to each of D servers and b) received back from the
    initial D-1 servers, and c) deliver to the second server. That makes a
    total bandwidth consumption of (N + (D-1)*M).
      Whereas with 307 only consumes (N + M).

It seems you are only counting proxy to server bandwidth.  Assuming the intermediary cannot buffer the body, and a 307 triggers the client to resend the entire request, that will consume 2*N + (N + M) or (3*N + M), compared to N + (N + (D-1)*M) or (2*N + (D-1)*M).

    Also keep in mind that even a blind intermediary just pushing data to a
    single server is handling twice the traffic that server does. That is
    the minimum best-case situation. With multiple servers and/or clients
    the difference increases rapidly to orders of magnitude more resource
    consumption. That is the existing situation, before this feature even
    starts to force more resource consumption.

We find that only a fraction of requests in progress at shutdown time are POST requests, and of those, an even smaller fraction have incomplete bodies say 1 second after shutdown is initiated.  The number of requests that get redirected in this manner from single server in our deployment (which spans hundreds of thousands of servers) is relatively small.  Though, we do find that a tiny number get quite unlucky during a web tier restart and get replayed a few times.

    * All this re-sending of data could delay the server shutdown an
    unreasonable amount of time. Turning what would be a few seconds into
    minutes or even hours. Depending on the intermediary load being
    reasonable is not a good idea.

I agree with you that in a completely generic and blind case, this is a problem.  In our deployment, we find this feature does not impact shutdown time of the webserver, which is 10-15 seconds.

    * Every millisecond of delay added by the re-receive and re-send of data
    makes it more likely the client will terminate early. If that happens
    all this time, bandwidth, memory, and CPU cycles spent are completely
    wasted.

This is precisely the reason I feel this is a valuable feature.  If the client must resend the entire request it's likely to take even longer.

    Consider the case of a system which is undergoing a DoS at the
    public-facing interface of the intermediary. Enacting this feature is a
    huge resource expenditure for an already highly loaded intermediary.

If a system is undergoing DoS, I expect it to be able to jettison less critical functions to save resources.  Such an intermediary could always convert a partial POST replay into a 307 or 500, and reset the upstream request.  I can add this to the security considerations.

    * Section 2.1 says "The server MUST have prior knowledge"

    Yet no mechanism(s) are even hinted at how a server may acquire such
    knowledge. Defining a specific negotiation signal would be far better
    for this and avoid a huge headache with implementations choosing
    different signals and mechanisms for negotiating that knowledge.

I agree.  I will elaborate in the draft on feature negotiation.  We set this up out of band by configuring the proxies and servers to use this feature.  H2 or H3 SETTINGS are appropriate to announce the support for the feature, but H1 would most likely need to add a header to each request indicating support.

    * The Echo- or Pseudo-Echo mechanism is very clunky. I believe it to be
    unlikely that any intermediary implementing this feature is unable to
    simply store the initial request headers for re-use as needed.

That is valuable feedback.  Our implementation stores the initial request headers, but I didn't know if that was a valid assumption, so I added the Echo- mechanism to the draft.  I can remove it entirely and make it a requirement that the request be stored to use the feature.

Thanks

-Alan