H2 vs HPACK header table size... again

Hi all!

It was already mentioned long ago but never really settled, there is an
ambiguity in the HPACK and H2 specs regarding the header table size a
decoder assumes the encoder will use.

Today I got an haproxy user report an issue with the nghttp client when
forcing the header table to zero, because nghttp's decoder expects a table
size update from the encoder while the encoder considers it only conforms
to what was advertised and never changes it, thus never updates it.

For those who want a bit of background, the report was made here:
   https://github.com/haproxy/haproxy/issues/1498
   https://github.com/nghttp2/nghttp2/issues/1660

I discussed a bit with Tatsuhiro about this, who agreed that it was best
to bring the issue here. We're not trying to decide who's right or wrong,
and we're both fine with updating our implementations to improve
interoperability but it's annoying to bindly have to do something we
don't seem to be reading in the spec.

The H2 spec says about the SETTINGS frame:

   SETTINGS_HEADER_TABLE_SIZE (0x1):  Allows the sender to inform the
      remote endpoint of the maximum size of the header compression
      table used to decode header blocks, in octets.  The encoder can
      select any size equal to or less than this value by using
      signaling specific to the header compression format inside a
      header block (see [COMPRESSION]).  The initial value is 4,096
      octets.

Note the word "initial" here which may or may not participate to the
confusion. My reading is that until SETTINGS_HEADER_TABLE_SIZE the
value is 4096, and that SETTINGS_HEADER_TABLE_SIZE changes that value
to the newly advertised one.

The HPACK spec says in RFC7541#4.2:

   Protocols that use HPACK determine the maximum size that the encoder
   is permitted to use for the dynamic table.  In HTTP/2, this value is
   determined by the SETTINGS_HEADER_TABLE_SIZE setting (see
   Section 6.5.2 of [HTTP2]).

Thus for me it means that the value learned from the H2 layer has to
be used at the HPACK layer.

Then it continues:
   An encoder can choose to use less capacity than this maximum size
   (see Section 6.3), but the chosen size MUST stay lower than or equal
   to the maximum set by the protocol.

   A change in the maximum size of the dynamic table is signaled via a
   dynamic table size update (see Section 6.3).  This dynamic table size
   update MUST occur at the beginning of the first header block
   following the change to the dynamic table size.  In HTTP/2, this
   follows a settings acknowledgment (see Section 6.5.3 of [HTTP2]).

Thus my understanding is that an encoder that chooses to use a different
value than the one learned from the H2 layer has to notify it via an
update.

And I think this is where there are two different readings which possibly
depend on how the implementation is built:

  - if you write some client code, it seems natural that HPACK is
    instantiated before the H2 layer, and in this case I wouldn't be
    surprised that the "initial" term used above is interpreted as
    "first set your HPACK default value to 4096 and announce it if
    you learn otherwise, including from the H2 SETTINGS frame",
    which seems to be how Tatsuhiro and Pat have interpreted it in
    the thread below (do not hesitate to correct me, I'm not trying
    to put words in your mouth at all):

      https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0586.html

  - if you're writing some server code, it seems natural that H2 is
    instanciated before HPACK and that HPACK comes with the negotiated
    settings. After all we could even imagine a SETTINGS parameter to
    switch to a totally different encoder. In this case HPACK starts
    with an initial value that matches the one advertised by the client,
    which seems to be what Cory, Hervé and I have interpreted in the
    thread below:

      https://lists.w3.org/Archives/Public/ietf-http-wg/2015OctDec/0107.html

I do not care about systematically emitting one extra byte to update
a table size that we already know if that improves interoperability,
but I find it pretty annoying that 6 years later we continue to face
interoperability issues because there seem to be two incompatible
readings of this corner case of the spec. And I suspect that the issue
also exists for situations where a large SETTINGS_HEADER_TABLE_SIZE
value is emitted by the decoder, the encoder is not willing to change
from 4kB and stays at 4kB, but the decoder assumes that the encoder
agreed with the initially advertised one. This can be worse because
it could work without triggering compression errors but possibly
result in damaged messages.

As such I'd like that we could collectively decide what to do for
this (even if that means relaxing both clients and servers to
accommodate for the other interpretations that are already deployed).

And maybe once we're fixed with whatever possible interpretation, we
should possibly file an errata to amend this part of the HPACK spec
in a way that doesn't leave any more room for two interpretations.

What do implementers (or others) think ?

Thanks,
Willy

Received on Wednesday, 22 December 2021 15:11:59 UTC