Re: More on allowed field characters

Martin, Roy,

How about the latest text is augmented to say something like:

The HPACK is able to transport characters that are not valid HTTP and as
such HTTP/2 implementations may be used for
non-HTTP semantic transport.  HTTP/2 implementations that are used to
represent HTTP semantics MUST validate field
names and values according to their definitions in Sections <xref
target="HTTP" section="5.1" format="counter"/>


I.e make it really clear that when HTTP/2 impl is being used with the core
HTTP spec then it MUST comply with that spec.  If it is transporting
non-HTTP compliant headers, then it make it clear it is not representing
HTTP semantics.

regards


On Fri, 27 Aug 2021 at 11:24, Martin Thomson <mt@lowentropy.net> wrote:

> How is this not a core HTTP problem?
>
> Part of the intent of the change is to leave the validation to core
> specs.  If you are using h2 fields as HTTP fields, the rules for HTTP
> already cover that use.
>
> There are two extra things we do in h2:
>
> 1. deny uppercase field names
> 2. mandate extra checks from an abundance of caution
>
> (The latter might be mere paranoia, but it's demonstrably necessary.)
>
> On Fri, Aug 27, 2021, at 04:13, Roy T. Fielding wrote:
> > > On Aug 25, 2021, at 10:51 PM, Martin Thomson <mt@lowentropy.net>
> wrote:
> > >
> > > On Mon, Aug 23, 2021, at 15:03, Martin Thomson wrote:
> > >> It seems like the allowed characters in fields is a gift that keeps
> on giving.
> > >
> > > Thanks everyone for all the words you gave.
> > >
> > > Based on feedback from Willy and Greg in particular, I've taken
> another go at this:
> > >
> > >  https://github.com/httpwg/http2-spec/pull/936/files
> > >
> > > It says that:
> > >
> > > * fields SHOULD be validated properly (according to HTTP §5.1 and §5.5)
> > >
> > > * failure to validate fields might enable attacks, especially if the
> message ends up in HTTP/1.1 somehow (that is, providing motivation that was
> lacking from previous iterations on this)
> > >
> > > * if fields aren't fully validated, attacks might happen, so minimal
> validation MUST be performed (with the checks previously agreed)
> > >
> > > This does not address Roy's original point directly.  Yes, code that
> makes assumptions without taking responsibility for checking them might be
> exposed to the full consequences of poor decisions.  However, I believe
> that a lot of implementations will abide by the SHOULD here.  This is about
> levying requirements on implementations that might have expected to avoid
> having to validate fields; because we've learned that copying and pasting
> without checking happens.
> > >
> > > (I do worry that this is an overreaction.  The original text in the
> spec was arguably fine.  It was just being ignored.)
> >
> > I think my issue is still being misunderstood.
> >
> > I don't think we need every implementation to do field validation on
> > every receipt
> > or forwarding of HPACK fields. I think we need to be clear on what the
> > requirements
> > for field validation are when extracting HPACK-encoded strings and
> > using them
> > in an HTTP context, whether that context be for HTTP fields in other
> > versions,
> > CGI environment variables, Servlet tables, or an internal data
> > structure for request
> > processing. The point is that the implementation translating HPACK to
> > an abstract
> > HTTP message (regardless of version) MUST ensure that the result fits
> > within
> > all of HTTP's field requirements when that field is used as HTTP, since
> > the
> > implementation cannot trust the HPACK encoder.
> >
> > Specifically, coming up with an arbitrary set of different requirements
> for that
> > process is NOT GOOD. I understand one desire to do that is because h2 has
> > the additional restriction of all-lowercase names, but the right way to
> say that
> > is as an additional requirement for h2, not by trying (and failing) to
> subsume
> > the existing requirements of HTTP.
> >
> > I think it is fine if this is limited specifically to those recipients
> > actually doing
> > HTTP things with HTTP fields, as opposed to merely forwarding arbitrary
> > fields
> > at scale, but it cannot be limited just to HTTP/1. This is an
> > application issue for
> > anything that consumes an HTTP message. It is still necessary even if an
> > h2-only server receives a request and directly handles it via an
> > internal
> > application library. The vulnerability may be inside the application
> > library,
> > but the h2 server is expected to prevent it because that's where the
> > rubber
> > meets the road.
> >
> > Likewise, an HPACK decoder MUST produce valid HTTP field values if they
> > are going to be used as HTTP field values.
> >
> > I am fine with HPACK also being used to convey UTF-8 named fields and/or
> > carrying binary field values, but only when that is clearly indicated
> via the
> > protocol and processed as such.
> >
> > I want the h2 spec to say something like:
> >
> >    When an implementation extracts a field name string from HPACK and
> >    intends to use that string as an HTTP field name (semantically), the
> >    implementation MUST validate the strict intersection of the existing
> >    MUST requirements on syntax (ref. field-name ABNF [SEMANTICS]
> >    with uppercase excluded) when extracted from HPACK and used
> >    with the semantics of an HTTP field name. This is particularly
> important
> >    when the received header fields are transformed as a whole from one
> >    form (i.e., HPACK) to another (e.g., a translated header section, an
> >    internal hash table, or a set of environment variables).
> >
> > IOW, don't make up new requirements based on perceived syntax issues.
> > Refer to the specific syntax required by HTTP and add the h2-specific
> > limitations, since any variation might result in unforeseen differences
> > in downstream handling of that message.
> >
> > ....Roy
> >
> >
>
>

-- 
Greg Wilkins <gregw@webtide.com> CTO http://webtide.com