Re: Declarative HTTP Spec Test Suite

On Tuesday, May 28th, 2024 at 11:26 AM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:

> --------
> > It's also isn't clear how to make HTTPS requests.
> 
> 
> Right now VTest does not support SSL/TLS natively, but that is clearly
> something we should add.
> 
> There are two major issues, the first being where does the certs come from?
> 
> A) VTest creates self-signed certs as needed on the fly. That would
> probably be intolerably slow.
> 
> B) VTest creates a cache of self-signed certs, either with Y2K38
> like expiry or with logic to periodically refresh them.
> 
> C) The user of VTest have to provide the certs to be used, possibly
> from a helper script which does B)
> 
> The second major issue is: Which SSL/TLS implementation, and here
> there are only two options:
> 
> A) Pick one, and forget about using VTest to probe the edges of
> the SSL/TLS implementations.
> 
> B) Write one, so that VTest can test the edges of SSL and TLS traffic
> 
> I dont know what the situation is with respect with tools to test
> SSL/TLS protocol implementations. If they already have such, B)
> would be surplus to requirements, if not, somebody should write it.

The security team of Go have developed what they call BoGo test suite for their crypt/tls implementation to be run against BoringSSL test suite (https://go-review.googlesource.com/c/go/+/486495). The test suite (https://github.com/golang/go/tree/master/src/crypto/tls/testdata) is a set of plaintext files describing the (TLS) bytes flowing between the client and the server for a particular cipher suite. I don't remember (or don't know) the details of how they get their hands on the bytes recordings, but I learned about this from the GoTime interview with Filippo Valsorda, Roland Shoemaker, and Nicola Murino (Episode: https://changelog.com/gotime/313, transcript: https://github.com/thechangelog/transcripts/blob/master/gotime/go-time-313.md). They use the same test suite both ways, to test Go's implementation as well as BoringSSL testing their implementation and interoperability. Their approach solidified the idea of declarative tests for me.

I got myself side-tracked, but the point I'm driving to is: perhaps starting with BoringSSL is a safe bet, and then using the bytes records in github.com/golang/go repository to build a minimal and compliant SSL/TLS implementation in vtest satisfy point B.


> Where the subject matter is one-way, for instance Nicolas Seriot's
> JSON beastiarium (https://github.com/nst/JSONTestSuite), the success
> of compliance testing is entirely up to the malicious creativity
> of the person(s) writing the tests, and Nicolas did really well,
> breaking (almost?) all JSON parsers in existence.
> 
> But where the subject matter is two way, like communication protocols,
> I have never seen compliance testing deliver anywhere near promise.
> 
> The first problem is that even when everybody agree 100% on what
> the protocol definition says, most protocols explode combinatorically
> in very few exchanges.
> 
> A HTTP server can return 3xx to almost any request, that is not
> a compliance failure, it can send "Connection: Close" or Keepalive,
> that is not a compliance failure either.
> 
> Imagine this:
> 
> compliance-client:
> HEAD /
> server-under-test:
> 301 to /justkidding
> compliance-client:
> HEAD /justkidding
> server-under-test:
> 301 to /
> 
> How many times is a compliant server allowed to yank a client
> around like that, before it actually answers ?
> 
> (The above is surprisingly common patterns of identification)
> 
> The second problem is timing. HTTP doesn't have rigidly specified
> timeouts, but there is still stuff like "If-Modified-Since", which
> the server is allowed to just ignore whenever it feels like it.

Those are good catches. I think they can be resolved by stating requirements and expectations of the behavior of system under test. For instance, it can be a stated that only X redirections allowed, and optional values may have multiple test cases where in one of them the server isn't permitted to ignore.

> But the biggest problem of all, is that people simply do not agree
> what standards say or mean.
> 
> I served time in meetings with angry IBM people, where we could not
> agree what "have received" meant. (Their mainframes crashed when
> my code answered too fast because they (tacitly) assumed modems and
> a telco would always be involved, where I used a 5 meter null-modem
> cable.)

Isn't natural language lovely? Not to make this subject my Moby-Dick, but I honestly believe this is also a great reason to have the standards be translated into declarative test suite for clarity of intent.

> That is not to say that such testing is never valuable, it is, and
> IETF have often arranged "bake-offs", where different implementations
> would meet, and after first getting things to work, they would try
> to break each other's implementations with hostile traffic.
> 
> So as I said: I do not want to discourage you, and as long as you dont
> expect to hand out "HTTP-compliant" gold star stickers, I dont see
> why it cannot or should not be done.
> 

Let me put it in writing for posterity: The test suite is diagnostic and is not a certification for rubber-stamps. That wasn't very discouraging :) Concerns and contrary opinions are helpful in probing ideas and approaches to solidify the scope and implementation. I believe we see eye-to-eye on the value. I'm eager to learn the next steps. If the preference is have the majority of the discussion at the Workshop, that's also fine. I'm signing up shortly. 


All the best,
Mohammed

Received on Saturday, 1 June 2024 01:28:25 UTC