W3C home > Mailing lists > Public > public-webappsec@w3.org > June 2014

Re: [webappsec] Help build the CSP test suite at Test the Web Forward Portland, August 3

From: Odin Hørthe Omdal <odinho@opera.com>
Date: Wed, 04 Jun 2014 22:44:32 +0200
Message-Id: <1401914672.2225.125245825.429B3D2F@webmail.messagingengine.com>
To: "Hill, Brad" <bhill@paypal.com>
Cc: WebAppSec WG <public-webappsec@w3.org>
-CC webappsec-testsuite (public-webappsec should be a superset of the
other, no?)

On Wed, Jun 4, 2014, at 21:41, Hill, Brad wrote:
> As I've mentioned before during the requirements and development phase of
> wptserve, CSP is a bit more complex than most specs in its test
> requirements and I'm not sure the environment has what we need.  
> I'd love it if we could make this work, and maybe I don't understand all
> of what the new framework can do, but here's a few things we need:
> 1) Wildcard, or at least multiple-cname, TLS support.  This is absolutely
> critical for the scenarios where CSP actually matters and will be
> deployed.

As far as I know, this is something that is detached from the actual
framework itself. You can use the same SSL server setup and hosts with
wptserve as with PHP.

In fact, w3c-test.org should probably be set up like that.  It currently

Those tests that rely on that will not be as easy as other
wptserve-based tests though. They would indeed require much more
substantial setup than the common case. However, if w3c-test.org is
being set up correctly, there is currently live-syncing from pull
requests on GitHub over to that server.  That way it'd be possible to
have a live server you could test directly on with very little setup.

jgraham also wanted to make it easy to do HTTPS-tests locally, but that
would as far as I'm aware require some bad certificate hacks.

Anyway, there are two different things here; what can be done with one
single server or AWS instance (that's set up correctly), and what can be
done easily locally and $random_user's computer.

> 2) The ability to not only set headers, but to access the value of the
> headers from within the tests. (e.g. to compare reports vs. the policies
> that were set) In PHP this is simple.  I don't know if one can do this
> with the .headers file approach.

That is simple, you just write that in the test directly.  It's also
easier to understand and more to the point.  For production code we
often Don't Repeat (Y)Ourselves, but for tests it's okay as long as it
is of basic relevancy for the test.  And for included files, there is
also the substitution part of wptserve that can change parts of a file
with substitution variables.

If you wanted to do it in a more advanced way, it's possible to write
full Python scripts that can do anything.  But it's nice to keep simple
things simple.

I'd like trying to port a test that you find of complex enough character
to see if it's doable.  If you can point one out.

> 3) We also need to test that reports are sent correctly.  Right now, We
> have a support PHP script that actually writes to temp files on disk and
> then allows the test case to call back and see what the results were.
> ...

CORS tests do this a lot for preflight.  In fact, I was very happy about
how wpt handles this.  It's much better than what we have for PHP.  It's
called Stash, and even has its own help page[0] :D

  0. http://wptserve.readthedocs.org/en/latest/stash.html

> This is why I'm running an AWS server at my own cost so I can provide a
> turn-key environment for people who want to contribute tests, with all
> the necessary setup done, including a valid wildcard TLS certificate. 
> If there's a better way that we can get a simpler setup working before
> TTWF, I'd love to hear it, but I'm not sure there is.

I think your "give ssh accounts" is a very good method.  But there is no
reason why that AWS server can't run wptserve.  Getting  w3c-test.org 
to do all we want is better though, so that you won't have to maintain
that server on your own.

An added benefit of getting these tests in the upstream is that
web-platform-tests is the canonical place for all tests of type web
platform, and thus run by the browser vendors* :)

 [*] Currently not everyone is really running everything as well as they
     But as far as I know they're all working on it.

  Odin Hørthe Omdal
Received on Wednesday, 4 June 2014 20:45:04 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 18:54:39 UTC