Re: [webappsec] Help build the CSP test suite at Test the Web Forward Portland, August 3

OK.

I'll give it a shot.  I've been too busy with other things to keep up with wptserve recently, but I need to invest some time in the suite before TTWF anyway, so I'll see what I can do.

Can you point me at how users can see/test changes in real-time?  That was the other issue I had with using the "canonical" repo - with only me contributing, there was nobody who really had it on their TODO list to review and approve my submissions, so merge times when I was working on CORS tests were on the order of 6-8 weeks, and only after I jumped up and down and waved a big flag.  That's just not workable to have multiple people contributing and be so out of sync.

-Brad

On Jun 4, 2014, at 1:44 PM, Odin Hørthe Omdal <odinho@opera.com> wrote:

> -CC webappsec-testsuite (public-webappsec should be a superset of the
> other, no?)
> 
> On Wed, Jun 4, 2014, at 21:41, Hill, Brad wrote:
>> As I've mentioned before during the requirements and development phase of
>> wptserve, CSP is a bit more complex than most specs in its test
>> requirements and I'm not sure the environment has what we need.  
>> 
>> I'd love it if we could make this work, and maybe I don't understand all
>> of what the new framework can do, but here's a few things we need:
>> 
>> 1) Wildcard, or at least multiple-cname, TLS support.  This is absolutely
>> critical for the scenarios where CSP actually matters and will be
>> deployed.
> 
> As far as I know, this is something that is detached from the actual
> framework itself. You can use the same SSL server setup and hosts with
> wptserve as with PHP.
> 
> In fact, w3c-test.org should probably be set up like that.  It currently
> isn't.
> 
> Those tests that rely on that will not be as easy as other
> wptserve-based tests though. They would indeed require much more
> substantial setup than the common case. However, if w3c-test.org is
> being set up correctly, there is currently live-syncing from pull
> requests on GitHub over to that server.  That way it'd be possible to
> have a live server you could test directly on with very little setup.
> 
> 
> jgraham also wanted to make it easy to do HTTPS-tests locally, but that
> would as far as I'm aware require some bad certificate hacks.
> 
> Anyway, there are two different things here; what can be done with one
> single server or AWS instance (that's set up correctly), and what can be
> done easily locally and $random_user's computer.
> 
>> 2) The ability to not only set headers, but to access the value of the
>> headers from within the tests. (e.g. to compare reports vs. the policies
>> that were set) In PHP this is simple.  I don't know if one can do this
>> with the .headers file approach.
> 
> That is simple, you just write that in the test directly.  It's also
> easier to understand and more to the point.  For production code we
> often Don't Repeat (Y)Ourselves, but for tests it's okay as long as it
> is of basic relevancy for the test.  And for included files, there is
> also the substitution part of wptserve that can change parts of a file
> with substitution variables.
> 
> If you wanted to do it in a more advanced way, it's possible to write
> full Python scripts that can do anything.  But it's nice to keep simple
> things simple.
> 
> 
> I'd like trying to port a test that you find of complex enough character
> to see if it's doable.  If you can point one out.
> 
>> 3) We also need to test that reports are sent correctly.  Right now, We
>> have a support PHP script that actually writes to temp files on disk and
>> then allows the test case to call back and see what the results were.
>> ...
> 
> CORS tests do this a lot for preflight.  In fact, I was very happy about
> how wpt handles this.  It's much better than what we have for PHP.  It's
> called Stash, and even has its own help page[0] :D
> 
>  0. http://wptserve.readthedocs.org/en/latest/stash.html
> 
> 
>> This is why I'm running an AWS server at my own cost so I can provide a
>> turn-key environment for people who want to contribute tests, with all
>> the necessary setup done, including a valid wildcard TLS certificate. 
>> 
>> If there's a better way that we can get a simpler setup working before
>> TTWF, I'd love to hear it, but I'm not sure there is.
> 
> I think your "give ssh accounts" is a very good method.  But there is no
> reason why that AWS server can't run wptserve.  Getting  w3c-test.org 
> to do all we want is better though, so that you won't have to maintain
> that server on your own.
> 
> An added benefit of getting these tests in the upstream is that
> web-platform-tests is the canonical place for all tests of type web
> platform, and thus run by the browser vendors* :)
> 
> [*] Currently not everyone is really running everything as well as they
> should.
>     But as far as I know they're all working on it.
> 
> 
> -- 
>  Odin Hørthe Omdal
>  odinho@opera.com

Received on Wednesday, 4 June 2014 20:54:02 UTC