Re: [webappsec] Help build the CSP test suite at Test the Web Forward Portland, August 3

(note: I am not on public-webappsec so please CC me explicitly)

On 04/06/14 22:39, Odin Hørthe Omdal wrote:
> On Wed, Jun 4, 2014, at 22:53, Hill, Brad wrote:
>> Can you point me at how users can see/test changes in real-time?  That
>> was the other issue I had with using the "canonical" repo - with only me
>> contributing, there was nobody who really had it on their TODO list to
>> review and approve my submissions, so merge times when I was working on
>> CORS tests were on the order of 6-8 weeks, and only after I jumped up and
>> down and waved a big flag.  That's just not workable to have multiple
>> people contributing and be so out of sync.
> 
> The review time is indeed a problem. But the outstanding reviews is much
> more visible now than before. And we are reviewing more than before.
> It's still a problem though. More people should review, especially
> people who know the specs.
> 
> For events such as TTWF it is quite crucial to have people reviewing
> tests just as they come in.  It can be the present experts doing it in
> GitHub at the place, or people contributing remotely.  I also think it
> would be very helpful if some of the best people reviewed others' tests
> at the event.

Yes, this should be an important function of experts at events. Indeed
the best way to work would be to have the expert review the test
alongside the author in realtime so they can easily understand how
review works, what is being looked for, and what the issues are with
their test.

> We had an automatic pull request viewing system before, that was synced
> to w3c-test.org, -- that seems to currently be down (due to changes?).
> Having  pr-<number>.w3c-test.org  would be quite cool. I'd like to see
> something like that turn up again. I remember seeing many reviews about
> syncing and server setup, so I hope that's what those reviews were for.

That still happens; submissions go under
w3c-test.org/submissions/<pr-number>. If the submitter is not a
"collaborator" on the w-p-t repo, someone who is needs to add a comment
"w3c-test:mirror" to cause the mirroring to occur.

Regarding the earlier discussion about using web-platform-tests vs a
custom repository, there are a number of reasons to prefer using
web-platform-tests even if some tests require unusual server side setups
that cannot be provided in all environments yet.

As Odin pointed out, wptserve provides the same level of control as PHP
for inspecting the request and sending the response. You have absolute
control over which bytes are sent over the wire and when. There are also
a number of features specifically optimised for writing tests.

Using web-platform-tests makes it possible to reuse almost all the
documentation that exists at testthewebforward.org. People who are
confortable with writing tests for some other spec can dive right in
without having to learn anything new about where the tests are hosted,
how they are written, or whatever.

Self-hosted tests won't actually get used in browser CI systems, since
those are generally forbidden from accessing external hosts for reasons
of reliability. Therefore such an approach is of limited usefulness. On
the other hand, at least at Mozilla, we have made big progress in
running web-platform-tests on CI and, with a little more work on
stability, should be running them on each commit. At present encrypted
connections are not supported simply because it's a difficult problem
and it's a higher priority to get things working at all. But other
Mozilla testsuites such as mochitest do support this, and we should be
able to leverage the techniques they use for
web-platform-tests-on-Mozilla-infrastructure. I imagine other vendors
have similar solutions. So once webappssec tests are in
web-platform-tests it's much more likely that we will do any extra work
needed to get them running on our CI system because it will fit with our
general future plans in that area.

Received on Wednesday, 4 June 2014 22:27:32 UTC