Re: [svgwg] Should the SVG 2 test suite use Shepherd?

To start with, a major point: the CSS WG has two (almost) entirely 
distinct systems, the [Test Harness](http://test.csswg.org/harness/), 
which provides a UI to run tests and stores results, and 
[Shepherd](http://test.csswg.org/shepherd/), the Test Suite Manager, 
which is essentially a glorified issue tracker for the testsuite (it 
has some other functions, but they're not really relevant here, as 
they mostly relate to storing data to compute cross-spec 
cross-references). Your above points all pertain to the Test Harness, 
rather than Shepherd.

Now:

The CSS WG has essentially decided to move away from Shepherd for 
issue tracking in favour of GitHub, so the SVG WG would become the 
only users, and I think everyone involved in web-platform-tests would 
prefer to have a single issue tracker for the repo! So I think the 
answer wrt Shepherd is a clear no, don't use it.

The Test Harness, on the other hand, is a thorn in my back. :) In 
short, it is currently reliant on the csswg-test build system, which 
the CSS WG want to get rid of, but can't because the CSS WG's tools 
rely on it. I suspect we're going to end up with a from-scratch 
rewrite, which I expect will end up being my major 4Q project. I think
 we're going to end up discussing what exactly it should be on 
Thursday/Friday at TPAC, so I'd suggest you come along and thrown your
 2ยข into the bucket.

With regards to your question above: the Test Harness has, in theory, 
support for (whitelisted) users to bulk-upload results. In practice, 
it relies on users running all the tests in it. One thing we 
absolutely want a new system to do is be able to take results 
submitted from browser's CI systems, which should be running the vast 
majority of the tests (this obviously isn't workable currently for 
csswg-test given only MS are running any large part of it!).

-- 
GitHub Notification of comment by gsnedders
Please view or discuss this issue at 
https://github.com/w3c/svgwg/issues/270#issuecomment-247363693 using 
your GitHub account

Received on Thursday, 15 September 2016 15:35:01 UTC