Re: Linking test suites to specs

On Jun 25, 2013, at 12:14 AM, Tobie Langel wrote:

> On Tuesday, June 25, 2013 at 5:42 AM, L. David Baron wrote:
>> On Monday 2013-06-24 16:30 -0700, Rebecca Hauck wrote:
>>> 1. Add the Test Suite section to all of the specs – those with no tests labeled "none yet"
>>> 2. Link to the test suite cover page per fantasai's suggestion
>> 
>> There should be instructions somewhere on how to cause such a test
>> suite cover page to exist for a new spec. Part of the reason I
>> wrote "none yet" was because I couldn't (I think) do it myself, and
>> it didn't seem worth bothering somebody else about when there
>> weren't any tests yet.
>> 
>> I think the cover page should also link to existing results for the
>> tests (which is why I had two links in one spec, I suspect), though
>> we should also start gathering such existing results through
>> automation rather than encouraging manual running of the tests.
> 
> I'm working on a solution to automate running test suites (including reftests) and gathering results using WebDriver. We presented this project to Saucelabs which offered to provide free, unlimited access to their infrastructure (over 150 browser/OS combinations) on the condition that we'd keep the output results publicly available. I don't have an ETA for this yet, but it's high up in my priority list. Of course resources (financial or human)  to help build this would be welcomed.

I may be able to spend some cycles on this once I'm done with the build code and Shepherd work I have on my plate.
> 
> As a side note, are there plans to integrate the CSS test suites in the main web-platform-tests repo? Anything I can do to help with this transition?

Yes, the plan is to augment Shepherd to integrate with GitHub's API so we can sync the review data bidirectionally between Shepherd and GitHub. This way we won't lose our existing review data and Shepherd will also serve as a backing store to GutHub. Once that's in place, I can migrate our tests into the main repo (and optionally provide Shepherd's features to other areas of the main repo, like automatic validation of metadata, metadata searches, etc).

Peter

Received on Tuesday, 25 June 2013 17:43:23 UTC