Re: Adding support for test requests in Shepherd

From: Mihai Balan <mibalan@adobe.com<mailto:mibalan@adobe.com>>
Date: Friday, September 13, 2013 7:03 AM
To: "Public CSS Test suite mailing list (public-css-testsuite@w3.org<mailto:public-css-testsuite@w3.org>)" <public-css-testsuite@w3.org<mailto:public-css-testsuite@w3.org>>
Subject: Adding support for test requests in Shepherd
Resent-From: <public-css-testsuite@w3.org<mailto:public-css-testsuite@w3.org>>
Resent-Date: Friday, September 13, 2013 7:04 AM

Hello everybody,
Recently I have participated in discussions, in different contexts, about the usefulness of recording test suggestions for W3C tests when implementing them on the spot is just not feasible because of resource/priority constraints.
This becomes even more interesting/useful in the context of events like Test the Web Forward, where simply pointing to a specification might prove discouraging for a person looking to start writing a test, but instead pointing to a list of proposed/requested test cases can be a lot more approachable.
Yes, this is a recurring issue at TestTWF – one of our biggest challenges in fact.  However, the challenge hasn't been in how we present the info to attendees. As long as it's in one place, that part isn't problematic (more on this below). The hard part has been collecting the info because it entails understanding the spec, knowing what tests exist already, and identifying risk areas based knowledge of some implementation(s) and on known interoperability issues out in the wild.  While a coverage report is certainly helpful here, the human assessment & risk analysis is really the toughest part of testing.


I initially had some off-list exchanges with Peter about a gitHub-centric process that used issues on the csswg-test repo[1] to do that, but it proved unfeasible for a number of reasons (most important being access control).
AFAIK, anyone can create issues on Github if the Issue Tracker is on.  Labels, milestones and assigning issues is reserved for repos Collaborators. If access control to the csswg-test repo needs to be tightly controlled for some reason, that still doesn't rule out github because you can reference/close issues across repos very easily – e.g. "Closes Issue #42 in some-user/some-repo-or-forked-repo"

The proposed solution would be adding a new feature in Shepherd to create/track/edit test request that would sync with the gitHub repository eventually (but without the access control mess).

I would not be in favor of  a Shepherd-only because of TestTWF, where it could cause more confusion to newcomers coming to write tests for specs across the entire W3C.  Part of our pitch to them is how easy and cool it is to contribute new tests.  Looking for this information in different places depending on which spec (working group) you're interested in is not easy, and that's not cool.  It's been identified by many (and a lot through our experiences with hundreds of people at several TestTWF events) that testing info and processes are too fractured, thus raising the barrier to entry. We're making a concerted effort to fix that by centralizing everything on testthewebforward.org, which will be relaunched soon [1].  In fact, in preparation for the upcoming Shenzhen event, I've flagged this very thing – of course, with Github's Issue Tracker :) [2].  We can certainly do a short term experiment with Github for the Shenzhen event and see how it goes in practice.


I put together a rough draft of the proposed workflows and needed features over at [2], and I’m interested in your opinion about it:

·        Does the whole idea of test requests/test suggestions make sense?

·        Do the proposed workflows/features make sense and/or are enough?

[1] https://github.com/w3c/testtwf-website
[2] https://github.com/w3c/testtwf-website/issues/113