Re: Who currently executes the tests in the w3c repos?

On Aug 7, 2013, at 1:49 PM, Dirk Pranke wrote:

> 
> 
> 
> On Wed, Aug 7, 2013 at 1:44 PM, Peter Linss <peter.linss@hp.com> wrote:
> 
> On Aug 7, 2013, at 12:38 PM, Dirk Pranke wrote:
> 
>> {snip}
>> 
>> Thanks for the responses, Peter!
>> 
>> The last I looked (last week), there were thousands of tests still in the "submitted" (not "approved") directories in the csswg repo; how does the use of those two directory names map on to what you wrote above about tests moving from unreviewed to reviewed?
> 
> The 'submitted' and 'approved' directories in that repo are a hold-over from before we had a review tool (Shepherd). Shepherd still uses those directories as cues to shift an asset's status (e.g. if someone with approval rights moves a file into 'approved', then it marks the test as approved).
> 
>> 
>> I was perhaps naively assuming that "approved" meant "reviewed and approved" and "submitted" meant "not yet reviewed and approved". Is that accurate, or is the source of truth in Shepherd or somewhere else?
> 
> The current state of any given asset is what's found in Shepherd. It's possible for tests to be approved in either directory, also tests in the approved directory can lose their approved status (if they're edited by someone who doesn't have approval rights, or if its status changes in Shepherd due to an issue being filed). In general, we no longer move tests when they are approved. But sometimes we'll move an entire directory's contents into the 'approved' directory to do a quick bulk approval.
> 
> 
> So it sounds like whether a test is in an "approved" directory or a "submitted" directory is a rough guideline at best and probably should be ignored. Does Shepherd expose APIs that would allow me to query the status of a particular test path in the repo?

Not yet, though it is planned as part of the build code re-work I'm doing. If you have a short term need I can add APIs quickly.

> Also, do you have plans to eventually remove the approved/submitted directories?

Yes.

> It seems like there is value in having tests be in the main branch of the main repo even if they are not actually approved, so that people can run tests prior to their approval, right?

Right.

> 
> (It seems like this would be valuable for web-platform-tests as well, right?)

I believe so, yes.

>  
>> It also looks like (according to your documentation) that there is an "accepted" state perhaps not reflected in the repo?
> 
> Right, 'accepted' means it's been reviewed by someone who is not a trusted approver. This could come from a review in Shepherd or the addition of a 'reviewer' meta tag inside the file. This status was created after we had Shepherd and there is no corresponding directory in the repo.
> 
>> 
>> Also, am I correct in assuming that we should completely ignore anything under the "incoming" directories as files in those directories? Why do they exist in the shared repo at all?
> 
> Correct. These are also a carry over from the good old days when we were using SVN (our repository dates back to 2008). They exist to allow people to commit and potentially share tests that are not ready for review or inclusion in a test suite. Today we'd encourage new users to clone the repo on GitHub and commit their work to their own clone, but there are still users of our repo that prefer using the incoming directories. 
> 
> Peter
> 
> Got it, thanks!

No problem.