Re: Who currently executes the tests in the w3c repos?

On Wed, Aug 7, 2013 at 1:44 PM, Peter Linss <peter.linss@hp.com> wrote:

>
> On Aug 7, 2013, at 12:38 PM, Dirk Pranke wrote:
>
>  {snip}
>
>
> Thanks for the responses, Peter!
>
> The last I looked (last week), there were thousands of tests still in the
> "submitted" (not "approved") directories in the csswg repo; how does the
> use of those two directory names map on to what you wrote above about tests
> moving from unreviewed to reviewed?
>
>
> The 'submitted' and 'approved' directories in that repo are a hold-over
> from before we had a review tool (Shepherd). Shepherd still uses those
> directories as cues to shift an asset's status (e.g. if someone with
> approval rights moves a file into 'approved', then it marks the test as
> approved).
>
>
> I was perhaps naively assuming that "approved" meant "reviewed and
> approved" and "submitted" meant "not yet reviewed and approved". Is that
> accurate, or is the source of truth in Shepherd or somewhere else?
>
>
> The current state of any given asset is what's found in Shepherd. It's
> possible for tests to be approved in either directory, also tests in the
> approved directory can lose their approved status (if they're edited by
> someone who doesn't have approval rights, or if its status changes in
> Shepherd due to an issue being filed). In general, we no longer move tests
> when they are approved. But sometimes we'll move an entire directory's
> contents into the 'approved' directory to do a quick bulk approval.
>
>
So it sounds like whether a test is in an "approved" directory or a
"submitted" directory is a rough guideline at best and probably should be
ignored. Does Shepherd expose APIs that would allow me to query the status
of a particular test path in the repo? Also, do you have plans to
eventually remove the approved/submitted directories? It seems like there
is value in having tests be in the main branch of the main repo even if
they are not actually approved, so that people can run tests prior to their
approval, right?

(It seems like this would be valuable for web-platform-tests as well,
right?)


> It also looks like (according to your documentation) that there is an
> "accepted" state perhaps not reflected in the repo?
>
>
> Right, 'accepted' means it's been reviewed by someone who is not a trusted
> approver. This could come from a review in Shepherd or the addition of a
> 'reviewer' meta tag inside the file. This status was created after we had
> Shepherd and there is no corresponding directory in the repo.
>
>
> Also, am I correct in assuming that we should completely ignore anything
> under the "incoming" directories as files in those directories? Why do they
> exist in the shared repo at all?
>
>
> Correct. These are also a carry over from the good old days when we were
> using SVN (our repository dates back to 2008). They exist to allow people
> to commit and potentially share tests that are not ready for review or
> inclusion in a test suite. Today we'd encourage new users to clone the repo
> on GitHub and commit their work to their own clone, but there are still
> users of our repo that prefer using the incoming directories.
>
> Peter
>

Got it, thanks!

-- Dirk