Re: Who currently executes the tests in the w3c repos?

On Wed, Aug 7, 2013 at 9:34 AM, Peter Linss <peter.linss@hp.com> wrote:

>
> On Aug 7, 2013, at 2:22 AM, Ms2ger wrote:
>
> > On 08/07/2013 02:43 AM, Dirk Pranke wrote:
> >> Hi all,
> >>
> >> I am a relative newcomer to this group but I have been working off and
> on
> >> recently (quite a bit just now) getting the tests running as part of the
> >> automated tests for Blink and WebKit.
> >>
> >> I believe I'm probably missing quite a bit of context or history that
> makes
> >> it difficult for me to understand some of the design decisions and
> >> processes around getting tests written and submitted and run.
> >>
> >> So forgive me if this sounds like a brash question, but it's honestly
> one
> >> coming from ignorance and not meant to be snarky ...:
> >>
> >> Who actually currently runs these tests, and how?
> >>
> >> As far as I know, no one in Blink (or WebKit) regularly runs any of
> these
> >> tests, even manually, with a few exceptions where we have manually
> imported
> >> some suites into our existing repos. It may also be the case that some
> >> times individual developers or spec editors have run some of the tests.
> >>> From my limited conversations w/ Fantasai, I believe the situation is
> >> similar for Mozilla. I do not know about efforts inside Microsoft or at
> >> Opera, or at any other browser vendor or third party.
> >>
> >> Are there groups that actually do attempt to run the tests somehow on
> the
> >> different browsers? Does that somehow happen in Shepherd in a way I
> don't
> >> know about (or understand)?
> >>
> >> I would like to be able to usefully contribute to threads like
> >> "consolidating css-wg and web-platform-tests repositories" and talk
> about
> >> the pain points I'm hitting as I try to get the tests running, but it's
> >> hard for me to say useful things w/o knowing more about how others are
> >> using all of this. So, I'm looking to become educated?
> >
> > Mozilla imports a subset of the web-platform-tests repository into its
> main repository (based on the MANIFEST files you might have noticed). The
> importing is completely scripted, including writing the annotations for
> failing tests. The code for that is available [1]. Those tests are run both
> in automation and by developers locally from there.
> >
> > I understood Ryosuke Niwa was working on importing tests in WebKit
> before the fork; have you talked to him?
> >
> > There are various tools to run tests semi-automatically [2-4].
> >
> > The Shepherd tool [5] is a Test Suite Manager used by the CSS WG for
> reviewing; there are some review comments there, but in my personal
> experience those are largely ignored. This may be because there is no value
> in getting tests reviewed; the CSS WG only cares about tests to get
> specifications to CR, and for that purpose, unreviewed (and probably
> incorrect) tests are used.
>
> This statement is untrue and misleading. Firstly, our primary focus of
> testing is getting specs _out_ of CR and into REC, getting to CR does not
> require tests, but this is not the "only" thing we care about regarding
> tests. We do care about the health of the platform and want to promote
> proper, interoperable implementations as well. However, we currently have
> approximately 60 specifications on the REC track and we do respect the REC
> track process. A lack of testing is one of our primary blockers getting
> specs to REC so this is a major focus for us.
>
> Secondly, this implies that we're gaming the system and don't care about
> the quality of the tests we use, which is completely false. We used to have
> a policy of not including tests in our suites until they were reviewed. We
> found that our review backlog was too long (by thousands of tests) and
> changed our review policy to allow unreviewed tests into the suites and
> allow the reviews to be triggered by looking for unexpected results. Yes,
> there are most likely a small percentage of tests that are incorrect, but
> the review backlog is still being worked even after the relevant spec has
> exited CR and the test suite shifted into the conformance stage.
>

Thanks for the responses, Peter!

The last I looked (last week), there were thousands of tests still in the
"submitted" (not "approved") directories in the csswg repo; how does the
use of those two directory names map on to what you wrote above about tests
moving from unreviewed to reviewed?

I was perhaps naively assuming that "approved" meant "reviewed and
approved" and "submitted" meant "not yet reviewed and approved". Is that
accurate, or is the source of truth in Shepherd or somewhere else? It also
looks like (according to your documentation) that there is an "accepted"
state perhaps not reflected in the repo?

Also, am I correct in assuming that we should completely ignore anything
under the "incoming" directories as files in those directories? Why do they
exist in the shared repo at all?

-- Dirk