Re: Knowing which tests are in the repository

On Thu, Aug 15, 2013 at 3:30 PM, Tobie Langel <tobie@w3.org> wrote:

>
>
> On Thursday, August 15, 2013 at 10:26 PM, James Graham wrote:
>
> > On 15/08/13 20:42, Tobie Langel wrote:
> > > On Thursday, August 15, 2013 at 5:27 PM, James Graham wrote:
> > > > and unusually long timeouts for testharness tests.
> > >
> > > Doesn't testharness already let you handle timeouts?
> >
> > Sure. But the general problem is this:
> >
> > I am writing a test runner for these tests. In order for this to work I
> > need to know at least which files are tests. But for a good
> > implementation I also need various other pieces of data about the tests.
> > For example I need to know the timeouts because I want the runner to
> > have it's own timeout in case the test causes the browser to hang or
> > crash. This obviously needs to be at least as long as the test timeout.
> > Therefore the timeout in the test is relevant, but the fact that
> > testharness.js will timeout itself isn't really relevant.
>
> Can't you parse the test to get that data out?
>
> Else couldn't we set a convention to add this info in a meta tag?
>

I don't yet have enough experience running a wide variety of the tests in
the repo, but what range of test speeds is considered acceptable? Is there
some reason you really need to be able to customize test harness timeouts
per-test?

(Note that I am quite familiar with this desire; we support it in Blink and
in WebKit, but historically WebKit tended to just have a single long global
timeout rather than one for fast and one for slow tests).


> > Now, it is true that for some of the information I could parse it out of
> > the files. But if I do that I am certainly going to store it in some
> > intermediate format so that I don't have to redo the expensive parsing
> > step for each test run.
>
> Is it really going to be that expensive?
>

Parsing each test to extract fields, assuming you're doing this in a
separate step from actually running the test, is indeed expensive. For
example, if your test harness is parsing the test in python, that can
easily be slower than actually running the test in the browser, meaning
that we could end up doubling test execution time. At least, this is what
I'm seeing in my efforts to get Blink running the tests.

It seems likely that I can optimize away much of the parsing time, but
probably not all of it.

Really, a bigger question is if you are assuming a build step or not prior
to running the tests. If you are, then you'll definitely want to extract
the fields. If you aren't, then adding a build step just for parsing out
the metadata is painful (and is something I'd currently really like to
avoid but I don't know how successful I can really be).

> So at some level I am going to end up with a
> > manifest format. Since everyone else trying to run the tests in bulk
> > will also need this it makes sense to share the format and the files.
>
> Possibly.
> > Moreover, the filename-only solution doesn't work. You don't load files,
> > you load URLs. A particular file can have different behavior depending
> > on non-path parts of the URL. For example the parser tests work
> > differently depending on the query string that you pass in.
>
> Pointers to these tests? This seems a rather isolated case for which there
> might be a dedicated solution that would work as well.
>

I would also like to see this. Historically in Blink we can accomplish such
tests using iframes to resources or other script-based mechanisms; we've
never needed a way to specify query strings as part of a test input.

In general, I think it's much easier to understand the layout of the
repository if we follow naming conventions, rather than requiring lookups
through a manifest. To that end, Blink and WebKit have always preferred the
-expected or -ref convention for references over the "rel=match" metadata
approach, but I think that ship has sailed anyway.

So, I'd strongly encourage naming conventions even if we can't get 100%
accuracy that way and do end up needing a manifest for some things.

-- Dirk