- From: Dirk Pranke <dpranke@chromium.org>
- Date: Fri, 23 Aug 2013 10:48:10 -0700
- To: James Graham <james@hoppipolla.co.uk>
- Cc: "public-test-infra@w3.org" <public-test-infra@w3.org>
- Message-ID: <CAEoffTBp8a9p8j8bTGaYze5g0H9x4XX_kJPgr-WtvCHjjThjvA@mail.gmail.com>
On Fri, Aug 23, 2013 at 3:58 AM, James Graham <james@hoppipolla.co.uk>wrote: > On 22/08/13 23:10, Tobie Langel wrote: > >> On Thursday, August 22, 2013 at 11:41 PM, Dirk Pranke wrote: >> >>> On Thu, Aug 22, 2013 at 2:27 PM, Kevin Kershaw >>> <K.Kershaw@cablelabs.com (mailto:K.Kershaw@cablelabs.**com<K.Kershaw@cablelabs.com>)> >>> wrote: >>> >> > >>> However, I think you raise a valid question: if we did support >>> manifests, would we require that all tests be in a manifest? Here >>> there's two possible interpretations: you can't run a test at all >>> if it's not in the manifest, or you can run the test interactively, >>> but a test harness/runner might ignore it (and thus we might want a >>> commit check to enforce this). Or, of course, being in the manifest >>> could be strictly optional. >>> >> >> We want to avoid test never getting run because they weren't added to >> the manifest (or they were, but the url has a typo). >> > > I feel like I didn't explain my previous proposal very well. > > Automation harnesses need a manifest or some other external way of storing > which tests exist and what settings they have. That is, I think, > uncontroversial. It is simply not viable to parse that information out of > the test files when they are actually being run. > > It's at least a little controversial :). The way Blink and WebKit's test harness works (for their own checked-in tests, not the W3C's), we walk a directory looking for files that have specific file extensions and aren't in directories with particular names. All (and only) the matches are tests; end of story. References can be found either by filename convention or (in a few rare cases not really used today) by parsing a reftest manifest. I think we really only support the manifest format for feature completeness and under the belief that we would need it sooner or later when importing the W3C's tests. It was (and remains) a fairly controversial practice compared to using filename conventions. We handle the timeout problem as follows: First, we expect tests to be fast (sub-second), and we don't expect them to timeout regularly (since running tests that timeout don't really give you much of a signal and wastes a lot of time). Second, we pick a default timeout. In Blink, this is 6 seconds, which works well for 99% (literally) of the tests on a variety of hardware (this number could probably be a couple seconds higher or lower without much impact), but the number is adjusted based on the build configuration (debug builds are slower) and platform (android is slower) . Second, We have a separate manifest-ish file for marking a subset of tests as Slow, and they get a 30s timeout. In WebKit, we have a much longer default timeout (30s) and don't use Slow markers at all. There is no build step, and no parsing of tests on the fly at test run time (except as part of the actual test execution, of course). It works well, and any delays caused by scanning for files or dealing with timeouts is a small (1-3%) part of the total test run. More importantly, it's very easy to understand and very transparent. I write all this not to argue that this is way we must do things; I recognize that a lot of discussion has preceded me getting involved in this group. However, I do think we have a ways to go before the tests are being run as part of well-oiled machines by multiple browsers, and I'm trying to provide feedback based on what we've found to work well and be liked by developers. The main point is, the more things we can make work by convention and reasonable default, the better, and if we can extract more things at little performance cost by computers, we should do so. At any rate, I haven't yet given much thought to the rest of your email; the examples you cite are quite helpful and I will take a look. -- Dirk
Received on Friday, 23 August 2013 17:48:57 UTC