- From: Dirk Pranke <dpranke@chromium.org>
- Date: Thu, 22 Aug 2013 14:41:50 -0700
- To: Kevin Kershaw <K.Kershaw@cablelabs.com>
- Cc: "public-test-infra@w3.org" <public-test-infra@w3.org>
- Message-ID: <CAEoffTAmO_TMs+MJRpYkBX+gqY6n1Rsbgo118wMS2guL5TX==Q@mail.gmail.com>
On Thu, Aug 22, 2013 at 2:27 PM, Kevin Kershaw <K.Kershaw@cablelabs.com>wrote: > Folks – > > Though we’re novices in this group, CableLabs has been developing > automated testing frameworks for a while now and I’d like to offer a couple > thoughts based on our past experience. WRT: Dirk’s restatement of the 3 > fundamental questions, I’d offer the following: > > 1) Do we want to require (or at least strongly recommend) file naming > conventions for tests and references? > > A planned set of directory structures and names is really useful for > helping developers know where to put common use or special purpose files > and it helps users of the tests find the information they're looking for. > A complex set of rules for naming individual files (like trying to > describe what a test does in its filename) is difficult to enforce or > verify and generally leads to a maintenance headache. That kind of > information is best saved within the file as metadata. Also, if you end up > renaming files over time, you generally have a more difficult time > following their history in the repo. > I agree with this; I'd be reluctant to standardize anything beyond the very simple prefix/suffix sorts of things we've been discussing. > > 2) Do we want to require the use of a manifest as a source-level object in > the repo (rather than something that could be generated via a build > process)? > > Speaking of metadata, our experience is that keeping metadata inside the > test files (e.g., the html files) themselves is the best way to keep that > test and metadata in sync. If a separate manifest file is needed to > support test runtime, then that should be generated from the test files > with an automated tool. Candidate metadata we've seen mentioned in this > thread or would like to propose are: Test ID (file name); test timeout; > test type (harness/ref/manual/etc); nontest files (e.g., helper files); > spec references. > > 3) Do we want to allow tests that are specified only in a manifest (e.g., > tests with query parameters) rather than being initiated from a > non-manifest file? > > I may be misunderstanding the intent of this question - sorry if so. I > took it to mean providing an environment where a test developer can write > and run a test w/o manifest info. I think that option should be supported. > Manifest constructs are often better suited for running lots of tests in > an automated fashion and get in the way during individual test development. > I was referring specifically to James' examples early in the thread, where a manifest would specify things like "test2.html?foo=bar" indicating that you should run test2.html and pass the query string in. However, I think you raise a valid question: if we did support manifests, would we require that all tests be in a manifest? Here there's two possible interpretations: you can't run a test at all if it's not in the manifest, or you can run the test interactively, but a test harness/runner might ignore it (and thus we might want a commit check to enforce this). Or, of course, being in the manifest could be strictly optional. I'm more strongly vested in no manifests at all, but if I get outvoted on that I'd want to think about this more :). -- Dirk
Received on Thursday, 22 August 2013 21:42:38 UTC