Re: Knowing which tests are in the repository

On Thu, Aug 22, 2013 at 2:27 PM, Kevin Kershaw <K.Kershaw@cablelabs.com>wrote:

> Folks –
>
> Though we’re novices in this group, CableLabs has been developing
> automated testing frameworks for a while now and I’d like to offer a couple
> thoughts based on our past experience.  WRT: Dirk’s restatement of the 3
> fundamental questions, I’d offer the following:
>
> 1) Do we want to require (or at least strongly recommend) file naming
> conventions for tests and references?
>
> A planned set of directory structures and names is really useful for
> helping developers know where to put common use or special purpose files
> and it helps users of the tests find the information they're looking for.
>  A complex set of rules for naming individual files (like trying to
> describe what a test does in its filename) is difficult to enforce or
> verify and generally leads to a maintenance headache.  That kind of
> information is best saved within the file as metadata.  Also, if you end up
> renaming files over time, you generally  have a more difficult time
> following their history in the repo.
>

I agree with this; I'd be reluctant to standardize anything beyond the very
simple prefix/suffix sorts of things we've been discussing.


>
> 2) Do we want to require the use of a manifest as a source-level object in
> the repo (rather than something that could be generated via a build
> process)?
>
> Speaking of metadata, our experience is that keeping metadata inside the
> test files (e.g., the html files) themselves is the best way to keep that
> test and metadata in sync.  If a separate manifest file is needed to
> support test runtime, then that should be generated from the test files
> with an automated tool.  Candidate metadata we've seen mentioned in this
> thread or would like to propose are:  Test ID (file name); test timeout;
> test type (harness/ref/manual/etc); nontest files (e.g., helper files);
> spec references.
>
> 3) Do we want to allow tests that are specified only in a manifest (e.g.,
> tests with query parameters) rather than being initiated from a
> non-manifest file?
>
> I may be misunderstanding the intent of this question - sorry if so.  I
> took it to mean providing an environment where a test developer can write
> and run a test w/o manifest info.  I think that option should be supported.
>  Manifest constructs are often better suited for running lots of tests in
> an automated fashion and get in the way during individual test development.
>

I was referring specifically to James' examples early in the thread, where
a manifest would specify things like "test2.html?foo=bar" indicating that
you should run test2.html and pass the query string in.

However, I think you raise a valid question: if we did support manifests,
would we require that all tests be in a manifest? Here there's two possible
interpretations: you can't run a test at all if it's not in the manifest,
or you can run the test interactively, but a test harness/runner might
ignore it (and thus we might want a commit check to enforce this). Or, of
course, being in the manifest could be strictly optional.

I'm more strongly vested in no manifests at all, but if I get outvoted on
that I'd want to think about this more :).

-- Dirk