Re: Knowing which tests are in the repository

Hi Kevin,

On 22/08/2013 23:27 , Kevin Kershaw wrote:
> 1) Do we want to require (or at least strongly recommend) file naming
> conventions for tests and references?
>
> A planned set of directory structures and names is really useful for
> helping developers know where to put common use or special purpose
> files and it helps users of the tests find the information they're
> looking for.  A complex set of rules for naming individual files
> (like trying to describe what a test does in its filename) is
> difficult to enforce or verify and generally leads to a maintenance
> headache.  That kind of information is best saved within the file as
> metadata.

I agree that we don't want a complex naming convention as it will simply 
never get followed. I think that the one proposed by James, with perhaps 
minor amendments, is easy to follow though.

>  Also, if you end up renaming files over time, you
> generally  have a more difficult time following their history in the
> repo.

Actually for git that's not normally true.


> 2) Do we want to require the use of a manifest as a source-level
> object in the repo (rather than something that could be generated via
> a build process)?
>
> Speaking of metadata, our experience is that keeping metadata inside
> the test files (e.g., the html files) themselves is the best way to
> keep that test and metadata in sync.

On this my experience differs. When I was working on the previous test 
framework, I got to import a lot of test suites (and back then inline 
metadata was all the rage). An awful lot of the files had broken 
metadata simply because the way that contributors produce new tests is 
by copying an existing test over and changing the content. Many of these 
tests with bad metadata had passed review with flying colours.

> Candidate metadata we've seen
> mentioned in this thread or would like to propose are:  Test ID (file
> name); test timeout; test type (harness/ref/manual/etc); nontest
> files (e.g., helper files); spec references.

My preference when it comes to metadata is to have as little of it as 
technically possible (ideally none). We can't get to none in our case, 
but we can get close, with only two items (perhaps three):

ID: that is a URL. This can either be inferred from the position of the 
file in the directory tree. In cases where it can't (because we need 
query strings or the such) then a small manifest can override (but that 
ought to be rarely used).
Type: that's harness/ref/manual/helper. We can identify this with the 
simple rules we've been discussing.
Timeout: still under discussion. I reckon that adding "-long" in the 
file name can in fact be more than enough to identify the cases that 
might take longer than usual.

Everything else I believe should come from services built as other 
layers. Author and reviewer information can be extracted from GitHub. 
Things like spec references can be built as a third-party service. But I 
would rather not crowd the repo with extraneous metadata: it makes the 
reviewing more painful, it leads to inconsistencies, it requires more of 
contributors.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Friday, 23 August 2013 13:32:42 UTC