Re: Towards a better testsuite: Build System

On Mon, Apr 18, 2016 at 12:28 AM, Peter Linss <peter.linss@hp.com> wrote:
>
>> On Apr 14, 2016, at 9:47 PM, Geoffrey Sneddon <me@gsnedders.com> wrote:
>>
>> On Fri, Apr 8, 2016 at 6:00 PM, fantasai <fantasai.lists@inkedblade.net> wrote:
>>> I don't have much opinion on whether we keep or discard the
>>> build system, but I don't think in any case that it should
>>> be necessary in order for people to run or otherwise use the
>>> tests. Tests are identified by filename, and run just fine
>>> without the build system, so there's no need to build in the
>>> general case.
>>
>> For those who don't read minutes: on the telecon I took ACTION-766 to
>> essentially discard the build system.
>
> The action wasn’t to “discard” the build system, it was to make building un-necessary to run the tests. As I said on the call, we have infrastructure (the harness, the spec annotation system, et al) that currently relies on the build output. I thought it was fairly clear that we’d still be running the existing build system on the server until the infrastructure gets updated to not rely on it.

OK, I obviously hadn't understood that our infrastructure was
necessarily a blocker. :)

What I think was clear from the call is that outside of the WG's
infrastructure, nothing relied upon the build system. IMO, it adds
sufficient complexity to authoring tests (how many long-term
contributors have fallen foul of unique-filename restriction, for
example) that ideally we should just discard it if it affects nobody
outside of the WG, and we should just fix our infrastructure.

How much of the infrastructure actually relies on it? Taking a look
around the test harness source code I found very little, but I could
just be missing things (I after all don't really understand much of
how it works, especially given its dearth of documentation). It seems
like it shouldn't be too much work to get it working with the unbuilt
testsuite if we introduce a new "unbuilt" format.

That said, quite how it builds the testsuite and populates the
database is opaque to me: there certainly doesn't appear to be
anything that runs build.py in any of the server apps repositories, so
I could be missing some fairly complex part of the system that would
need updating!

Lastly, just to make sure I'm not missing anything: the spec
annotation system is that which adds the "x tests" annotation to
specs, loading a script and data from the test harness?

/Geoffrey

Received on Tuesday, 19 April 2016 02:32:05 UTC