Re: The Future of the Build System

On Mon, Nov 2, 2015 at 7:45 AM, Linss, Peter <peter.linss@hp.com> wrote:

>
> > On Nov 1, 2015, at 12:53 AM, Geoffrey Sneddon <me@gsnedders.com> wrote:
> >
> > Hi!
> >
> > Yeah, it's me again. (I feel like I've single-handedly increased the
> traffic on here sevenfold. Sorry!)
> >
> > I believe that we have rough consensus (mostly from TPAC rather than
> on-list discussions) that we should ensure that it's possible to run the
> test suite without building anything except a manifest. To that end (mostly
> also with rough consensus):
> >
> >  * We should ensure the build system is capable of handling HTML as
> source files and generating XHTML from them. (We're now at a point where
> every known user of the CSS test suites support HTML, but not all support
> XHTML—Servo is the notable exception here; furthermore, HTML content vastly
> dominates and hence should be the preferred format.)
>
> It already does.
>

There's a comment in w3ctestlib saying that that is untested—hence the
ensuring it's capable. (On the other hand, we seem to rely on this in some
places anyway!)


> >  * We should convert the source files to HTML. (We should probably move
> away from HTML 4.01/XHTML 1.1 to HTML 5 which we're at it.)
>
> My concern here is introducing unnecessary churn. We have a lot of result
> data in the test harness and a change to the test source causes the old
> results to no longer apply.
>
> There’s a path forward here as the test harness has the capability to know
> that different revisions are equivalent (for metadata changes, etc), we
> just need to make sure that the conversion process gives this data to the
> test harness so we don’t lose thousands of tests’ result data.
>

Hmm, yeah, that sounds like something we need to work out.


> (It should also go without saying that the build tools should be used to
> do the conversion so that the new source files are identical to the build
> output.)
>
> I can help here to make sure we do this right.


Does the build system not do some path rewriting within tests, or am I
misremembering how much it actually does? I'm also unsure if we should use
the build output anyway, given that'll significantly increase the size of
the diffs (due to attribute reordering and whitespace changes).


> >  * We should introduce a lint tool given building now becomes optional,
> and we need to introduce a manifest builder for the unbuilt test suite. We
> should make sure the lint tool is a superset of the warnings from the build
> system.
>
> Sure. There’s already some code for this in Shepherd and w3ctestlib, we
> can do some factoring to make a stand-alone linter.
>

We should probably make sure that everything agrees what's an error and
what's a warning. And mark things as expected failures (so that we can
simply check that any new commit doesn't add any new failures.)


> The build tools also already generate manifests, making a manifest-only
> build should be trivially turning off parts of the build process. Let me
> know if you need a different manifest format.
>

I *think* the only thing we need different is the whole path for the
test/ref fields, because obviously an extensionless id isn't very useful
there.


> >  * We should make the build system capable of grabbing authorship from
> hg/git if it isn't explicitly defined.
>
> Shepherd already has the concept of a file ‘owner’ in addition to the
> authors (the owner is the first person to commit the file). I can add an
> API to expose that data. Note that (especially for older files) the owner
> often had nothing to do with creating or editing the test. Furthermore,
> there are often edits (by myself for example) that do nothing but correct
> spec links, reference links, or other minor metadata issues (or rename
> files), these kinds of edits should not be considered ‘authorship’.
>
> Also, Shepherd already has all the revision information for every file in
> the repo, so again, I can add an API to expose this data easily.
>

Hmm, this goes back to the whole metadata discussion… I don't really know!

/g