Re: Simplifying metadata

FWIW, in the meeting this afternoon myself and fantasai chaired, it became
clear that what's holding back several of the browser vendors from is the
overheads of contributing, and relevant to this thread is that they'd much
rather have no metadata in the tests.

On Wed, Oct 28, 2015 at 8:50 AM, Linss, Peter <peter.linss@hp.com> wrote:
>
> On Oct 27, 2015, at 5:50 AM, Geoffrey Sneddon <me@gsnedders.com> wrote:
>
> On Tue, Oct 27, 2015 at 5:10 PM, Linss, Peter <peter.linss@hp.com> wrote:
>>
>> On Oct 27, 2015, at 12:31 AM, Geoffrey Sneddon <me@gsnedders.com> wrote:
>>
>> b) We don't need author metadata on any new tests, because that metadata
>> is stored in git/hg. (It's essentially been entirely redundant since we
>> moved away from SVN, as git/hg can store arbitrary authorship data
>> regardless of whether the author has source-tree access.)
>>
>>
>> The author metadata was originally provided because authors didn’t
>> necessarily have access to the repo and sometimes submitted tests via email
>> or other means. So the first committer was often not the author. I’m fine
>> with only requiring author metadata in that case.
>>
>
> Oh, what I said was based on mistaken belief about hg. In git there are
> separate "Author" and "Committer" fields—and the committer can set the
> Author field to whatever they want, hence in git there's no reason to ever
> require author metadata in the file. This further makes me feel we should
> drop hg! As far as I'm aware the only reason why we have the hg/git
> mirroring is because much of the Shepard tooling integrates with hg—someone
> (probably you or me!) should investigate how much work it would be to
> migrate purely to git. (I heard something about contributors before. Will
> we actually lose any contributors if we move to git exclusively?)
>
>
> Both because of the Shepherd tooling and because some contributors prefer
> hg (and several don’t know git). I can’t speak to how many we’d lose by
> switching to git exclusively, but I’ve been told by more than one that they
> don’t want to learn yet another source control system. Whether or not that
> burden would merely be an inconvenience or would drive some of them away I
> can’t say, but our contributors are so few and so valuable, that I’d rather
> not lose a single one.
>

The overwhelming majority of CSS test cases are written by browser vendors:
we should do what's needed to make it easier for them to upstream their
tests. (This isn't really a response to that, but a comment on contributors
in general.)


> d) Specification links I'm kinda unconvinced by, but I don't really care
>> enough to argue over. I know Shepard uses it.
>>
>>
>> Shepherd uses it for tracking test to spec mapping, but it’s also
>> required by the build system. This is how the build system determines which
>> tests go in which test suites. The test harness also relies on the spec
>> links to generate the spec annotations (visible in the CSSWG drafts). So
>> this one is most definitely necessary (and is fact is the only metadata our
>> tooling really relies on).
>>
>
> As I said, I don't care to argue over this! I'm surprised we determine
> test suite membership based on it, though—why don't we just base this on
> source directory?
>
>
> Because many tests are linked to more than one spec so don’t live in a
> single test suite. (and I wasn’t arguing, just explaining)
>

Having heard what's holding people up from contributing more tests, I want
to revisit this. I want to try and get rid of all metadata in the common
case. Do you see any way in which we can get rid of this?


>
>
>
>> e) Requirement flags I feel we should really revisit. We want to have
>> enough flags to be able to run the tests, especially in CI. I'm not so
>> interested in running tests through Shepard's UI, because I simply think
>> it's not valuable—it's almost never done, because it's pointlessly slow.
>> Browsers we should aim at getting them run in CI systems (probably with
>> some way to upload results to Shepard so we can have the CR-exit-criteria
>> views there), and minor UAs also likely can run the tests in a more
>> efficient way (as you want to determine pass/fail by unique screenshot, not
>> looking at a thousand tests all of which say, "This text should be green"
>> identically).
>>
>> So:
>> […]
>>
>> * HTMLonly — why do we have this rather than just using asis and HTML
>> source files?
>>
>>
>> This is an indicator to the build system to not generate an XHTML version
>> of the test (as it wouldn’t make sense)
>>
>
> Right, I get that. But isn't this equivalent to having an HTML source file
> and using the asis flag in it? What's the benefit of it going through the
> build system?
>
>
> All the files go through the build system in one way or another.
>
> As it’s currently implemented, the HTMLonly/nonHTML flags currently don’t
> imply asis, they still can get modified by the build system, as in updating
> paths to reference files, etc, they just avoid the conversion to to the
> other format. The asis flag prevents any modification of the file (and
> would be used for files with bad markup or other features that would not
> survive being parsed and re-serialized).
>

Ah, I'd forgotten the build system did so much.


>
>> f) Test assertions… are we doing anything we this, beyond what we use
>> specification links for? If not, we should stop requiring them.
>>
>>
>> These are for test reviewers to help understand what the test is trying
>> to do. It’s often not obvious and can be helpful.
>>
>
> In my experience I don't find the test assertions help much—a few comments
> places at the relevant places in the test would be far more helpful to
> understand what's going on IMO.
>
>
> Others have had different experiences and have explicitly asked for
> assertions to help with reviews. I accept that the same job can be done
> with comments, but having it in an extractable format is useful too.
>

So the WebKit guys apparently have a policy of having as little as possible
in tests (and to try and have them obvious what they're testing), which
means they often don't have assertions/comments. I'm not sure I really like
that, but it's something to consider when it comes to getting them to
upstream their tests.


> Again, we’ve never rejected a test for missing this metadata, it’s at the
> author’s discretion.
>

That's not the impression I or many of those representing browser vendors
had.

/g