Re: Some updates to do in documentation files, or discussions/clarifications to have

FWIW, this probably more lives in issues on the GitHub repository
(https://github.com/w3c/web-platform-tests) rather than in email,
because it'll likely get lost over time otherwise.

On Sat, Mar 25, 2017 at 9:13 PM, Gérard Talbot <www-style@gtalbot.org> wrote:
> Geoffrey,
>
> 1-
> Reference File
> "All metadata is removed."
> http://web-platform-tests.org/writing-tests/reftests.html#reference-file
>
> Actually, that is usually true but sometimes not true. If a reference file
> uses some flag(s), then a <meta name="flags" content=""> will be required.

AFAIK, nothing has ever done anything with those flags. As far as I'm
aware, they've practically always been required to be on the test
itself.


> 2-
> Support Files
> http://web-platform-tests.org/writing-tests/general-guidelines.html#support-files
>
> Somehow, somewhere, it must be said that all images (new and old ones) must
> be put into a separate, distinct /support folder. Images can not be
> alongside tests. At least 20 tests (imported from vendors) into
> /css-backgrounds-3 folder have problems because of that.

Yeah, that needs to be put somewhere as a css/ specific restriction.
To be fair, I didn't even know what exactly caused the file mismatch
merge errors until a few weeks ago (tl;dr: find all tests for a given
spec, then all support directories alongside them get merged in their
entirety and must have unique filenames or identical files). Probably
as a new page alongside the CSS Metadata stuff? Can you file an issue
for that?


> 3-
> Controlling When Comparison Occurs
> http://web-platform-tests.org/writing-tests/reftests.html#controlling-when-comparison-occurs
>
> "reftest-wait" and "reftest-print"
>
> Mozilla people have been adding "reftest-print" to print related tests.
> Discussion starts here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=685012#c49
> We may have to have a discussion on this and eventually add a paragraph
> about this.

There's been a few discussions about print from time-to-time, but thus
far nothing has happened at level of any shared testsuite.


> 4-
> Ahem usage
> http://web-platform-tests.org/writing-tests/ahem.html#usage
>
> "as such, the font shorthand should normally be used." "doesn't use font
> shorthand;": font shorthand should not be recommended. I think it should be
> neutral: not encouraged, not discouraged. When I create a test that uses
> font shorthand, then I can not easily and quickly examine subproperties with
> web inspection tools. Also, font shorthand is more "sensitive", demanding:
> if only 1 subproperty is invalid, then the shorthand is rejected, ignored
> while declaring subproperties in individual declarations will only affect
> the invalid declarations and not the other valid subproperty declarations.

The reason why the shorthand is recommended is it avoids any risk of
accidentally inheriting non-default property values, which would
likely result in Ahem not rendering as expected.

As far as I'm aware, all web development tools provide ways to see
both defined and computed values; the latter should show the
individual properties. I don't think anyone should be using any font
value for Ahem that isn't valid CSS1, hence I don't think we should be
concerned about the shorthand being rejected.


> 5-
> Recommended versus to avoid:
> http://web-platform-tests.org/writing-tests/ahem.html#usage
>
> a) I would put a green border around the recommended block and a red border
> around the to avoid block. Otherwise would use a light red-ish background
> and a light green-ish background.
>
> b) Also
> .highlight .c {
>     color: #998;
> this color is too light; insufficient color contrast.

Can you file issues for those? (The second is just the default
stylesheet of the theme!)


> 6-
> Be short
> http://web-platform-tests.org/writing-tests/general-guidelines.html#be-short
> "
> For reftests in particular scrollbars at 800×600px window size must be
> avoided unless scrolling behavior is specifically being tested.
> "
> This, I believe, was changed to 600px by 600px. Thread starts here:
> Reftest Maximum Viewport Size
> https://lists.w3.org/Archives/Public/public-css-testsuite/2012Sep/0020.html
>
> Also:
> "the two files render pixel-for-pixel identically within a 600x600 window
> including scroll-bars if present;"
> http://web-platform-tests.org/writing-tests/reftests.html#components-of-a-reftest
>
> "The device has a viewport width of at least 800px."
> http://web-platform-tests.org/writing-tests/assumptions.html
> If pixel comparison involves only 600px, then why should we require a
> minimum of 800?
>
> I am wondering if the documentation is consistently coherent with regards to
> viewport width and presence of scrollbars.
>
> "The test renders within a 600x600 viewport, only displaying scrollbars if
> their presence is being tested. "
> http://web-platform-tests.org/reviewing-tests/checklist.html

We have total disagreement about this, some documentation saying
800x600, some saying 600x600, some implementations doing one, some the
other… It's a complete and utter mess. :\

This was a pre-existing issue with the old documentation for the CSS
testsuite (at testthewebforward.org), hence I didn't consider it a
priority to fix with the cleanup recently. This needs buy-in from
implementors, hence is more than a documentation issue, really. (For
reference, the old wiki documentation didn't define any resolution, or
even a minimum, so who knows how you were meant to run them.)


> 7-
> Fixing lint errors
> http://web-platform-tests.org/writing-tests/lint-tool.html#fixing-lint-errors
>
> fix: replace all setTimeout(...) calls with step_timeout(...) calls.
>
> What is the problem, what is wrong with
> <body onload="setTimeout(test, 5)">
> ?
> eg
> http://test.csswg.org/shepherd/testcase/background-root-101/

The problem is many tests use timeouts for "something happened before
this" and these become unreliable and randomly fail on, e.g., a
low-end ARM device. (And with tens of thousands of tests, unreliable
tests become a real problem very quickly when you're running them all
hundreds of times per day.)

In the case of that test, it's unlikely to cause problems; we also
need to have some better way to deal with reftests for that lint. Can
you file an issue for that?


> 8-
> Linking to a shared reference file which is across folders is no longer
> recommended or doable. But this is not mentioned in
> http://web-platform-tests.org/writing-tests/reftests.html#writing-a-good-reftest

Why is it not recommended or doable? As far as I'm aware, it works
fine with the build system and is recommended insofar as it avoids
having more duplicated files around the testsuite?


> 9-
> Use extended form in title for abbreviations. eg <abbr title="Common
> Interface">CI</abbr>
> Can't remember what CI means...
> http://web-platform-tests.org/reviewing-tests/checklist.html#all-tests

We should probably call them "status checks" given that's what GitHub
calls them. Can you file an issue? Or open a pull request to fix that
(the site is built out of the docs/ subdirectory of
web-platform-tests), if you want something that should be relatively
easy to get some experience with git and GitHub?

> Avoid "here" as links.
> http://web-platform-tests.org/writing-tests/general-guidelines.html#be-cross-platform

Yeah, that's another easy fix. I think at the time I couldn't work out
how to phrase that without having the whole sentence to be a link. :)

Issue/pull request, please!


> 10-
> http://web-platform-tests.org/writing-tests/css-metadata.html#requirement-flags
> Example 2 uses 2 obsolete flag tokens.

Uh, yeah.

Can you file an issue or a pull request for that?


> 11-
> <link rel="help" href="https://www.w3.org/TR/CSS21/visudet.html#q7" />
> in
> http://web-platform-tests.org/writing-tests/css-metadata.html#specification-links
> should be removed from example 2 since q7 target has no anchor and never was
> in specification's Table of Contents on top of everything.

Pretty sure that comes from testthewebforward.org too. Can you file an
issue or a pull request for that?


> 12-
> Nit: "whether a test has passed of failed.": or
> http://web-platform-tests.org/writing-tests/general-guidelines.html#be-self-describing
>
> Nit: "Various support files are available in in the /common/ and /media/": 2
> in
> http://web-platform-tests.org/writing-tests/general-guidelines.html#support-files

Can you file an issue or a pull request for those?


> HTML5 doctype is recommended but is XHTML 1 strict doctype declaration
> forbidden for tests? The current documentation does not mention this.

Recommendations shouldn't be taken to exclude what *isn't*
recommended: people should just be pushed towards them, but there's
almost always reasons not to (e.g., testing behaviour with the
different DOCTYPEs!). Do you have any suggestion as to how to make
that clearer? Feel free to similarly open an issue or a pull request
for that?

/g

Received on Monday, 3 April 2017 19:09:14 UTC