CSS Testsuite Status

As of today, posting mostly so this is written up somewhere prior to TPAC:

 * Almost possible to build a manifest, giving a list of all test
   files, using web-platform-test tools. The largest part of the diff
   compared with what the CSS Testsuite build system outputs is caused
   by tests that do not link to a specification anchor that Shepherd
   knows about, which the web-platform-test tools care nothing for.

 * The CSS Testsuite build system is, once that has landed, only
   needed for the CSS Test Harness. Given the status quo is many of
   the restrictions required by the build system are oft forgotten or
   otherwise neglected (we have no broken links in the unbuilt tests,
   but a reasonable number in the built tests), it would be nice to
   drop the requirement from the CSS Test Harness or replace it
   (discussion on Friday; what do we need from it?).

 * It would be good to move away from relying on Apache with no
   server-side scripting available; presently to work around this we
   have a number of tests that rely on `hixie.ch`. The unbuilt tests
   currently work with wptserve without issue, and wptserve provides
   the necessary scripting functionality built-in. As far as I'm
   aware, the only thing stopping us from changing policy here is the
   CSS Test Harness and its reliance on the built tests, with which
   wptserve does not currently work.

 * Circular references (`test.html` -> `ref.html` -> `test.html`),
   despite the documentation recommending them for years, do not
   exist. Given cycles can make the reftest graph unsatisfiable, it
   would be nice to forbid them.

 * Do we want to allow tests to point to other tests as references?
   Presently we have only two such cases in the entirety of the
   testsuite:
   `css-transforms-1/transform3d-rotatex-perspective-001.html` and
   `css21/text/text-indent-114.xht`.

 * What blocks us disabling pushes to Mercurial? What blocks us from
   disabling new issues being filed on Shepherd? (i.e., what blocks us
   from moving entirely to GitHub?)

 * Microsoft still, as far as I'm aware, only vendor running the
   majority of the testsuite. Most other vendors are waiting for the
   merge with web-platform-tests to happen.

/gsnedders

Received on Sunday, 18 September 2016 16:54:46 UTC