Re: CTS integration with LayoutTests

Hey Myles, sorry for the delay. This is going to be a bit of a brain-dump,
and I'm sure there's more to say, but hopefully this is helpful.

I'll start by saying that the tests *can* be run under WPT, *basically* by
copying the out-wpt directory
<https://github.com/gpuweb/cts/blob/main/docs/build.md> into WPT. Chromium
has done this up until now - specifically, copying into our "internal" WPT
test directory
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/wpt_internal/>
which
isn't synced with upstream WPT - but a number of things can make it quite
painful. Chromium is currently switching our CTS automation off of the
blink_web_tests infrastructure (which runs WPT tests) to our Telemetry
infrastructure (which runs WebGL tests and a lot of other things that we
run on non-virtual devices with real GPUs).

- CTS tests are diverse and complicated, so we prioritize coverage and ease
of review
<https://github.com/gpuweb/cts/blob/main/docs/intro/plans.md#1-write-a-test-plan>.
Many tests are highly parameterized, and take highly variable amounts of
time, so unfortunately they do not naturally fit within the time
constraints of WPT test runs (6 seconds per page load / "variant
<https://web-platform-tests.org/writing-tests/testharness.html#variants>"
by default). Thus far, we've taken advantage
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/webgpu/internal_cts_test_splits.pyl>
of the ability to split tests into finer chunks
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/wpt_internal/webgpu/cts.https.html>
(we
do this in Chromium - the CTS repo just generates one variant per test
<https://github.com/gpuweb/cts/blob/gh-pages/out-wpt/cts.https.html> by
default).
Splitting was originally designed to allow us to make very targeted test
suppressions, rather than suppressing entire tests (when e.g. only one
texture format is broken). However, finding the appropriate splits in order
to keep things under 6 seconds is a painful manual process, especially
because any borderline cases manifest as flakes. WPT timeouts count against
the entire page load from beginning to end, whereas ideally, we would
prefer a way to count timeouts for individual subtests (or reset the
timeout between each subtest. In principle, it would be possible to run *every
test case* in its own page load, greatly simplifying things, but the huge
number of them means this would have a lot of overhead.

- On the topic of suppressions, we do suppress tests at the sub-test level
("test cases"), but the situation is unfortunately complicated. We process
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/webgpu-cts/scripts/run_webgpu_cts.py;l=99;drc=9d114249aae9cc0b304a6989ba6b341e5d3c2def>
our
expectations file
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/WebGPUExpectations>
into
two outputs: one for Fail/Skip expectations that we pass into the CTS's own
runtime code <https://github.com/gpuweb/cts/search?q=loadWebGPUExpectations>,
and one for cases that must be run separately from others so
blink_web_tests can handle the result (Slow/Crash/Timeout/RetryOnFailure
expectations)

- Many tests will have hardware-specific results, and therefore it's
helpful to be able to write hardware-specific test expectations, which your
WPT infrastructure may or may not be able to do (so far, we've done without
this, but it means our suppresions are overbroad). WebKit's EWS also runs
on a much smaller range of hardware, so this may be less of a problem for
you.

On Wed, Mar 2, 2022 at 7:50 PM Myles Maxfield <mmaxfield@apple.com> wrote:

> Hello!
>
> During today’s WebGPU call, there was a brief discussion about integrating
> the WebGPU CTS with browsers' LayoutTests. It would probably be valuable
> for this discussion to be archived, so I thought I’d start this thread
> about it.
>
> I believe there are 2 general approaches one could take with integrating
> WebGPU’s CTS:
> 1. Integrate the repo as it stands, and teach the browsers’ test runners
> how to compile TypeScript (and potentially where to find necessary
> dependencies), etc.
>

This is essentially what we've done, although we compile the TypeScript
code at build time. (We were lucky that Chromium already had TypeScript
build support.) We used to do the TypeScript compilation in the roll
process and check in the generated files, but this was harder to maintain.
It's still doable though. Another possibility is to depend on the cts
gh-pages branch <https://github.com/gpuweb/cts/tree/gh-pages> instead,
which is precompiled (I feel like we briefly considered this, but don't
remember for sure if we did, or if so why we decided not to do it). You
would still likely need to regenerate the cts.https.html file with
different splits, as discussed above.

2. All browsers’ can run Web Platform Tests, and (I believe) there is/will
> be a periodic export of the precompiled packaged CTS into WPT. Browsers
> could import these prepackaged dumps.
>

We haven't been doing this periodic export (and deleted
<https://github.com/web-platform-tests/wpt/tree/master/webgpu> the old
export from the WPT repository) because we realized that, pre-1.0, breaking
changes were too disruptive and made it impractical to take advantage of
the WPT auto-import/auto-export behavior. Instead, until the CTS is stable,
I recommend "vendoring" a particular revision of the CTS, even though it
means you'll have to do manual (or semi-manual
<https://autoroll.skia.org/r/webgpu-cts-chromium-autoroll>) rolls.

Kai mentioned he had experience with this and could comment; are there any
> recommendations about how to proceed?
>
> Thanks,
> Myles


-Kai (he/they)