Re: Tests that fail due to server problems

On Tue, Sep 13, 2016 at 11:53 PM, Mark Watson <watsonm@netflix.com> wrote:
> We have some tests (for encrypted-media) which rely on a 3rd party server.
> Presently, if that server fails then the tests report TIMEOUT, which is then
> indistinguishable (for some of the tests) from certain kinds of test failure
> (for example, video was expected to start but never did).
>
> What is the appropriate result when the inability to complete the test is
> due to such a 3rd party dependency, rather than a problem with the
> implementation under test ? Should this be NOTRUN ? How do we trigger this ?

Is there any way to avoid the 3rd party dependency? Typically browser
vendors have attempted to avoid 3rd party dependencies in tests at all
costs given it leads to a substantial increase in intermittent
failures, and tests that fail intermittently typically get disabled,
especially when they cannot be fixed. (You can't gate landing a patch
on tests passing if they fail intermittently 0.01% of the time, as
with a large number of tests it becomes a significant percentage of
test runs that end up with some intermittent failure, hence you end up
unable to land any patch.)

As a result, practically the tests are going to end up being manual
tests regardless of automation or not. As such, I'd suggest what the
actual status returned is is relatively irrelevant.

I presume from your question that there's some way to detect when the
3rd party server fails to respond—I guess what I'd suggest is having
the tests fail if the server fails to respond, with a description
saying as much (and probably saying to try running it again).

/gsnedders

Received on Tuesday, 13 September 2016 23:12:55 UTC