W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2016

Re: Tests that fail due to server problems

From: Mark Watson <watsonm@netflix.com>
Date: Wed, 14 Sep 2016 08:20:11 -0700
Message-ID: <1819809459256899744@unknownmsgid>
To: James Graham <james@hoppipolla.co.uk>
Cc: "public-test-infra@w3.org" <public-test-infra@w3.org>
> On Sep 14, 2016, at 1:28 AM, James Graham <james@hoppipolla.co.uk> wrote:
>
>> On 14/09/16 00:12, Geoffrey Sneddon wrote:
>>> On Tue, Sep 13, 2016 at 11:53 PM, Mark Watson <watsonm@netflix.com> wrote:
>>> We have some tests (for encrypted-media) which rely on a 3rd party server.
>>> Presently, if that server fails then the tests report TIMEOUT, which is then
>>> indistinguishable (for some of the tests) from certain kinds of test failure
>>> (for example, video was expected to start but never did).
>>>
>>> What is the appropriate result when the inability to complete the test is
>>> due to such a 3rd party dependency, rather than a problem with the
>>> implementation under test ? Should this be NOTRUN ? How do we trigger this ?
>>
>> Is there any way to avoid the 3rd party dependency? Typically browser
>> vendors have attempted to avoid 3rd party dependencies in tests at all
>> costs given it leads to a substantial increase in intermittent
>> failures, and tests that fail intermittently typically get disabled,
>> especially when they cannot be fixed. (You can't gate landing a patch
>> on tests passing if they fail intermittently 0.01% of the time, as
>> with a large number of tests it becomes a significant percentage of
>> test runs that end up with some intermittent failure, hence you end up
>> unable to land any patch.)
>
> To put this differently, in automation Firefox is configured to crash if it tries to access a non-local server. This is because we have a great deal of experience which suggests that all such tests are intermittent failures waiting to happen. Therefore any such tests will merely slow down our automation without testing anything. As soon as someone notices what's going on they will be disabled.

Yes, understood. In this case, these tests will need to be run
manually unless you yourselves have an appropriate DRM server that you
could integrate into your test framework.

...Mark
>
>
Received on Wednesday, 14 September 2016 15:20:46 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC