Re: Tests that fail due to server problems

On 14/09/16 00:12, Geoffrey Sneddon wrote:
> On Tue, Sep 13, 2016 at 11:53 PM, Mark Watson <watsonm@netflix.com> wrote:
>> We have some tests (for encrypted-media) which rely on a 3rd party server.
>> Presently, if that server fails then the tests report TIMEOUT, which is then
>> indistinguishable (for some of the tests) from certain kinds of test failure
>> (for example, video was expected to start but never did).
>>
>> What is the appropriate result when the inability to complete the test is
>> due to such a 3rd party dependency, rather than a problem with the
>> implementation under test ? Should this be NOTRUN ? How do we trigger this ?
>
> Is there any way to avoid the 3rd party dependency? Typically browser
> vendors have attempted to avoid 3rd party dependencies in tests at all
> costs given it leads to a substantial increase in intermittent
> failures, and tests that fail intermittently typically get disabled,
> especially when they cannot be fixed. (You can't gate landing a patch
> on tests passing if they fail intermittently 0.01% of the time, as
> with a large number of tests it becomes a significant percentage of
> test runs that end up with some intermittent failure, hence you end up
> unable to land any patch.)

To put this differently, in automation Firefox is configured to crash if 
it tries to access a non-local server. This is because we have a great 
deal of experience which suggests that all such tests are intermittent 
failures waiting to happen. Therefore any such tests will merely slow 
down our automation without testing anything. As soon as someone notices 
what's going on they will be disabled.

Received on Wednesday, 14 September 2016 08:28:11 UTC