RE: Tests returning an inconsistent number of results

We started down this path with Navigation Timing with checkin #170.  At the time we wanted to avoid scripting errors when a UA had not fully implemented part of a spec while running the tests and made it such that you didn't have to put every call within an individual try/catch block.  What did you have in mind?

-Karen

-----Original Message-----
From: James Graham [mailto:jgraham@opera.com] 
Sent: Thursday, September 13, 2012 1:31 AM
To: public-web-perf@w3.org
Subject: Tests returning an inconsistent number of results

Several test submissions here use a design where if the feature is not available they fail to run the rest of the testsuite (e.g. [1]). This is an antipattern because it makes it unnecessarily difficult to compare the results between builds; ideal tests should always return the same number of results.

Will anyone object if I patch the problematic tests and commit the fixes?

[1]
http://w3c-test.org/webperf/tests/submission/Google/HighResolutionTime/test_cross_frame_start.html

Received on Tuesday, 18 September 2012 22:54:13 UTC