[Bug 14839] New: Separate WebKit into Safari and Chrome

http://www.w3.org/Bugs/Public/show_bug.cgi?id=14839

           Summary: Separate WebKit into Safari and Chrome
           Product: Testing
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P3
         Component: Test Framework
        AssignedTo: mike@w3.org
        ReportedBy: tonyg@chromium.org
         QAContact: dave.null@w3.org
                CC: public-test-infra@w3.org


For the Navigation Timing tests, it is not accurate to group Safari and Chrome
into one WebKit engine.

The summary page makes it look like WebKit often fails the tests:
http://w3c-test.org/framework/results/nav-timing-default/

However, drilling into one test it becomes clear what is going on:
http://w3c-test.org/framework/details/nav-timing-default/test_navigate_within_document/engine/webkit/

All Chrome runs pass and all Safari runs fail.

So the test result is really just indicating whether the test was run more
often in Safari or Chrome. This needs to be fixed to separate them out. I
suggest columns "WebKit (Safari)" and "WebKit (Chrome)". WebKit variants other
than Safari/Chrome should be sufficiently rare to discard.

I suspect this may be true for test suites other than Navigation Timing, but
haven't verified. Very often there is browser code outside of WebKit that is
necessary to support web platform APIs. It is also very common to have flag
guards to enable/disable web platform APIs in WebKit.

-- 
Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

Received on Wednesday, 16 November 2011 09:44:37 UTC