[proposal] the Spotlight Project

Hi Team,

               The Spotlight Project is intended to provide insights into the overall user performance of Web pages by making test results public. In conversation with Phillipe earlier this year, I suggested this to the W3C. With his encouragement, I'm writing down the basic concept for this group to discuss prior to moving forward. I'm hoping this discussion will lead to further refinement of the concept, solutions to outstanding problems (and there are a few) and a better shared understanding of how we might put this plan into action. With that being said, the remainder of this (long) email is a description of the project. I'd appreciate your taking the time to read it and share your thoughts on this mailing list.
Regards,
D-
The Spotlight Project

1.      Overview

The Spotlight Project is intended to shine a light on the end-user performance of Web pages by making test results public, transparent, and easily understood by users. The essential concept is to establish a Website with current and past performance test results for the top N web sites and make the results publicly available to anyone in an easily understood manner. We would also perform continuous testing of Web performance under a number of different conditions, with different user scenarios and devices. This project would be open to participation and contributions from the community in a way similar to webplatform.org.

By making comparative test results public, using a credible, transparent methodology, we can call attention to current performance issues, provide users with a better understanding of their Web experience, and shine a spotlight on slow Web pages, encouraging the entire industry to focus on making Web surfing faster for everyone.



2.      Making Performance Data Public

The key idea behind the Spotlight Project is that performance data is public data. By testing popular sites and services in a public, transparent, and credible way, we can help focus attention on Web performance. Most companies with a significant Web presence have data on their site's performance. The Spotlight Project aims to provide a comparative view of the end user's experience for the most popular Web sites over time.



3.      Methodology

Web testing methodologies are often controversial, and there's no one 'right' answer on Web performance testing. However, I think that everyone can agree that an open, credible methodology would require testing along at least 4 axes:

*        Geographical distribution

*        Network context

*        Client diversity

*        User scenarios
In the end the precise details of the methodology would need to be worked out among the participants, and like many other difficult topics a compromise that integrates the wisdom of many viewpoints is the desired result. The important points are that the methodology is sound, transparent, and credible.

4.      Operational Issues

Web performance testing is not simple - that's part of why W3C's help is needed for this project to succeed. In order to make performance data not only consumable but appetizing to our intended audience, we'll need some support. Behind the scenes, data has to be collected, stored, and reported to users. The associated infrastructure and operational aspects of the Spotlight project would be borne by the community at large, hopefully with donations from our contributors.



5.      Why W3C?

W3C is uniquely situated to make the Spotlight project work. W3C's ability to bring the major players together to support this project, and to lend credibility and technical insight are a key part of making the project successful. In addition, the value of the Spotlight Project's data will increase due to W3C's longevity as an institution on the Web, providing continuity that no one else can provide.



6.      Long-term Evolution

The Spotlight Project is intended to make a persistent set of performance data available for the long term. The need for continuity and a longitudinal view of Web performance across different browsers, geographies, and sites is central to the idea of making the data public and transparent. Inevitably the tests performed and data collected, and the associated methodology, would change over time. We should be prepared to continuously evolve along with the Web.



7.      Making the Web Faster

A big part of the Spotlight Project's mission would be educating the community about Web performance. Beyond publishing data about how popular sites are doing, we need to provide users with the means to make them better. This might include guidelines, articles, forums, and other means of helping the community make Web pages load faster.



8.      Fellow Travelers

A number of other people have had similar ideas and proposed similar projects. I'm indebted to them all. Multiple different aspects of the project proposed here have been implemented on webpagetest.org, HTTParchive.org, measurementlab.net, and stevesouders.com, among others. Akamai publishes a great report on the State of the Internet quarterly, and I'd like to propose something similar for this project as well. Recently, x509labs.com created a similar project for OCSP providers, publishing comparative tests in a very similar way to what's proposed here, and achieved remarkable results in reducing response times globally - no mean achievement.  This proposal intends to build on all of these and more, and with W3C's help, provide a spotlight on Web performance.