RE: [proposal] the Spotlight Project

HI Mark,

	Back in the office today - sorry for the delay. In answer to your questions, no, the Spotlight Project isn't intended to be competitive in any way, but to build on the work of others in order to achieve a slightly different goal. I would certainly expect that all of the existing 'fellow traveler' sites would continue as they are. While each of them covers a specific aspect of the project, none of them fulfill the goals we've set for Spotlight. I would certainly welcome all the help that the community can provide -we really want this to be a way for the community to learn and improve, but also to contribute. W3C's got enormous technical depth in this area, that is one reason why I'd like this project to be done under W3C auspices. 

	Since I know you have a lot of experience in this area Mark, I'm wondering if you would be willing to help participate and set this in motion? 

Regards,

D-

-----Original Message-----
From: Mark Nottingham [mailto:mnot@mnot.net] 
Sent: Sunday, June 09, 2013 5:41 PM
To: Austin,Daniel
Cc: Mark Tomlinson; public-html5-performance@w3.org; public-web-perf@w3.org
Subject: Re: [proposal] the Spotlight Project

Hi Daniel,

Just curious - how much buy-in does this have from the existing efforts you list? I.e., is this going to supplant HTTP Archive et al, or will it co-exist, or is it competitive? What does it add above those efforts -- or is it just pulling them all under one roof?

Cheers,


On 05/06/2013, at 5:45 AM, "Austin,Daniel" <daaustin@paypal-inc.com> wrote:

> Hi Mark,
> 
>  
> 
>                 Thanks for the feedback! It looks like loadosophia is among the group of fellow travelers for this project. I really like their displays, very nice. There are still a lot of differences between loadosophia and Spotlight however. Spotlight's intention is to make performance data publicly available to everyone, and to provide baselines for the most popular sites. I'm also (deliberately) avoiding questions about load, to simplify things and also because we're talking about the user's experience.  There's a lot here to be considered, thanks for the link.
> 
>                 Do you also have a link to your BOK?
> 
>  
> 
> Regards,
> 
>  
> 
> D-
> 
>  
> 
> From: Mark Tomlinson [mailto:mark@careytomlinson.org] 
> Sent: Friday, May 24, 2013 11:44 AM
> To: Austin,Daniel
> Subject: Re: [proposal] the Spotlight Project
>  
> 
> Hi Daniel - this sounds like a cool idea, similar to the public data sets on Loadosophia.org
> 
>  
> 
> We are also working on a Performance Engineering Book of Knowledge that you might be interested in joining.
> 
>  
> 
> -mt
> 
>  
> 
>  
> 
> --
> 
> Mark Tomlinson | 215.520.5450 | mark@careytomlinson.org | mtomlins.blogspot.com
> 
> 
> On May 24, 2013, at 2:16 PM, "Austin,Daniel" <daaustin@paypal-inc.com> wrote:
> 
> Hi Team,
> 
>  
> 
>                The Spotlight Project is intended to provide insights into the overall user performance of Web pages by making test results public. In conversation with Phillipe earlier this year, I suggested this to the W3C. With his encouragement, I'm writing down the basic concept for this group to discuss prior to moving forward. I'm hoping this discussion will lead to further refinement of the concept, solutions to outstanding problems (and there are a few) and a better shared understanding of how we might put this plan into action. With that being said, the remainder of this (long) email is a description of the project. I'd appreciate your taking the time to read it and share your thoughts on this mailing list.
> 
> Regards,
> 
> D-
> 
> The Spotlight Project
> 
> 1.      Overview
> The Spotlight Project is intended to shine a light on the end-user performance of Web pages by making test results public, transparent, and easily understood by users. The essential concept is to establish a Website with current and past performance test results for the top N web sites and make the results publicly available to anyone in an easily understood manner. We would also perform continuous testing of Web performance under a number of different conditions, with different user scenarios and devices. This project would be open to participation and contributions from the community in a way similar towebplatform.org.
> By making comparative test results public, using a credible, transparent methodology, we can call attention to current performance issues, provide users with a better understanding of their Web experience, and shine a spotlight on slow Web pages, encouraging the entire industry to focus on making Web surfing faster for everyone. 
>  
> 2.      Making Performance Data Public
> The key idea behind the Spotlight Project is that performance data is public data. By testing popular sites and services in a public, transparent, and credible way, we can help focus attention on Web performance. Most companies with a significant Web presence have data on their site's performance. The Spotlight Project aims to provide a comparative view of the end user's experience for the most popular Web sites over time.
>  
> 3.      Methodology
> Web testing methodologies are often controversial, and there's no one 'right' answer on Web performance testing. However, I think that everyone can agree that an open, credible methodology would require testing along at least 4 axes:
> *        Geographical distribution
> *        Network context  
> *        Client diversity
> *        User scenarios
> 
> In the end the precise details of the methodology would need to be worked out among the participants, and like many other difficult topics a compromise that integrates the wisdom of many viewpoints is the desired result. The important points are that the methodology is sound, transparent, and credible.
> 
> 4.      Operational Issues
> Web performance testing is not simple - that's part of why W3C's help is needed for this project to succeed. In order to make performance data not only consumable but appetizing to our intended audience, we'll need some support. Behind the scenes, data has to be collected, stored, and reported to users. The associated infrastructure and operational aspects of the Spotlight project would be borne by the community at large, hopefully with donations from our contributors.
>  
> 5.      Why W3C?
> W3C is uniquely situated to make the Spotlight project work. W3C's ability to bring the major players together to support this project, and to lend credibility and technical insight are a key part of making the project successful. In addition, the value of the Spotlight Project's data will increase due to W3C's longevity as an institution on the Web, providing continuity that no one else can provide.
>  
> 6.      Long-term Evolution
> The Spotlight Project is intended to make a persistent set of performance data available for the long term. The need for continuity and a longitudinal view of Web performance across different browsers, geographies, and sites is central to the idea of making the data public and transparent. Inevitably the tests performed and data collected, and the associated methodology, would change over time. We should be prepared to continuously evolve along with the Web.
>  
> 7.      Making the Web Faster
> A big part of the Spotlight Project's mission would be educating the community about Web performance. Beyond publishing data about how popular sites are doing, we need to provide users with the means to make them better. This might include guidelines, articles, forums, and other means of helping the community make Web pages load faster.
>  
> 8.      Fellow Travelers
> A number of other people have had similar ideas and proposed similar projects. I'm indebted to them all. Multiple different aspects of the project proposed here have been implemented on webpagetest.org, HTTParchive.org,measurementlab.net, and stevesouders.com, among others. Akamai publishes a great report on the State of the Internet quarterly, and I'd like to propose something similar for this project as well. Recently, x509labs.com created a similar project for OCSP providers, publishing comparative tests in a very similar way to what's proposed here, and achieved remarkable results in reducing response times globally - no mean achievement.  This proposal intends to build on all of these and more, and with W3C's help, provide a spotlight on Web performance.
> 
>  
> 
>  

--
Mark Nottingham   http://www.mnot.net/

Received on Monday, 17 June 2013 17:47:57 UTC