RE: [Resource Timing] Unified Timing v3

Jatinder will be sending out an email shortly to solicit feedback on use-cases.  That will help with some of our discussions!

In the meantime, I can imagine a website that is only interested in their custom scenarios (that they've instrumented via marks/measures) and not all other PerformanceEntries.  Uploading just the measure times could be just a few bytes vs. uploading a full list of NavTiming, ResourceTiming, UserTiming, XYZTiming in their full, verbose JSON object format, could easily be several KB.  The getMarks() interface also "groups" same-named marks into a single entry, with a name:[timestamp array] association.  getEntriesByType() would return a new object for every mark timestamp.

For the circular buffer, I could see analytics scripts and dev toolbars, which come late to the party (after the page load is done), wanting to see the entire set of Performance Entries.  NavigiationTiming doesn't need to be forced into a buffer (as it's a singleton), and UserTiming is a pay-for-play model where the site can maintain the size of the non-circular buffer via clearMarks()/clearMeasures().  As long as we don't require all PerformanceEntries to live in the same buffer, they can still choose to implement a buffer (circular or not), similar to ResourceTiming.

I agree that sites may want to describe multi-phase operations.  Though it sounds like that might be one abstraction higher than measures, named for the sake of this discussion "scenarios", which have several sub-marks and measures.  Since we're providing the basic means of a high resolution timestamp (marks) and a standard way of describing phases (measures), I could see a JavaScript performance library providing the layer of abstraction for "scenarios".  You could have a JS "scenario" object that kept track of all of its related marks and measures.

What do others think about these proposals in general?  We haven't heard a lot of other feedback.

- Nic

From: public-web-perf-request@w3.org [mailto:public-web-perf-request@w3.org] On Behalf Of James Simonsen
Sent: Monday, June 20, 2011 4:07 PM
To: public-web-perf@w3.org
Subject: Re: [Resource Timing] Unified Timing v3

Nice work! I really like this a lot.

I see your point about marks and measures. Do we have a known case where we expect the developer to only uses marks or measures, but nothing else? If so, then this definitely seems worthwhile. If not, I think it'd be easier just to ping back the PerformanceEntry, since it can be easily processed along with the rest of the entries on the timeline.

My thinking with the circular buffer was that if the developer was really interested in something, they would grab it. And if they haven't grabbed it by the time the buffer was full, they probably didn't care. I guess this is mainly a question for long-running sites. Would they be more likely to wait for the buffer to be full and then pick out the ones they want? Or would they rather monitor for an important resource and then record it as soon as they see it?

I also kind of liked the idea of letting measures be multi-phase operations just like resources and the main document. Adding a bunch of marks to one measure seems desirable to me, so that you can group together all the steps in one place. Any thoughts on that one?

Thanks for coming up with this proposal!

James

On Thu, Jun 16, 2011 at 2:07 PM, Nic Jansma <Nic.Jansma@microsoft.com<mailto:Nic.Jansma@microsoft.com>> wrote:
As discussed on yesterday's conference call, we see three primary changes with the Unified Timing v3 proposal:


1.       A base object that all performance types inherit from.  The base object defines a few required attributes (eg., name, getDuration())

2.       A circular buffer model that all performance types must participate in

3.       A restricted API set that performance types must be queried through (getEntryByType(), getEntryByName())

We feel that #2 and #3 are significant changes to the behavioral models of the existing specs and may limit future development needs.  Jatinder has gone into some details of our concerns with these changes in his last email.

We like #1, and think that it alone could open up new possibilities for a more unified interface.  What if we incorporated the idea of a base object type into our existing Navigation Timing, Resource Timing and User Timing specs that all performance entries must inherit from?  Then, we add getEntryByType() and getEntryByName() to the API set we have today to allow for the "playbook" scenario, where you can easily query all performance types at once and place them on a single timeline.  We can remove any redundant methods that have been replaced by getEntryByType() and getEntryByName(), e.g., getResourceTimings().  Since they all inherit from the base class, they will all have a name, entryType, startTime, and duration associated with them that can be visualized.

If we do this, any future performance entry that wishes to participate in this model simply inherits from the base type and be exposed through the unified getter, and since they're not restricted by #2 or #3, they could also implement any additional specialized APIs that they need.

We think this integration meets the goals we have all discussed.

Here's what the proposed updated IDL for all of the interfaces might look like, with significant changes bolded:

interface Performance {
    // Unified API set
    const unsigned short PERF_NAVIGATION = 0;
    const unsigned short PERF_RESOURCE = 1;
    const unsigned short PERF_DEVELOPER_MARK = 2;
    const unsigned short PERF_DEVELOPER_MEASURE = 3;

    PerformanceEntryList getEntriesByType(in unsigned short entryType);
    PerformanceEntryList getEntriesByName(in DOMString name, in optional unsigned short entryType); // Added optional entryType

    // Navigation Timing
    readonly attribute PerformanceTiming timing;
    readonly attribute PerformanceNavigation navigation;

    // Resource Timing
    const unsigned short INITIATOR_OTHER = 0;
    const unsigned short INITIATOR_LINK = 1;
    const unsigned short INITIATOR_CSS = 2;
    const unsigned short INITIATOR_SCRIPT = 3;
    const unsigned short INITIATOR_IMAGE = 4;
    const unsigned short INITIATOR_OBJECT = 5;
    const unsigned short INITIATOR_SUBDOCUMENT = 6;
    const unsigned short INITIATOR_XMLHTTPREQUEST = 7;
    const unsigned short INITIATOR_EMBED = 8;
    const unsigned short INITIATOR_AUDIO = 9;
    const unsigned short INITIATOR_VIDEO = 10;
    const unsigned short INITIATOR_SVG = 11;

    void clearResourceTimings();
    PerformanceResourceTimingList getResourceTimings(); // is now getEntriesByType(PERF_RESOURCE)
    PerformanceResourceTimingList getResourceTimingsByURL(in DOMString url); // is now getEntriesByName(url, PERF_RESOURCE)
    void setResourceTimingBufferSize (in unsigned long maxSize);

    attribute Function onresourcetimingbufferfull;

    // User Timing
    const string MARK_FULLY_LOADED = "fullyLoaded";
    const string MARK_FULLY_VISIBLE = "fullyVisible";
    const string MARK_ABOVE_THE_FOLD = "aboveTheFold";
    const string MARK_TIME_TO_USER_ACTION = "timeToUserAction";

    void mark(in DOMString markName);
    Array getMarks(in optional DOMString markName); // see below*
    void clearMarks(in optional  DOMString markName);

    void measure(in DOMString measureName, in optional DOMString startMark, in optional DOMString endMark);
    Array getMeasures(in optional DOMString measureName); // see below*
    void clearMeasures(in optional DOMString measureName);
};

interface PerformanceEntry {
    readonly attribute DOMString name;
    readonly attribute unsigned long long startTime;
    readonly attribute unsigned short entryType; ; // A PERF_* constant
    readonly attribute unsigned long long duration; // may be 0
}

interface PerformanceEntryList {
    readonly attribute unsigned long length;
    PerformanceEntry item(in unsigned long index);
}

interface PerformanceNavigation { // does not inherit from PerformanceEntry
    const unsigned short TYPE_NAVIGATE = 0;
    const unsigned short TYPE_RELOAD = 1;
    const unsigned short TYPE_BACK_FORWARD = 2;
    const unsigned short TYPE_RESERVED = 255;
    readonly attribute unsigned short type;
    readonly attribute unsigned short redirectCount;
};

interface PerformanceTiming : PerformanceEntry {
    readonly attribute unsigned long long navigationStart;
    readonly attribute unsigned long long unloadEventStart;
    readonly attribute unsigned long long unloadEventEnd;
    readonly attribute unsigned long long redirectStart;
    readonly attribute unsigned long long redirectEnd;
    readonly attribute unsigned long long fetchStart;
    readonly attribute unsigned long long domainLookupStart;
    readonly attribute unsigned long long domainLookupEnd;
    readonly attribute unsigned long long connectStart;
    readonly attribute unsigned long long connectEnd;
    readonly attribute unsigned long long secureConnectionStart;
    readonly attribute unsigned long long requestStart;
    readonly attribute unsigned long long responseStart;
    readonly attribute unsigned long long responseEnd;
    readonly attribute unsigned long long domLoading;
    readonly attribute unsigned long long domInteractive;
    readonly attribute unsigned long long domContentLoadedEventStart;
    readonly attribute unsigned long long domContentLoadedEventEnd;
    readonly attribute unsigned long long domComplete;
    readonly attribute unsigned long long loadEventStart;
    readonly attribute unsigned long long loadEventEnd;

    // name: "document"
    // entryType: PERF_NAVIGATION
    // startTime: navigationStart (keep navigationStart as well since the attribute is available in UAs today)
    // duration: loadEventEnd - navigationStart
};

interface PerformanceMark : PerformanceEntry {
    // name: mark's name
    // entryType: PERF_DEVELOPER_MARK
    // startTime: mark time
    // duration: always 0
}

interface PerformanceMeasure : PerformanceEntry {
    // name = measure's name
    // entryType: PERF_DEVELOPER_MEASURE
    // startTime: measure's start time
    // duration: measure's duration
}

interface PerformanceResourceTiming : PerformanceEntry {
    readonly attribute unsigned short initator;

    readonly attribute unsigned long long redirectStart;
    readonly attribute unsigned long long redirectEnd;
    readonly attribute unsigned long long fetchStart;
    readonly attribute unsigned long long domainLookupStart;
    readonly attribute unsigned long long domainLookupEnd;
    readonly attribute unsigned long long connectStart;
    readonly attribute unsigned long long connectEnd;
    readonly attribute unsigned long long secureConnectionStart;
    readonly attribute unsigned long long requestStart;
    readonly attribute unsigned long long responseStart;
    readonly attribute unsigned long long responseEnd;

    // name: url renamed to 'name'. this is the resource URL
    // entryType: PERF_RESOURCE
    // startTime: resourceStart renamed to 'startTime'
    // duration: responseEnd - startTime
};

*Note:
Our thoughts were to keep getMarks() and getMeasures(), even though they can be satisfied by getEntriesByType().  They can provide a simpler return value than getEntriesByType():

mark("a");

mark("a");

mark("a");

mark("b");



getMarks();

// returns {a:[1,2,3],b:[4]}



getEntriesByType(performance.PERF_DEVELOPER_MARK);

// returns [{name:"a",startTime:1,entryType:2,duration:0},

//          {name:"a",startTime:2,entryType:2,duration:0},

//          {name:"a",startTime:3,entryType:2,duration:0},

//          {name:"b",startTime:4,entryType:2,duration:0}]

- Nic

From: public-web-perf-request@w3.org<mailto:public-web-perf-request@w3.org> [mailto:public-web-perf-request@w3.org<mailto:public-web-perf-request@w3.org>] On Behalf Of Jatinder Mann
Sent: Tuesday, June 14, 2011 7:09 PM
To: public-web-perf@w3.org<mailto:public-web-perf@w3.org>
Subject: RE: [Resource Timing] Unified Timing v3

Thank you James for putting the time into formulating this proposal. As per action items from the last conference call, we have been actively evaluating this proposal.

You are right that the current specification for Navigation, Resource and User Timings has a drawback that all APIs are exposed on window.performance. Your goal with the unified model is to attempt to give structure to the APIs and make them future proof - which are excellent goals. However, as we have evaluated this proposal, we have found some major drawbacks:s


1.      Generalized vs. specialized.

The proposal attempts to create a single interface that all Timings must use for data collection, storage and retrieval. At first glance, the unified interface looked like a very good idea. However, we are concerned that this single interface constrains current and future Timings to be generalized and risks losing, or making it difficult, to have access to functionality specific to that Timing.



For example, we lose the ability to do custom filtering of data querying. With the Unified Timing API, there is no way to do custom UA-based filtering of the data, which can do the filtering more efficiently than script, as we limit ourselves to just getEntriesByType() and getEntriesByName(). Let's consider a fictional future Widget Timings API. Imagine a future WidgetEntry that fires 1000s of times a second, yet one of the common ways of interacting with it is to only get Widgets of a specific "size".  Instead of having to return 1000s of objects back via getEntriesByType(), it would be nice to allow for getWidgetsBySize(500) or getWidgetsWithSizeBetween(50, 100). The generalized API has made this task difficult by requiring that the complete results be returned by getEntriesByType() by iterating over the entire set, at a loss of efficiency. Further, if the WidgetEntry only has "duration" and "size" attributes, but is logistically linked to a "parent" Widget and we don't put that "parent attribute" as part of its WidgetEntry, there would be no way to loop over all of the widgets to find children of a specific parent. The generalized API has made this task difficult. A specialized API would allow us to do something as simple as getWidgetsByParent(foo).



While this is just one fictional example we're portraying, we think a generalized interface will introduce these types of complexities and limitations.



2.       Single, circular buffer.

This model is heavily structured around a single circular buffer. However, this has a few downsides.



a.       Items stored in the buffer. Navigation Timing is a singleton and Developer Timing is a pay per use model; do either of these need to be stored in a buffer? What if Graphics Timing stores frames per second data; does this go in the same buffer?


b.       Circular buffer and fullness. The buffer is defined as a circular buffer, which is different from the capped/limited buffer discussed in Resource Timing.  The advantages of a circular buffer is that there is no need to monitor the "fullness" of it, but there are disadvantages.  Since the items aren't stable within the buffer, sequential querying of the interface can lead to inconsistent results if the developer isn't aware of the circular behavior.  Consider this naive way of using the interface:

var theFirstResource = getEntriesByType(PERF_RESOURCE)[0];
// the 151st resource is downloaded, and kicks the first entry out since it's a circular buffer
var theSecondResource = getEntriesByType(PERF_RESOURCE)[1]; // INCORRECT -- this is actually the 3rd downloaded resource as the buffer wrapped

On the other hand, with the capped buffer in Resource Timing, no external events modify the state or order of "current" items in the array, unless the user explicitly clears the buffer (clearResourceTimings()).



c.        clearBuffer() clears the entire buffer.  What if I'm not only interested clearing Marks and not Resource Timing or Navigation Timing?

d.       Default buffer size of 150 was designed for the estimated average number of resources on a future page. With Unified Timing, this buffer size requirements will limit memory growth of all types. Consider the case of Widgets Timing, where Widgets are added 1000 times every second.  While Widgets are not enabled by default per your suggestion, developers may want to enable them because they're very common.  The circular buffer will mean previous Navigation and Resource Timing data is lost immediately. This forces the user agent to either specify a large buffer size, which is a waste of memory, or the developer must increase the buffer size in the <head>, which is a poor performance practice, or the developer must let data be lost, which is a poor experience. None of these options are attractive.

3.      Additional concerns:

a.       Unified Timing's getEntriesByName() could have collisions amongst the various types.  For example, ResourceTiming is keyed by an URL.  If we add a future timing type that is keyed by URL, there would be a collision.  Developers querying the interface would have to be careful to check the returned timing type.

b.       Disabling new Timing types by default. Considering adding script to the <head> is a poor performance practice and analytics libraries will want timing enabled by default, disabling Timings by default is not the best approach.

c.        The "onentryadded" event isn't filtered by type, so if we have future high-frequency timing events, the event could introduce overhead if script is just looking for a specific subsets of events.  Dealing with the various timing types will require lots of "if typeof" statements.

d.       The Unified APIs make querying the data a bit more verbose. Consider querying the results of NavigationTiming, ResourceTiming and UserTiming:

performance.timing

performance.getResourceTiming()

performance.getMarks()


The Unified APIs is a bit longer to read:
performance.getEntriesByType(performance.PERF_NAVIGATION)[0];

performance.getEntriesByType(performance.PERF_RESOURCE);
performance.getEntriesByType(performance.PERF_DEVELOPER_MARK);

Since the Unified API might kick timings out of the buffer, just querying for navigationStart changes from:
var navStart = performance.timing.navigationStart;

           To:
              var navs = performance.getEntriesByType(performance.PERF_NAVIGATION);
              if (navs[0]) {
                     var navStart = navs[0].navigationStart;
              }

I think we need to step back and try to understand what kind of an API a web developer would find useful that allows for specialized methods and at the same time gives a consistent structure. For example, a hierarchy under the window.performance namespace may preserve both structure and specialization of the API. Eg., performance.navigation, performance.resource, performance.developer, performance.graphics would contain all APIs for Navigation Timing, Resource