Re: Keynote Systems tools for researchers

From: Henrik Frystyk Nielsen (
Date: Tue, Dec 01 1998

Message-Id: <>
Date: Tue, 01 Dec 1998 21:28:37 -0500
To: "Eric Siegel" <>
From: Henrik Frystyk Nielsen <>
Subject: Re: Keynote Systems tools for researchers

Oups - wrong subject - please try this one instead!


Thanks Eric - I think this is an excellent way to start the discussion
about which tools are available and how we can make them play together in
order to obtain the web characterization information requested by the Web

What other tools are out there and what are their status/usability for the
Web Characteriztaion Activity?



>W3C / WCA members:
>Keynote Systems management has agreed to offer data and tools to WCA
>researchers without charge, assuming that nothing breaks and the workload
>isn't overwhelming!  We hope that this is useful, and we look forward to
>doing more work with W3C.
>Contact me by email or by phone if you want any of this data or access to
>the tools. Also remember to contact us with any ideas about ways to measure
>overall web performance and about ideas on placement of measurement agents.
>-- Eric Siegel (
>=====> Example: Keynote Systems database queries for W3C:
>Keynote stores approximately six million web page download measurements per
>day, from more than 60 agents wordwide (approximately 2/3 in the US).  Most
>of the measured web servers are in the US; some are outside. Downloads occur
>very 15 minutes or every hour; approximately 50% of these six million
>downloads also include detailed timings of the download phases, including
>the DNS lookup phase, time to establish TCP connect, first packet received,
>last byte of index page received, etc. Errors detected in downloads are also
>recorded. Six weeks of data are maintained on-line in an SQL (Oracle)
>The presentations can be broken out by:
>* Time Interval (e.g., between 9am and 8pm EST M-F; or for an entire week or
>* Geographic distribution of measurement agents (e.g., for the 43 US agents;
>for the 18 outside-US agents; for particular agents or sets of agents as
>listed in our web page
>* Other breakouts are possible.
>Repetitive data summaries are possible; e.g., we could email you a data
>summary every week.
>Sample uses of this data:
>*** DNS Success and Performance:  Whenever a web page download aborts with a
>DNS failure, that's recorded in the database. DNS lookup performance is also
>recorded for approximately 50% of the downloads.
>The obvious DNS statistics we can produce from this database are:
>* Number of failed DNS queries and total number of queries, and
>* Distribution of DNS query timings, giving mean and median, for the subset
>that has such data.
>*** TCP Connect Time:  We could produce the time needed for a TCP Connect
>(three-way handshake) using the same raw data as are used for the DNS
>statistics. Some caching researchers may be interested in this data.
>=====> Example: Traceroute assistance to W3C:
>We can let researchers use Traceroute, Ping, and nslookup facilities from
>Keynote's remote agents worldwide if the total number of requests per agent
>per minute is reasonable (e.g., one request or fewer per minute).  One of
>the problems is that our agents permit only one traceroute, ping, or
>nslookup to be active at a time; therefore, our customers could be blocked
>if these facilities are used intensively.
>The request for Traceroute, Ping, or nslookup is made from a web page, and
>the data is returned in HTML.
>Eric D. Siegel
>Keynote Systems, Inc., Two West Fifth Ave., San Mateo CA 94402
>  Phone: (650) 524-3028  Fax: (650) 524-3099
>Home email:  Home Phone: (650) 570-2266
Henrik Frystyk Nielsen,
World Wide Web Consortium