Re: Offline Web Applications status

Hi again,

I have in mind several extensions to the ApplicationCache that I think could
address some of the additional desirements from the web developement
community. I'll post them here because people seem to be more willing to
have a discussion on the topic here than over in whatwg.

1. Allow cross-origin HTTPS resources to be included in manifest files. This
much is actually done already in chromium impl as described on the whatwg
list.

2. Allow a syntax to associate a page with an application cache, but does
not add that page to the cache. A common feature request also mentioned on
the whatwg list, but it's not getting any engagement from other browser
vendors or the spec writer (which is kind of frustrating). The premise is to
allow pages vended from a server to take advantage of the resources in an
application cache when loading subresources. A perfectly reasonable
request, <http useManifest='x'>.

3. Introduce a new manifest file section to INTERCEPT requests into a prefix
matched url namespace and satisfy them with a cached resource. The resulting
page would be free to interpret the location url and act accordingly based
on the path and query elements beyond the prefix matched url string. This
section would be similar to the FALLBACK section in that prefix matching is
involved, but different in that instead of being used only in the case of a
network/server error, the cached INTERCEPT resource would be used
immediately w/o first going to the server.
   INTERCEPT:
   urlprefix redirect newlocationurl
   urlprefix return cachedresourceurl

Here's where the INTERCEPT namespace could fit into the changes to the
network model.
    if (url is EXPLICITLY_CACHED)  // exact match
      return cached_response;
    if (url is in NETWORK namespace) // prefix match
      return network_response_as_usual;
    if (url is in INTERCEPT namespace) // prefix match <---- this is the new
section ----
      return handle_intercepted_request_accordingly
    if (url is in FALLBACK namespace) // prefix match
      return
repsonse_as_usual_unless_fallback_conditions_are_met_by_that_response;
    otherwise
      return synthesized_error_response;

4. Allow an INTERCEPT cached resources to be "executable". Instead of simply
returning the cached resource in response to the request, load it into a
background worker context (if not already loaded) and invoke a function in
that context to asynchronously compute response headers and body based on
the request headers (including cookie) and body. The background worker would
have access to various local storage facilities (fileSystem, indexed/sqlDBs)
as well as the ability to make network requests via XHR.
   INTERCEPT:
   urlprefix execute cachedexecutableresourceurl

5. Create a syntax to allow FALLBACK resources to be similarly executable in
a background worker context.

As an author of gears and chromium's appcache, I've been interested in
pursuing something along these lines for a while. Chrome is finally getting
built up to the point where this could be done, the file system and better
support for many worker threads are prerequisite.

With the outline above in mind, this is why earlier in the thread i
questioned the need for an explicit add(url) and remove(url) sort of api.
That capability can be composed out of the file system and an executable
intercept handler. And there is also window.createObjectUrl() and
FileEntry.toUrl() which could cover many use cases that are behind the
feature request for add() and remove().

-Michael


On Sat, Mar 26, 2011 at 8:32 PM, Jack Coulter <jscinoz@jscinoz.org> wrote:

> IndexedDB would be more suited to what you're doing Nathan, I've always
> seen
> ApplicationCache as something to only use on the core HTML/JS/CSS and
> perhaps
> small images, like icons (none of this would change often, and would
> generally
> be rather small) whereas IndexedDB sounds more like what you did with
> gears.
>
> Or, there's always the File System API, but I don't think it's widely
> implemented yet.
>
> Excerpts from Nathan Kitchen's message of Sun Mar 27 08:46:35 +1100 2011:
> > A couple of other app cache observations from a hobbyist who's played
> around
> > with Google's Gears...
> >
> > I built an offline web application based on Gears, with the intention to
> > migrate to something a bit more standardized as it became available. That
> > was a good two years ago now, but the existing and proposed
> implementations
> > still don't offer the capability that I can get from Gears.
> >
> > If you're in Chrome, point your browser at: http://my.offlinebible.com/.
> My
> > observations are going to be related to this app. I apologise if they're
> > quite specific, but they present a snapshot of some real-world issues
> > encountered while developing an offline app.
> >
> >    - *App Cache*
> >    The first task that the app undertakes is downloading a large amount
> of
> >    offline data. This data includes the usual web page resources, plus a
> >    massive amount of data destined for the Gears/WebSQL database.
> >
> >    Ideally, I want to keep this in a single easy installation process.
> With
> >    Gears I can do this entirely from JavaScript. First, the database is
> set up.
> >    Then, AJAX is used to request data in chunks (JSON, about 60Mb,
> gzipped for
> >    bandwidth gets it down to 10Mb iirc). Once it's all requested, it gets
> >    dropped into Gears/WebSQL. Finally all the usual web assets (html,
> css, js,
> >    img) are cached. Throw up a progress bar, run everything in a web
> worker to
> >    keep the UI from freezing, done.
> >
> >    With the existing specs, I'd have to do something slightly different.
> >    First, I'd have to hit a page with a manifest, and hook the relevant
> events
> >    for when the cache was ready. If I want to offer the user different
> offline
> >    capabilities (i.e. customize what's cached), afaik I can't do that
> from
> >    JavaScript. It requires some server-side code/processing to output a
> >    different manifest.
> >
> >    Once the cache was ready, I could carry on with the existing
> >    installation. However, I don't (think I) have the script-side control
> over
> >    adding and removing items from the cache to customize the user's
> offline
> >    experience.
> >
> >    - *Search
> >    *This is the use-case that's most important to me. I want to be able
> to
> >    search all the data which I took offline. My current implementation is
> built
> >    using manually indexed items and joins. In theory I could use the
> full-text
> >    capabilities of the underlying SqlLite Gears implementation, but this
> was a
> >    step too proprietary for me. I built all the data indexes to see what
> >    performance was like. Throw some words into the search capability,
> you'll
> >    see how long it takes to search. It's *fairly* quick but there's a
> slight
> >    lag (which locks the UI, it's synchronous ATM).
> >
> >    I know full-text indexing is on the cards for IndexedDB. I'd love to
> see
> >    some sample implementations of full-text to compare speed against the
> manual
> >    index. For single words there might not be too much difference, but
> for more
> >    complex multi-word or pattern-matching, the manual index is too
> >    slow/won't work.
> >
> > I don't think that my scenario is particularly unusual. Taking a large
> > amount of data offline and making it available to search seems like a
> pretty
> > common use-case. To support this, there are three capabilities which I'd
> > like to see:
> >
> >    - Script access to add or remove items from the application cache -
> >    document.manifest.add("");
> >    - Batch operations (or support for adding a lot of similar data as
> >    quickly as possible - this takes ages if you add each record as a
> single
> >    transaction)
> >    - Full-text search on data
> >
> > I'm looking forward to this coming together eventually, might be worth an
> > IndexedDB implementation soon : )
> >
> > On 24 March 2011 05:53, David John Burrowes
> > <bainong@davidjohnburrowes.com>wrote:
> >
> > > 2011/3/24 louis-rémi BABE <lrbabe@gmail.com>
> > >
> > >> ## Maybe Web devs don't use App Cache because they don't understand
> > >> what it is... ##
> > >>
> > >
> > > I think most webdevs are expecting more than what is offered. It seems
> like
> > > a half baked solution to a potentially useful requirement.
> > >
> > >
> > > I thought I'd add half a cent here, from the perspective of one who
> isn't a
> > > professional web developer... just a hobbyist.
> > >
> > > When I heard about the app cache, it seemed like a really great thing.
> > > Offline web apps! Cool! A way for the web to become even more
> ubiquitous!
> > >
> > > But, as the comment above hints, it really doesn't seem to be the full
> > > delivery of the solution (even when you get past the browser
> differences,
> > > setting up of mime types, debugging all this, etc).  An offline web app
> is
> > > certainly more than just caching the code and ui files, no?  It is also
> some
> > > kind of stand-in for the absent server... data storage, and cross-page
> state
> > > of some sort (e.g. I'd expected something like web workers that can
> live for
> > > a session, not a page).  These aren't all coming together at the same
> time,
> > > and aren't really being presented as a unified "feature" (indeed, I'm
> not
> > > sure that they are being thought of as that)
> > >
> > > I'm sure that as html continues it's forward evolution, these will all
> come
> > > into play and we'll eventually see more use of the feature.
> > >
> > > David
> > >
> > >
> > >
>