Using a headless browser

Hi,

The current mobileOK checker uses a well-defined algorithm to determine
what to download; the algorithm is clear enough so that multiple
mobileOK implementations could reproduce it, but unfortunately, the said
algorithm only remotely represents what most mobile browsers do today.

To take two examples:
* the mobileOK fetching algorithms only loads style sheets marked with
the "handset" css media type (which most mobile browsers ignore), and
ignore other style sheets, in particular the one marked with "screen" or
with media queries (which most mobile browsers in fact do download)

* mobileOK download any background image mentioned in a CSS, whereas
browsers only load background images defined in a selector block that is
matched in the DOM of the downloaded page

While we could probably hack around to fix the first example (but this
wouldn't be trivial), the second one requires building a DOM and doing
CSS selector matching, which is well beyond what the current tool is
ready to deal with. And even with that, we wouldn't actually deal with
images that are downloaded after changes brought e.g. by JavaScript.

All in all, it seems pretty clear that the simplest way to find out what
resources are downloaded is to use a browser, since by definition they
have to deal with all this stuff.

Now, given the context in which we would want to use this (that is, in a
number of cases, on the server side), it would seem that a headless
browser (i.e. a browser without a UI, but that can be interacted via an
API) would fit the bill.

Back in January, I looked quickly into which existing headless browser
would work best for this. I played with PhantomJS [1] and Mephisto [2].

PhantomJS is webkit-based, and a fairly popular solution, but from my
early explorations, it has limitations that would make it difficult to
report a number of useful information: for instance, it's not possible
to determine the amount of transfered bytes for gzipped resources [3].

Mephisto works on top of Firefox that one controls via an HTTP API; in
the limited tests I did, I haven't hit an issue with it.

An open question (for both approaches) is how to ensure that these
headless browsers work as closely as possible to real mobile browsers:
for instance, one would want to be able to set the viewport dimension
(which would possibly be a parameter from the tool UI), maybe limit the
number of concurrent connections, etc. I seem to recall that setting the
viewport at least on Mephisto wouldn't be a problem (via user_pref.js).

(as an aside, the Browser Testing and Tools Working Group [4] is
defining an API that in theory could be used for interacting with
headless browsers — we should probably keep track of this)

Comments and feedback welcome!

Dom

1. http://www.phantomjs.org/
2. http://temesis.github.com/Mephisto/index.html
3. http://code.google.com/p/phantomjs/issues/detail?id=156
4. http://www.w3.org/testing/browser/

Received on Friday, 30 March 2012 15:25:11 UTC