[W3C] BPWG CTG Guidelines / HTTP field user-agent issues

I posted the following message in the WMLProgramming mailing list.
People have suggested that I publish it as a formal comment to the CTG
draft, so here it is, under the heading "Allowing modifications of the
HTTP header field user-agent: rationale missing".

Eduardo Casais
areppim AG
Bern Switzerland
------------------

I would like to review (a last time) an issue that reoccurs in all
discussions about transcoders.

> Changing User Agent or other headers is not prohibited by HTTP

The first thing to stress is that the user-agent is essential to drive
content selection and generation processes, both in the mobile Web and
in the desktop Web. 

a) In the mobile Web, the user-agent is directly associated to the
actual device, and hence serves as a key to characteristics such as
screen dimensions, preferred content types, etc. The advent of
uaprof/ccpp was supposed to make this mapping unncessary, but it is
not the case: uaprof descriptions are often missing, point to invalid
URL, omit important information, or are just plain unreliable. Device
databases like WURFL, based on user-agent mappings, thus remain
indispensible.
b) In the desktop (non-mobile) Web, developers have long relied upon
the user-agent to identify the browsers issuing requests in order to
tailor content to their "quirks". This has been going on at least
since the times of the Netscape / IE wars.

Let us now examine the use cases of a mobile Web browser accessing the
Internet, and evaluate the relevance of the user-agent -- assuming
that transcoders systematically substitue the original value with a
new one.

1.a User-agent-switcher.
	The Web server is able, based on the user-agent, to
	provide a mobile-optimized or a full-web service.
	It therefore needs the original user-agent; modifying
	it is unhelpful.
2.a Mobile Web only.
	2.a.1 Generic content.
	The server returns generic mobile content, without
	customizing it for any specific user-agent.
	This kind of applications is rare, and often
	corresponds to surviving examples of text-only
	services developed for older PDA and WAP 1 devices.
	Since the server does not use the original user
	agent, modifying it is useless.
	2.a.2 Mobile with default.
	The server returns mobile-optimized content. When
	not recognizing the user-agent, it returns a default,
	best-effort representation, perhaps with a message
	suggesting that the content is tailored for mobile
	devices.
	Since the server relies upon the user-agent, and is
	able to return a default representation, modifying it
	is unhelpful.
	2.a.3 No default.
	The server returns mobile-optimized content, but will
	return an error (page with "unsupported browser" warning
	or return code "request not acceptable") whenever it
	does not recognize the user-agent. In this case, modifying
	the original user-agent is most unhelpful, as it guarantees 
	that the server will not recognize it as a valid
	mobile one. If the server does not, for whatever reason
	(e.g. incomplete device database), recognize a mobile
	user-agent, then there might be a case for modifying it
	towards an acceptable mobile one -- but transcoders 
	precisely do the reverse: they change a mobile user-agent
	to an exotic full-Web one. Hence, a modification of the 
	original user-agent is unhelpful all situations.
3.a Full Web only.
	3.a.1 Generic content.
	The Web server serves generic full-Web content, without
	looking at the user-agent. In this case, modification of
	the original user-agent is useless.
	3.a.2 Tailored full-web with default.
	The server returns full-web content customized for specific
	full-web user-agents (e.g. IE 6.0, IE 7.0 and Firefox 2.0),
	and serves a default representation, perhaps with a warning
	("This site is best viewed with the following browsers:...")
	for other user-agents. In this case, modification of the
	original user-agent is either useless (in any case a default
	representation will be returned), or unhelpful (the default
	representation is probably better downgradable than one
	specifically customized for a very specific full-Web browser).
	3.a.3 Tailored with no default.
	The server returns content tailored for specific full-Web
	browsers, and an error for other unrecognized or unsupported
	user-agents.
	Here there is a case to substitute the original user-agent to
	force retrieval of content. However, this works only if the
	"fake" user-agent precisely corresponds to one of those
	accepted by the server -- but transcoders do not tailor their
	substitute user-agents with respect to the application server:
	the include only general hints (like Mozilla/x.y) in the hope
	this is enough to determine content generation.
	Hence, the generic substitution of user-agents performed by
	transcoders is not appropriate here.

Conclusion: in two cases, modification of the user-agent is useless,
in three it is detrimental, in one it is either useless or
detrimental, and in one it could be helpful, but it is currently done
inappropriately.

Let us consider the interesting symetric use-cases: a full-Web mobile
device accessing the Internet.

1.b User-Agent switcher
	Following the same reasoning as in 1.a, we find that
        modification of the original user-agent is unhelpful.
2.b Mobile Web only.
	2.b.1 Generic content.
	Whatever user-agent, the server returns generic mobile-optimized
	content. A modification of the original user-agent is useless.
	2.b.2 Tailored with default.
	The server returns mobile-optimized content, and a default
	representation for unrecognized user-agent. Modifying it is
	therefore useless -- the default representation will be returned
	whether the original (full-Web) or the substitute (pseudo-full
	Web) agent appears in the request.
	2.b.3 Tailored, no default.
	Following the same reasoning as in 2.a.3, the substitution of 
	original (full-Web) user-agent by a fake (full-Web) one is 
	useless, as it will anyway return an error.
3.b Full Web only
	3.b.1 Generic content.
	If the server returns generic full-Web content whatever the user
	agent, then modifications of the user-agent are useless.
	3.b.2 Tailored with default.
	Following the reasoning in 3.a.2, modifying the original
        full-Web user agent is either unhelpful (because the server
        could have recognized the mobile device's agent), or useless
        (the same default representation would be returned).
	3.b.3 Tailored, no default.
	Here the server may recognize the full-Web user-agent of the
        mobile device; it is therefore unhelpful to modify it. Or it
        might not support that specific user-agent, in which case it
        would be sensible to substitute one that is effectively
        supported by the server; however, this is not what transcoders
        do: they provide a generic, not a real user-agent instead --
        this is inappropriate.

So one situation where it is detrimental, four where it is useless,
one  where it is either useless or detrimental, and one where it is
either useless or inappropriately done. It is also an acid test: do
transcoders modify requests from full-Web capable mobile browsers? If
so, something seriously weird is going on, as the excuse has generally
been to make full-Web content available to non-full-Web capable devices.

>From this examination, one can only conclude that proponents of the
preservation of the original user-agent do not have to justify their
position and established practice. Rather, the onus is on the
proponents of the substitution of the user-agent to argue in favour of
their approach, which disrupts established practice. There is
basically only one use case where changing the mobile user agent to a
desktop user agent might help, but it remains to demonstrate:

a) The relevance of the scenario. Perhaps people at Google could let
one of their crawlers roam over a few tens of thousands of WWW sites
to gather statistics on the relative frequency of each aforementioned
scenario.
b) The benefits resulting from handling that specific scenario.
c) That (a) and (b), taken together, are so overwhelming that they
more than compensate the disruptions caused in all other use-cases.

If another use case outside the framework I have presented here pops
up, this does not reduce the need for an assessment based on (a), (b),
(c). 

As a final remark, I would like to note that transcoders have been
operating in the mobile Web for a long time. It started with
adaptation of HTML for PDA (Web clipping) and HTML to HDML conversion,
continued with HTML to WML, before arriving at the current crop of
content adaptation. In the old times, developers of content adaptation
software were wary of modifying the user-agent: turning generic WWW
content into a form suitable for mobile devices is so fraught with
difficulties that one would take every chance to let a server return
mobile optimized content (based on the user-agent) if it could. It is
only fairly recently that, without much justification, transcoders
have started in a
systematic way to overwrite the user-agent field.

I think I have said everything I wanted regarding the CTG. The
document  requires quite some rework -- nothing exceptional, since it
is a draft. I will lean back and wait for the results of this round of
revisions. Till then, readers of the WMLprogramming and W3C BPWG lists
can rejoice in the knowledge that my long-winded posts are abating at
last.


E. Casais

Received on Tuesday, 12 August 2008 10:31:59 UTC