Policy Issues, Allow / Disallow lists and Bogus 200 responses (ISSUE-261) (was Re: Comment on latest CT Guideline Document)

Let's take a step back here, and please allow me to express some 
thoughts as a preamble to coming back around to the point about 
allow/disallow listing. Forgive me this is necessarily rather lengthy.

The extent to which the document should stray into policy

 From my point of view, the point of the document at all is to provide a 
reasonable balance between the needs of Content Providers to serve the 
user experience they choose to their users, and the needs of operators 
of transforming proxies to assist their customers to perceive that 
portion of the Web that would provide them with an unsatisfactory user 
experience without assistance.

We stray near the edges of policy when we say that our job, as the 
Mobile Web Best Practices Working Group, is to foster an environment in 
which an increasing portion of the Web does naturally provide at least a 
functional user experience, and preferably a harmonized user experience 
(DI Glossary Terms) for users who have a mobile context. This is based 
on a view, with a reasonable consensus behind it, that says that almost 
no matter how good the mobile equipment gets, the users' needs in the 
mobile context are generally distinct from their needs in a desktop or 
indeed other identifiable contexts.

We do not enter into policy or legal issues such as whether a mobile ISP 
has a specifically different relationship with their subscribers than a 
fixed line ISP; what the copyright implications are of altering a work 
in an unlicensed manner;  whether it is acceptable to remove or replace 
advertising and so on.

User Choice

However, as SeanP reminds us, we need to address the issue of User 
Choice of representation, but we don't discuss what the reasonable 
limits of user choice are, especially in respect of the policy issues 
noted above.

In order to accommodate user choice, we stray near the edges of policy 
when we say, for the sake of inter-operability and of promoting the 
creation of  mobile user experiences, that the Content Providers' wishes 
as to their content normally prevail, with the exception that if the 
user MAY specifically request over-riding of content provider preferences.

We don't advocate this user over-ride (we say that CT providers MAY 
rather than SHOULD allow this), and possibly we should be a little more 
careful in the document to make sure that we make it clear that we don't 
advocate this, as it may turn out that we are straying into policy 
areas, such as the ones above, which might cause those that followed 
them, in some jurisdictions, to fall foul of the law. I don't know how 
realistic a concern that is, but it is a concern.

We do believe in User Choice. We have previously expressed the view that 
the best place to implement that choice is at the origin server. I think 
that the Best Practices statements should make it clearer than they do 
that the Content Provider SHOULD allow the user to choose their 
presentation, where there is a choice. The relevant text is now at [1] 
which is a watered down version of text that was originally at 
[CAPABILITIES] (iirc). There's scope for a stronger statement of this as 
specific Best Practice in BP-2.

[1] http://www.w3.org/TR/mobile-bp/#iddiv2126652328

Where the Content Provider offers this option it seems clear to me that 
the Proxy should step back. However, once again I'm not sure that we 
have a clear way of helping the Content Provider inform a CT proxy that 
it is doing that. Nonetheless, I think it worth mentioning in the CT 

It seems to me that it would be inconsistent with the foregoing to say 
that the user's choice as expressed to a content transformation proxy 
can either be blanket "always give a transformed version of a desktop 
representation" or that a users preference for any site defaults to that.

I would like to make that clear in the document, at the moment "Always 
request desktop presentation" is mentioned specifically as an example of 
an expression of user preference (under 3.2.1).

Section 4.1.2

In that section my understanding of what we are trying to do is 
identify that the request headers should not be altered unless:

a) the user would be prohibited from accessing content as a result of a 
406 or bogus 200 response;
b) the user has specifically requested a restructured desktop experience;
c) the request is part of a session of some sort and either it is 
technically infeasible not to adjust the request because of earlier 
interaction, or because doing so preserves a consistency of user experience.

We don't say, and I think we should, that when we say "specifically 
requested" we mean that they have expressed a preference either as a 
result of having perceived the response and don't have the option of 
asking the content provider to switch presentations, or because they are 
aware that the original request has been rejected - though I would agree 
that we should careful about usability in saying so.

At the beginning of this section we say  that in deciding whether to 
alter the request headers it should take into account:

- user preferences
(this would appear to be redundant with the above),

- any knowledge of user agent capabilties
(this would appear to be irrelevant, and this clause should be moved, if 
anywhere, to section 4.4);

- any a priori knowledge of server behavior - derived from previous 
interaction with the server or to preserve session consistency
(I think this clause has lost its way.
-- The session consistency aspect is redundant with the above,
-- the a priori knowledge based on previous interaction implies that the 
server somehow "knows" from previous interactions that the server 
belongs in an "transform list". I don't think this is right, per Rotan's 
comment, an increasing number of sites do offer mobile presentation and 
there needs to be a way of the proxy becoming aware of that. In other 
words, our policy assumption is that transforming proxies should not 
assume that things remain the same, they should make sure that they are 
not thwarting Content Providers intentions. Especially when those 
intentions are to try to conform to Mobile Web Best Practice.
-- we removed from this clause the idea that the proxy could determine 
the server's preferences from a repository of preferences, and although 
in some sense going to retrieve a POWDER document is "a previous 
interaction with the server" I think we should probably put some forward 
looking statement in here to allow a simple addendum to be published 
which allows that to be inserted simply.

- the HTTP method of the request
(that's dealt with elsewhere in the section and is redundant)

So where does that leave us on bogus 200 responses?

In essence, the current wording in section 4.1.2 describes the building 
of a "transform list" which it suggests can be used in place of tasting 
content with the original headers. It does not refer to statically 
composed lists, yet on the other hand 3.2.3 specifically does, but then 
that section goes on to say that allow and disallow lists are a BAD 
THING [1], because of the inherent difficulty in maintaining them.

So that's a mess.

Like Rotan, I agree that the nature of the Mobile Web is that it is 
growing and evolving and that asking CPs to interact with allow and 
disallow lists via administrative means is impractical and undesirable 
for the reasons I have stated before and as elaborated eloquently by Rotan.

I think that we should keep a statement in the document noting the 
inherent intractability from the content providers viewpoint of 
requiring them to adjust their status in any such list "by 
administrative means".

I don't think that the document should in any way prescribe the internal 
behavior of transforming proxies. However I think it needs to take 
account of the fact that some proxies might implement allow lists, and 
equally that others might implement advanced heuristics to determine 
whether a 200 response is bogus or not - or indeed some combination of 
the two or other techniques not discussed.

So in taking account of the possibility that proxies have implemented 
such lists, how does a Content Provider ensure that their 200 response 
is known to be a "good" 200 response?

I think the answer might lie in examining the server response for Vary, 
  <link> element (or header) and no-transform as discussed under 4.2.

The logic of 4.1.2 would then be:

Request resource with original headers
- If the response is a 406 response, re-request with altered headers 
(unless the 406 response contains no-transform).
- If the response is a 200 response,
--	if the response contains vary: User-Agent, an appropriate link 
element or header, or no-transform, forward it
--	otherwise assess (by unspecified means) whether the 200 response is a 
bogus one
---		if it is not, forward it
---		if it is, re-request with altered headers

For subsequent requests which form part of the "same session"
- take into account user preferences (unless the server specifies 

(note that this has a dependency on the meaning of the link element, 
which afaik, we are still unsure about).

I think we should replace the text of 3.2.3 with a statement that the 
guidelines have been designed in a way that allows certain features to 
be implemented by allow and disallow lists, that the guidelines provide 
mechanisms for such lists to be "self healing" and that proxies should 
take note of such mechanisms and not rely on administrative behavior on 
the part of CPs to change their status.

I know this has all been a bit lengthy, but hopefully it moves the 
discussion along a bit.


[1] http://en.wikipedia.org/wiki/1066_and_All_That

On 13/06/2008 09:25, Rotan Hanrahan wrote:
> I have sympathy with the position of operators, and understand why some would implement a list-based filter as the most easy solution to their problem. As a supplier of technology to content/service providers who are transitioning from non-mobile to mobile-friendly, it is a concern to me that lists are being used. Even the availability of a form that developers can use to tell an operator of a change in the behaviour of a site is far from ideal. It would mean that a content/service provider would have to know where every form was, and that form would have to be in a language they could understand. Indeed, I expect that if the practice goes any further, our customers will be demanding that we (MobileAware) start filling in these forms on behalf of our customers. I don't find that any more appealing.
> It would be nice if operators that are using lists would take measures to automatically update those lists. Semantic mechanisms may not be widespread yet, but even a simple daily test of sites on the operator's "non-mobile" list would be easy to implement, and have little negative effect. Just keep cycling through the sites on the list, and use a signature of a mobile browser. If the response becomes positively disposed to the mobile device, remove the site from the list.
> It's not ideal, but if adopted by those operators who insist on using lists (regardless of whether or not the practice is endorsed/shunned by standards bodies) it would certainly alleviate some of the issues faced by content/service providers.
> The aim, however, should be to move in a direction that completely eliminates the need for lists. It is a practice, necessary in certain cases, but I'd hardly describe it as a "best" practice.
> ---Rotan.
> -----Original Message-----
> From: public-bpwg-ct-request@w3.org [mailto:public-bpwg-ct-request@w3.org] On Behalf Of Gerlach, Heiko, VF-Group
> Sent: 13 June 2008 08:27
> To: Sullivan, Bryan; Jo Rabin
> Cc: public-bpwg-ct@w3.org
> Subject: RE: Comment on latest CT Guideline Document
> Hi Jo,
> I 100% share the view of Bryan. For us as an Operator this feature is quite important as long as there is no option to overcome that situation for sites where we do not have any stake holder available.
> Cheers
> Heiko 
> -----Original Message-----
> From: Sullivan, Bryan [mailto:BS3131@att.com] 
> Sent: Freitag, 13. Juni 2008 01:33
> To: Jo Rabin
> Cc: Gerlach, Heiko, VF-Group; public-bpwg-ct@w3.org
> Subject: RE: Comment on latest CT Guideline Document
> Jo,
> I don't understand the reluctance to clarify this CT proxy behavior, which will be a configuration feature of CT proxies, at least as AT&T would deploy (and our needs are not so different from other other network operators). Regardless of whether the CT guidelines are silent on it, or specifically forbit it, CT proxy vendors will deploy the features that their customers require. It would be better if W3C would acknowledge the reality of business practice especially given the limitations of semantic methods, but either way we will deploy what we have to, and not be concerned whether this is viewed as a noncompliance to the W3C recommendations. Yet I still hold out hope that the consensus in the group will acknowledge this as a pragmatic solution to the likely lack or non-support of the semantic methods.
> Best regards,
> Bryan Sullivan | AT&T
> -----Original Message-----
> From: Jo Rabin [mailto:jrabin@mtld.mobi]
> Sent: Thursday, June 12, 2008 5:39 AM
> To: Sullivan, Bryan
> Cc: Gerlach, Heiko, VF-Group; public-bpwg-ct@w3.org
> Subject: Re: Comment on latest CT Guideline Document
> Bryan
> Respectfully I have to disagree strongly with this, except in the respect that if we had the ability for a Web site owner to declare their preferences in a single place, then this would be the right way forward.
> Once we have shipped this version of the document hopefully it will be the right time to go ahead and address this as the next high priority task.
> Jo
> On 12/06/2008 12:35, Sullivan, Bryan wrote:
>> Hi Jo,
>> Even if scalability of the manual configuration requirements was an issue for CP's (I'm not sure it is, given the limited number of Operators that serve regions, and the typical relationships that already exist between Operators and CP's), there are are readily deployable means to automate such a process. I could fairly quickly build a web-enabled content management database application that initiates the necessary changes using whatever web form a particular Operator has defined. I'm sure this would not be a significant challenge for CP's and could even be an application provided by the Operator specifically for that purpose, to be used by the CP to manage the CT preferences for their site. 
>> *If* CT proxy deployment becomes a common reality across many Operators (or 3rd-party CT service providers), and realtime/semantic methods for CT proxy control are undefined or not supported by many CP's (leaving them with no other option but using pre-configuration), the standards community could work on standardizing such CT proxy allow/deny list management methods. But I would recommend that we see how the market develops first.
>> Best regards,
>> Bryan Sullivan | AT&T
>> -----Original Message-----
>> From: Jo Rabin [mailto:jrabin@mtld.mobi]
>> Sent: Wednesday, June 11, 2008 8:44 AM
>> To: Sullivan, Bryan
>> Cc: Gerlach, Heiko, VF-Group; public-bpwg-ct@w3.org
>> Subject: Re: Comment on latest CT Guideline Document
>> Bryan
>> The idea that a CP has to be aware of and fill in a simple Web form on every operator's site is exactly the reason that this is bad practice. 
>> It's simply not reasonable or practical from the CP perspective. That's why it's noted in the way it is with the reasons included.
>> Cheers
>> Jo
>> On 11/06/2008 16:36, Sullivan, Bryan wrote:
>>> Heiko: "allow/deny" are intended as better terminology alternatives 
>>> to "white/black", e.g. more clear as to the intent of the designation
>>> Jo: the point of including these clarifications is that in some cases the CT proxy will not be able to reliably depend upon semantic means of discovering which sites should and should not be transformed (and how). An allow/deny list is a practical way of supplementing the decision process, and a common practice by WAP proxy/gateway vendors for example. There is certainly a process of list management, that is typically outside standards scope, and the CT operator is responsible for the simplicity of the process (a business decision in the end). It does not have to involve "great difficulty" (a simple web form on a CT proxy operator's developer/CP support site is all that is needed). If the CT recommendations do not include this as an option, it must certainly at least *not* prohibit it: vendors must have the freedom to meet customer (CT proxy operator) requirements without being called "non-compliant", if compliance to W3C recommendations is considered important.
>>> Best regards,
>>> Bryan Sullivan | AT&T
>>> -----Original Message-----
>>> From: public-bpwg-ct-request@w3.org
>>> [mailto:public-bpwg-ct-request@w3.org] On Behalf Of Jo Rabin
>>> Sent: Wednesday, June 11, 2008 8:19 AM
>>> To: Gerlach, Heiko, VF-Group
>>> Cc: public-bpwg-ct@w3.org
>>> Subject: Re: Comment on latest CT Guideline Document
>>> I think we have a terminology problem here, I assume that what you're saying is that an "allow list" (i.e. a list of sites for which transformation is allowed) should be recommended practice? I don't think I agree, anyway, in that it means that the CP then has great difficulty in getting changes to the behaviour of their site noticed.
>>> Jo
>>> On 11/06/2008 13:45, Gerlach, Heiko, VF-Group wrote:
>>>> Hi All,
>>>> Page 7,
>>>> 1) Does Allow and Disallow lists mean Black and White lists?
>>>> 2) If so, I strongly recommend to support a white lists within the 
>>>> CT Guidelines. We do not know owner/stakeholder of most of the 
>>>> websites which are "non made for mobile". So we can not expect their support.
>>>> But we can understand the customer needs and a customer could tell 
>>>> us that content adaptation failed e.g. due to 200ok instead of 406.
>>>> This is why I think we do need a white list. For Urls contained in 
>>>> that whitelist, Content adaptation shall be done in that way that a 
>>>> Mozilla User Agent (non mobile user agent) is setup instead of the 
>>>> mobile user agent.
>>>> This helps us to manage with the 406/200OK issue without buyin from 
>>>> the site owner.
>>>> Clarification:
>>>> I agree that blacklists should be ommitted. But white lists will 
>>>> deliver a clear advantage.
>>>> *3.2.3 Control by Administrative or Other Arrangements.*
>>>> The preferences of users and of servers* may* be ascertained by 
>>>> means outside the scope of this document, for example:
>>>>     * the use by transforming proxies of a disallow list of Web sites
>>>>       for which Content Transformation is known to diminish the user
>>>>       experience of content or be ineffective;
>>>>     * the use by transforming proxies of an allow list of Web sites for
>>>>       which Content Transformation is known to be necessary;
>>>>     * terms and conditions of service, as agreed between the user and
>>>>       the Content Transformation service provider.
>>>> *Note:*
>>>> Allow and disallow lists generally cause intractable problems for 
>>>> content providers since there is no mechanism for them to establish 
>>>> which lists they should be on, nor any generic mechanism though 
>>>> which they can check or change their status.
>>>> Cheers
>>>> *Heiko Gerlach*
>>>> *Vendor Manager / Product Owner*
>>>> *Global Consumer Internet Services & Platforms*
>>>> *Tel: +49 211 820 2168*
>>>> *Fax: +49 211 820 2141*
>>>> *Mobile +49 172 20 40 50 7*
>>>> *E-Mail:* ___*heiko.gerlach@vodafone.com*_
>>>> <mailto:heiko.gerlach@vodafone.com>**
>>>> * *
>>>> Vodafone Group Services GmbH
>>>> Mannesmannufer 2, D-40213 Düsseldorf Amtsgericht Düsseldorf, HRB
>>>> 53554
>>>> Geschäftsführung: Dr. Joachim Peters, Rainer Wallek
>>>> This message and any files or documents attached are confidential 
>>>> and may also be legally privileged or protected by other legal 
>>>> rules. It is intended only for the individual or entity named. If 
>>>> you are not the named addressee or you have received this email in 
>>>> error, please inform the sender immediately, delete it from your 
>>>> system and do not copy or disclose it or its contents or use it for any purpose. Thank you.
>>>> Please also note that transmission cannot be guaranteed to be secure 
>>>> or error-free.
>>>> *P* Please consider your environmental responsibility before 
>>>> printing this e-mail

Received on Friday, 13 June 2008 12:34:28 UTC