OOBC requirements

Hi all,

First, thanks so much to Ronan for detailing the requirements for OOBC.
I want to emphasize that there is a third option on the table for how to
handle OOBC, which is to not make a special exception to accommodate
this. I have a concern with an extra permitted use because it may create
a reason for keeping unique id cookies (I don't think there currently
are permitted uses on the table that require unique id cookies and I'd
like to keep it that way), there is less transparency to the user as
Nick emphasized ("do these people think they have OOBC from me or
not?"), and it is yet another permitted use, where we should be trying
to keep these as few and narrowly scoped as possible. I have a concern
about the original approach of using a separate TSV for OOBC, since I
think this adds a lot of complexity, and puts burden on the user agent
to appropriately report back to the user what is going on. Moreover,
given added complexity, there is more room for things to break, e.g. if
the tracking status resource is down for a week or a month, whose
responsibility is it to demand that it go back up? Are there any
repercussions for that sort of flakiness?

Given that, I'd like to avoid having to make a special exception for
OOBC if we can get away with it, and think it is appropriate to dive
more into the statistical weeds of how these panel studies are
conducted. Ronan, if this is not your area of expertise, could you ask
the person in your organization responsible for handling the panel study
significance tests to describe how these work to the group? I'd also be
happy to talk to this person 1:1 offline.

My null hypothesis is that you could easily normalize for missing DNT:1
data. A lot hinges on how panels are set up and I'd love to understand
this better. But suppose you make your initial panel larger than it has
to be (say, twice the size, but the multiplier will depend on total
DNT:1 adoption rate), with the idea that certain individuals will be
something like "alternates" whose data won't count. I sort of presume
this is done already if any statistical testing is happening. Now only a
subset of those participants who have given OOBC report back with data,
presumably due to DNT:1 (noting again as others have that many OOBC
mechanisms could set DNT:0 in-band). The question is whether you can
still conduct a statistically significant study given this restriction.
My contention is that this is easy, given background knowledge of the
histogram of DNT usage by demographic segment. For simplicity in order
to illustrate this point, let's just suppose that there is only one bit
of demographic information (say, male vs female). Let's suppose women
are twice as likely as men to turn DNT:1 on. Now a lot hinges on what
sort of statistical tests are being used to create a balanced panel. Is
it a manually curated panel where every demographic is represented (e.g.
1 man, 1 woman)? Is it a naive statistically produced panel reflecting
the demographic distribution (50% men, 50% women)? Or some more complex
frequentist or Bayseian statistical approach? Manual curated panels are
obviously still possible without problems given a moment's thought. As
for naive statistical panels, this is easy to normalize for -- just
build the assumption in that the missing person is twice as likely to be
a woman than a man. If there are more complex tests going on, let's hear
about them! I'm optimistic that DNT:1 users should not be devastating to
the creation of a fair and unbiased panel, but if I'm wrong, I'd like to
understand why in detail.

Finally, I think that periodically pushing static data structures with
500K records to CDNs that can be queried under 200ms is not a terribly
burdensome technical ask, and I'm happy to talk about the technical
challenges about how this is done.

I don't want to single out OOBC in particular with this analysis. Ronan
has been refreshingly forthcoming in terms of his requirements and I
appreciate this very much. We need to be rigorous and examine ALL
permitted uses closely, so that we can ensure that they are necessary
and appropriately scoped. I think we're far from that goal now, but
perhaps we can continue the constructive dialogue on OOBC and it can
serve as a model for digging into other areas of industry that touch on
permitted uses.

Best,
Dan

-- 
Dan Auerbach
Staff Technologist
Electronic Frontier Foundation
dan@eff.org
415 436 9333 x134

Received on Wednesday, 24 April 2013 20:55:20 UTC