Re: ISSUE-30 counter-proposal

Maciej Stachowiak wrote:
>
> On Feb 14, 2010, at 9:04 AM, Shelley Powers wrote:
>
>> Maciej Stachowiak wrote:
>>> Hi Shelley,
>>>
>>> If you want to argue against the studies Ian cited, please do so by 
>>> using facts to refute them, instead of just casting vague aspersions 
>>> on Ian's integrity. Examples of fact-based arguments would be: (1) 
>>> cite specific methodological flaws; (2) perform or cite a better 
>>> study that finds different results. That's what a scientist peer 
>>> reviewing a study would do, they don't just accuse each other "bias".
>>
>> I'm using the terminology that was established within the psychology 
>> field when referencing studies of this nature.
>>
>> The use of "bias" in the field, especially in regards to research is 
>> based more on a set of assumptions than something 'negative'. Every 
>> researcher is biased, no matter how much they try to approach a 
>> research topic in a neutral, "unbiased" manner.
>
> Sample bias is a relevant concept in statistics, as is systemic bias. 
> But we don't usually refer to the investigator being biased. In 
> science, having a guess what the outcome of an experiment will be is 
> called a "hypothesis", not "bias".

I was not talking about statistical bias. I was talking about researcher 
bias, which is a different thing.

The real point, though, is that you seem to be interpreting my earlier 
note to Ian in some arbitrarily negative light, that it was a personal 
attack of some form, and that was NOT the intention.

You're one of the first people in this email list to say we should not 
attribute a certain mindset or intention when reading these emails.


>
>>
>> The statements I made are legitimate statements when it comes to 
>> reviewing study results, and are common. I did actually refute Ian's 
>> statements, and studies, and there was nothing person in any of it.
>>
>> The only one out of line, Maciej, is you. I would suggest you may 
>> want to check my statements with your company's usability experts if 
>> you think my statements of some form of personal attack.
>
> I've done scientific research back in my school days, and I have 
> training in statistics. I am pretty sure I can tell what constitutes a 
> statistically valid study, or what sorts of errors of methodology 
> might invalidate its conclusion.
>

We have to disagree about the usefulness of these studies for forming a 
decision about longdesc.



> In any case, if anyone wants to question the statistical validity or 
> relevance of these studies, probably the best thing to do would be to 
> get that info in the rationale when updating the original issue-30 
> change proposal.

I'll leave that up to Charles if he wants to include my response to 
Ian's Counter-Proposal in an update to his change proposal.

Or are you making a call for a counter-counter-proposal? Would we have 
thirty days in order to write the support-proposal?

>
> Regards,
> Maciej
>
> P.S.  I did some back-of-the-envelope statistical analysis which I 
> won't bore the Working Group with, but if anyone is curious I can post 
> it to www-archive. For example I how likely it is, given Ian's study, 
> that when longdesc is used the value is actually useful over 75% of 
> the time, rather than the <1% he found. Also just how much he'd have 
> to oversample bad longdescs to get his result anyway in that case.
>
Maciej, we're talking past each other. I don't think you're hearing what 
I'm saying, even though I tried to establish the context of my comments. 
I was not writing as a statistician.

Regardless, I feel my point is solid, you don't, life goes on.

Shelley

>
>>
>> Shelley
>>
>>>
>>> Regards,
>>> Maciej
>>>
>>> P.S. Of the four studies cited below, only one was performed by Ian 
>>> and at least two include the full raw data on which they base their 
>>> conclusions. Some of the studies are also endorsed by noted Web 
>>> accessibility experts. A useful thing to do would be to follow the 
>>> links, read over the studies, and report back to the Working Group 
>>> on the specific problems you find with any of these studies.
>>>
>>> On Feb 14, 2010, at 7:37 AM, Shelley Powers wrote:
>>>
>>>> Ian Hickson wrote:
>>>>> Here's a counter-proposal for ISSUE-30:
>>>>>
>>>>> == Summary ==
>>>>>
>>>>> The longdesc="" attribute does not improve accessibility in 
>>>>> practice and should not be included in the language.
>>>>>
>>>>> == Rationale ==
>>>>>
>>>>> Several studies have been performed. They have shown that:
>>>>>
>>>>> * The longdesc="" attribute is extremely rarely used (on the order 
>>>>> of 0.1% in one study). [http://blog.whatwg.org/the-longdesc-lottery]
>>>>> * When used, longdesc="" is extremely rarely used correctly (over 
>>>>> 99% were incorrect in a study that only caught the most obvious 
>>>>> errors [http://blog.whatwg.org/the-longdesc-lottery]; the correct 
>>>>> values were below the threshold of statistical significance on 
>>>>> another study that examined each longdesc="" by hand 
>>>>> [http://wiki.whatwg.org/wiki/Longdesc_usage]).
>>>>> * Most users (more than 90%) don't want the interaction model that 
>>>>> longdesc="" implies. 
>>>>> [http://webaim.org/projects/screenreadersurvey2/#images]
>>>>> * Users that try to use longdesc="" find it doesn't work ("Who 
>>>>> uses this kind of thing? In my experience [...] it just didn't 
>>>>> work. There was no description.") 
>>>>> [http://www.cfit.ie/html5_video/Longdesc_IDC.wmv].
>>>>>
>>>>
>>>> I'll let the accessibility folks respond to the accessibility 
>>>> components of your proposal, but we've had discussions in the past 
>>>> about your "studies", and the flaws associated with them.
>>>>
>>>> First of all, you've not provided access to the same data, so your 
>>>> results cannot be confirmed or denied.
>>>>
>>>> Secondly, you have a bias in the results, and bias has been shown 
>>>> to compromise the integrity of studies. That's why researchers use 
>>>> blind studies, in order to ensure their biases and assumptions do 
>>>> not impact on the results.
>>>>
>>>> Third, there is no way to determine the cause of results found on 
>>>> the web. Were incorrect uses because longdesc is inherently too 
>>>> difficult for users? Or because it was inadequately documented? 
>>>> What is the age of the results, and is there a trend to a more 
>>>> positive useful effect, as understanding grows about longdesc?
>>>>
>>>> There is no way to form an irrefutable conclusion from nothing more 
>>>> than scraped data found on the web -- there are no controls in 
>>>> place to separate out the various cause agents, and focus 
>>>> specifically on one or another. The most you can do is make an 
>>>> anecdotal observation, and again,  your own biases in regard to 
>>>> longdesc undermine the effectiveness of the observation.
>>>>
>>>> You don't have to take my word for any of this: I would suggest you 
>>>> run my comments by your Google "experts", and I think you'll find 
>>>> that they agree with me.
>>>>
>>>> Therefore, your studies cannot, by themselves, be used to form a 
>>>> legitimate decision about longdesc, or any other aspect of the 
>>>> HTML5 specification. All that remains is your other arguments, 
>>>> which I'll leave to others to debate, or not.
>>>
>>>
>>>
>>>
>>
>
>
>

Received on Sunday, 14 February 2010 20:24:43 UTC