Re: ISSUE-30 counter-proposal

On Feb 14, 2010, at 9:04 AM, Shelley Powers wrote:

> Maciej Stachowiak wrote:
>> Hi Shelley,
>>
>> If you want to argue against the studies Ian cited, please do so by  
>> using facts to refute them, instead of just casting vague  
>> aspersions on Ian's integrity. Examples of fact-based arguments  
>> would be: (1) cite specific methodological flaws; (2) perform or  
>> cite a better study that finds different results. That's what a  
>> scientist peer reviewing a study would do, they don't just accuse  
>> each other "bias".
>
> I'm using the terminology that was established within the psychology  
> field when referencing studies of this nature.
>
> The use of "bias" in the field, especially in regards to research is  
> based more on a set of assumptions than something 'negative'. Every  
> researcher is biased, no matter how much they try to approach a  
> research topic in a neutral, "unbiased" manner.

Sample bias is a relevant concept in statistics, as is systemic bias.  
But we don't usually refer to the investigator being biased. In  
science, having a guess what the outcome of an experiment will be is  
called a "hypothesis", not "bias".

>
> The statements I made are legitimate statements when it comes to  
> reviewing study results, and are common. I did actually refute Ian's  
> statements, and studies, and there was nothing person in any of it.
>
> The only one out of line, Maciej, is you. I would suggest you may  
> want to check my statements with your company's usability experts if  
> you think my statements of some form of personal attack.

I've done scientific research back in my school days, and I have  
training in statistics. I am pretty sure I can tell what constitutes a  
statistically valid study, or what sorts of errors of methodology  
might invalidate its conclusion.

In any case, if anyone wants to question the statistical validity or  
relevance of these studies, probably the best thing to do would be to  
get that info in the rationale when updating the original issue-30  
change proposal.

Regards,
Maciej

P.S.  I did some back-of-the-envelope statistical analysis which I  
won't bore the Working Group with, but if anyone is curious I can post  
it to www-archive. For example I how likely it is, given Ian's study,  
that when longdesc is used the value is actually useful over 75% of  
the time, rather than the <1% he found. Also just how much he'd have  
to oversample bad longdescs to get his result anyway in that case.


>
> Shelley
>
>>
>> Regards,
>> Maciej
>>
>> P.S. Of the four studies cited below, only one was performed by Ian  
>> and at least two include the full raw data on which they base their  
>> conclusions. Some of the studies are also endorsed by noted Web  
>> accessibility experts. A useful thing to do would be to follow the  
>> links, read over the studies, and report back to the Working Group  
>> on the specific problems you find with any of these studies.
>>
>> On Feb 14, 2010, at 7:37 AM, Shelley Powers wrote:
>>
>>> Ian Hickson wrote:
>>>> Here's a counter-proposal for ISSUE-30:
>>>>
>>>> == Summary ==
>>>>
>>>> The longdesc="" attribute does not improve accessibility in  
>>>> practice and should not be included in the language.
>>>>
>>>> == Rationale ==
>>>>
>>>> Several studies have been performed. They have shown that:
>>>>
>>>> * The longdesc="" attribute is extremely rarely used (on the  
>>>> order of 0.1% in one study). [http://blog.whatwg.org/the-longdesc-lottery 
>>>> ]
>>>> * When used, longdesc="" is extremely rarely used correctly (over  
>>>> 99% were incorrect in a study that only caught the most obvious  
>>>> errors [http://blog.whatwg.org/the-longdesc-lottery]; the correct  
>>>> values were below the threshold of statistical significance on  
>>>> another study that examined each longdesc="" by hand [http://wiki.whatwg.org/wiki/Longdesc_usage] 
>>>> ).
>>>> * Most users (more than 90%) don't want the interaction model  
>>>> that longdesc="" implies. [http://webaim.org/projects/screenreadersurvey2/#images 
>>>> ]
>>>> * Users that try to use longdesc="" find it doesn't work ("Who  
>>>> uses this kind of thing? In my experience [...] it just didn't  
>>>> work. There was no description.") [http://www.cfit.ie/html5_video/Longdesc_IDC.wmv 
>>>> ].
>>>>
>>>
>>> I'll let the accessibility folks respond to the accessibility  
>>> components of your proposal, but we've had discussions in the past  
>>> about your "studies", and the flaws associated with them.
>>>
>>> First of all, you've not provided access to the same data, so your  
>>> results cannot be confirmed or denied.
>>>
>>> Secondly, you have a bias in the results, and bias has been shown  
>>> to compromise the integrity of studies. That's why researchers use  
>>> blind studies, in order to ensure their biases and assumptions do  
>>> not impact on the results.
>>>
>>> Third, there is no way to determine the cause of results found on  
>>> the web. Were incorrect uses because longdesc is inherently too  
>>> difficult for users? Or because it was inadequately documented?  
>>> What is the age of the results, and is there a trend to a more  
>>> positive useful effect, as understanding grows about longdesc?
>>>
>>> There is no way to form an irrefutable conclusion from nothing  
>>> more than scraped data found on the web -- there are no controls  
>>> in place to separate out the various cause agents, and focus  
>>> specifically on one or another. The most you can do is make an  
>>> anecdotal observation, and again,  your own biases in regard to  
>>> longdesc undermine the effectiveness of the observation.
>>>
>>> You don't have to take my word for any of this: I would suggest  
>>> you run my comments by your Google "experts", and I think you'll  
>>> find that they agree with me.
>>>
>>> Therefore, your studies cannot, by themselves, be used to form a  
>>> legitimate decision about longdesc, or any other aspect of the  
>>> HTML5 specification. All that remains is your other arguments,  
>>> which I'll leave to others to debate, or not.
>>
>>
>>
>>
>

Received on Sunday, 14 February 2010 18:54:30 UTC