Re: [testing] initial results from testing priorities survey

On Mon, 24 Jun 2013 08:33:45 +0200, Vickers, Mark  
<Mark_Vickers@cable.comcast.com> wrote:

> We did promise to only publish the aggregated results of the external  
> survey, but I think we could still present three >columns of results  
> publicly:1. external survey aggregation
> 2. internal doodle poll aggregation
> 3. the synthetic summary of the two surveys
>
> OK?

agree

>
> Thanks,
> mav
>
> On Jun 24, 2013, at 7:36 AM, Giuseppe Pascale <giuseppep@opera.com>
> wrote:
>
>> Please remember that the non aggregated results are member  
>> confidential, so not ro be discussed on this >>list.
>> Also remember that we are still waiting for a response or two.
>> --Sent from my phone
>>
>> Kazuyuki Ashimura <ashimura@w3.org> wrote:
>>> Thanks a lot for your initial discussion, Bin and Mark.
>>>
>>> It is fine by me to merge the internal/external results
>>> based on some measure (e.g., the following).  However,
>>> maybe we should be able to analyze/discuss the results
>>> separately based on the results themselves a bit more
>>> before merging.
>>>
>>> What do you think?
>>>
>>> Kazuyuki
>>>
>>>
>>> On 06/24/2013 06:06 AM, Vickers, Mark wrote:
>>>> Thanks! Works for me!
>>>>
>>>> mav
>>>>
>>>> On Jun 23, 2013, at 10:56 PM, "HU, BIN" <bh526r@att.com>
>>>> wrote:
>>>>
>>>>> Mark,
>>>>>
>>>>> Thank you for the great suggestion. So the mapping can be tweak!
>>>>> ed
>>>>> as:
>>>>>
>>>>> - Internal:
>>>>>
>>>>> OK: 4
>>>>> (OK): 2
>>>>> NO: 0
>>>>>
>>>>> - External:
>>>>>
>>>>> P/M/Now: 4
>>>>> F/M/*:  3
>>>>> P/O/Now: 2
>>>>> F/O/*:  1
>>>>> P/N:  0
>>>>> F/N:  0
>>>>> N/*:  0
>>>>>
>>>>> All,
>>>>>
>>>>> Feel free if you have other suggestions to the group.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bin
>>>>>
>>>>> -----Original Message-----
>>>>> From: Vickers, Mark [mailto:Mark_Vickers@cable.comcast.com]
>>>>> Sent: Sunday, June 23, 2013 9:32 AM
>>>>> To: HU, BIN
>>>>> Cc: Giuseppe Pascale; public-web-and-tv@w3.org
>>>>> Subject: Re: [testing] initial results from testing priorities survey
>>>>>
>>>>> Bin,
>>>>>
>>>>> This is a great start. I only have a couple of suggested tweaks:
>>>>>
>>>>> First, I think I'd prioritize Mandatory for Present or Future over  
>>>>> any Optional. Based on one reply I was involved with, >>>>>the  
>>>>> timing of future specs was as little as a few months from now, which  
>>>>> is less time than these tests will take to >>>>>create. The current  
>>>>> Testing schedule is a two year
>>>>> program with a July start date. Whereas, the need for tests for  
>>>>> optional parts of specs is by definition, well, optional. >>>>>This  
>>>>> also would mean that a 3 on the Internal survey would correspond to  
>>>>> Optional on the External survey, which is >>>>>how I think of it  
>>>>> (i.e. "Nice to have, but  not required").
>>>>>
>>>>> Second, it might be better to reverse the numbering and use a 4 to 0  
>>>>> rather than a 1 to 5 scale. That way, the >>>>>lowest level rating  
>>>>> of "No testing" is zero, which would be the same for any of the many  
>>>>> specs that weren't included >>>>>in the survey. No expressed need  
>>>>> for testing is presented as 0.
>>>>>
>>>>> Thanks,
>>>>> mav
>>>>>
>>>>>
>>>>>
>>>>> On Jun 21, 2013, at 11:08 PM, "HU, BIN" <bh526r@att.com> wrote:
>>>>>
>>>>>> Giuseppe, Clarke, and all,
>>>>>>
>>>>>> Thanks for collecting and putting those information together.
>>>>>>
>>>>>> The internal survey and exte!
>>>>>> rnal
>>>>>> survey use different metrics, i.e.:
>>>>>> - Internal: priorities
>>>>>> - External: reference status / mandatory v.s. optional / timing
>>>>>>
>>>>>> If we want to aggregate those results into a single column, e.g.  
>>>>>> priority level (1-5), I propose the mapping as >>>>>>follows:
>>>>>>
>>>>>> - Internal:
>>>>>>
>>>>>> OK: 1
>>>>>> (OK): 3
>>>>>> NO: 5
>>>>>>
>>>>>> - External:
>>>>>>
>>>>>> P/M/Now: 1
>>>>>> P/O/Now: 2
>>>>>> F/M/*:  3
>>>>>> F/O/*:  4
>>>>>> P/N:  5
>>>>>> F/N:  5
>>>>>> N/*:  5
>>>>>>
>>>>>> We can first map the individual results to the scale 1-5, and then  
>>>>>> calculate the scores for each spec (i.e. the >>>>>>highest votes)  
>>>>>> to conclude its final priority level.
>>>>>>
>>>>>> Feel free to suggest other methods for mapping the result and  
>>>>>> calculating the score.
>>>>>>
>>>>>> Thanks
>>>>>> Bin
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Giuseppe Pascale [mailto:giuseppep@opera.com]
>>>>>> Sent: Friday, June 21, 2013 4:15 AM
>>>>>> To: public-web-and-tv@w3.org
>>>>>> Subject: [testing] initial results from testing prioriti!
>>>>>> es
>>>>>> survey
>>>>>>
>>>>>> Clarke, all
>>>>>> I've started to copy the results from the responses received on the  
>>>>>> MEMBER
>>>>>> ONLY wiki
>>>>>>
>>>>>> https://www.w3.org/Member/wiki/Webtv/testing
>>>>>>
>>>>>> FYI, I added the link above also to the testing section of the  
>>>>>> public wiki
>>>>>> http://www.w3.org/2011/webtv/wiki/Testing#External_Groups
>>>>>>
>>>>>> Also, I've closed the internal doodle poll, and the result are now
>>>>>> available on the MEMBER ONLY wiki
>>>>>> https://www.w3.org/Member/wiki/Webtv/testing#Internal_members_survey
>>>>>>
>>>>>> What remains to be done:
>>>>>>
>>>>>> 1. wait for a couple of groups that have promised a reply soon
>>>>>> 2. copy those result on the MEMBER wiki, once received
>>>>>> 3. aggregate the internal member results, add a column in the table  
>>>>>> for
>>>>>> Web&TV
>>>>>> 4. define how we want to aggregate all the result in order to  
>>>>>> provide a
>>>>>> single "TV" column for the W3C testing group to consider. In  
>>>>>> particular
>>>>>> decide which kind of metric to use, as we have, for each spec, a  
>>>>>> variety
>>>>>> of values we can consider:
>>>>>>
>>>>>> - how many group already reference it
>>>>>> - how many group consider testing of it mandatory
>>>>>> - how many groups plan to reference it in future
>>>>>> - when testing would be needed
>>>>>> - how it scored in the member survey
>>>>>>
>>>>>> What we could do is to define a priority level (e.g. 1-5). If we do  
>>>>>> that,
>>>>>> we need to define how we map the info above on a given priority  
>>>>>> level.
>>>>>>
>>>>>>
>>>>>> /g
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Giuseppe Pascale
>>>>>> Product Manager TV & Connected Devices
>>>>>> Opera Software
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> --Kaz Ashimura, W3C Staff Contact for Web&TV, !
>>> MMI and
>>> Voice
>>> Tel: +81 466 49 1170
>



-- 
Giuseppe Pascale
Product Manager TV & Connected Devices
Opera Software