Re: [testing] initial results from testing priorities survey

We did promise to only publish the aggregated results of the external survey, but I think we could still present three columns of results publicly:
1. external survey aggregation
2. internal doodle poll aggregation
3. the synthetic summary of the two surveys

OK?

Thanks,
mav

On Jun 24, 2013, at 7:36 AM, Giuseppe Pascale <giuseppep@opera.com<mailto:giuseppep@opera.com>>
 wrote:

Please remember that the non aggregated results are member confidential, so not ro be discussed on this list.

Also remember that we are still waiting for a response or two.
--
Sent from my phone

Kazuyuki Ashimura <ashimura@w3.org<mailto:ashimura@w3.org>> wrote:

Thanks a lot for your initial discussion, Bin and Mark.

It is fine by me to merge the internal/external results
based on some measure (e.g., the following).  However,
maybe we should be able to analyze/discuss the results
separately based on the results themselves a bit more
before merging.

What do you think?

Kazuyuki


On 06/24/2013 06:06 AM, Vickers, Mark wrote:
Thanks! Works for me!

mav

On Jun 23, 2013, at 10:56 PM, "HU, BIN" <bh526r@att.com<mailto:bh526r@att.com>>
wrote:

Mark,

Thank you for the great suggestion. So the mapping can be tweak!
 ed
as:

- Internal:

OK: 4
(OK): 2
NO: 0

- External:

P/M/Now: 4
F/M/*:  3
P/O/Now: 2
F/O/*:  1
P/N:  0
F/N:  0
N/*:  0

All,

Feel free if you have other suggestions to the group.

Thanks

Bin

-----Original Message-----
From: Vickers, Mark [mailto:Mark_Vickers@cable.comcast.com<http://cable.comcast.com>]
Sent: Sunday, June 23, 2013 9:32 AM
To: HU, BIN
Cc: Giuseppe Pascale; public-web-and-tv@w3.org<mailto:public-web-and-tv@w3.org>
Subject: Re: [testing] initial results from testing priorities survey

Bin,

This is a great start. I only have a couple of suggested tweaks:

First, I think I'd prioritize Mandatory for Present or Future over any Optional. Based on one reply I was involved with, the timing of future specs was as little as a few months from now, which is less time than these tests will take to create. The current Testing schedule is a two year
program with a July start date. Whereas, the need for tests for optional parts of specs is by definition, well, optional. This also would mean that a 3 on the Internal survey would correspond to Optional on the External survey, which is how I think of it (i.e. "Nice to have, but  not required").

Second, it might be better to reverse the numbering and use a 4 to 0 rather than a 1 to 5 scale. That way, the lowest level rating of "No testing" is zero, which would be the same for any of the many specs that weren't included in the survey. No expressed need for testing is presented as 0.

Thanks,
mav



On Jun 21, 2013, at 11:08 PM, "HU, BIN" <bh526r@att.com<mailto:bh526r@att.com>> wrote:

Giuseppe, Clarke, and all,

Thanks for collecting and putting those information together.

The internal survey and exte!
 rnal
survey use different metrics, i.e.:
- Internal: priorities
- External: reference status / mandatory v.s. optional / timing

If we want to aggregate those results into a single column, e.g. priority level (1-5), I propose the mapping as follows:

- Internal:

OK: 1
(OK): 3
NO: 5

- External:

P/M/Now: 1
P/O/Now: 2
F/M/*:  3
F/O/*:  4
P/N:  5
F/N:  5
N/*:  5

We can first map the individual results to the scale 1-5, and then calculate the scores for each spec (i.e. the highest votes) to conclude its final priority level.

Feel free to suggest other methods for mapping the result and calculating the score.

Thanks
Bin

-----Original Message-----
From: Giuseppe Pascale [mailto:giuseppep@opera.com<http://opera.com>]
Sent: Friday, June 21, 2013 4:15 AM
To: public-web-and-tv@w3.org<mailto:public-web-and-tv@w3.org>
Subject: [testing] initial results from testing prioriti!
 es
survey

Clarke, all
I've started to copy the results from the responses received on the MEMBER
ONLY wiki

https://www.w3.org/Member/wiki/Webtv/testing

FYI, I added the link above also to the testing section of the public wiki
http://www.w3.org/2011/webtv/wiki/Testing#External_Groups

Also, I've closed the internal doodle poll, and the result are now
available on the MEMBER ONLY wiki
https://www.w3.org/Member/wiki/Webtv/testing#Internal_members_survey

What remains to be done:

1. wait for a couple of groups that have promised a reply soon
2. copy those result on the MEMBER wiki, once received
3. aggregate the internal member results, add a column in the table for
Web&TV
4. define how we want to aggregate all the result in order to provide a
single "TV" column for the W3C testing group to consider. In particular
decide which kind of metric to use, as we have, for each spec, a variety
of values we can consider:

- how many group already reference it
- how many group consider testing of it mandatory
- how many groups plan to reference it in future
- when testing would be needed
- how it scored in the member survey

What we could do is to define a priority level (e.g. 1-5). If we do that,
we need to define how we map the info above on a given priority level.


/g











--
Giuseppe Pascale
Product Manager TV & Connected Devices
Opera Software






--
Kaz Ashimura, W3C Staff Contact for Web&TV, !
 MMI and
Voice
Tel: +81 466 49 1170