Re: Sampling

I agree with Vivienne.  We need to ensure that we're still including
primary user scenarios and critical paths that are usually defined in
business test cases.  For our larger websites, we use automated scanning on
the whole site, identify those items mentioned below for representative
sampling of page/technology types, and then model our manual testing after
the top critical paths/scenarios that a user needs to be able to complete.
 If there's capacity to do more than that, we usually look at the most
heavily trafficked pages next.

Perhaps that isn't as random as it should be, but it seems to line up with
overall user experience goals.


Thanks,
Elle



On Thu, Dec 15, 2011 at 9:40 AM, Vivienne CONWAY <v.conway@ecu.edu.au>wrote:

> HI all
>
> Unless I'm missing something, if we are talking about random sampling
> methods, how do we make sure they include those 'complete processes'.  Do
> we look at doing random sampling, plus complete processes, plus core
> elements of the website (website purpose).?
>
>
> Regards
>
> Vivienne L. Conway, B.IT(Hons)
> PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
> Director, Web Key IT Pty Ltd.
> v.conway@ecu.edu.au
> v.conway@webkeyit.com
> Mob: 0415 383 673
>
> This email is confidential and intended only for the use of the individual
> or entity named above. If you are not the intended recipient, you are
> notified that any dissemination, distribution or copying of this email is
> strictly prohibited. If you have received this email in error, please
> notify me immediately by return email or telephone and destroy the original
> message.
> ________________________________________
> From: Velleman, Eric [evelleman@bartimeus.nl]
> Sent: Thursday, 15 December 2011 7:35 PM
> To: RichardWarren; Eval TF
> Subject: RE: Sampling
>
> Hi,
>
> Yes, agree, the evaluation will need to specify the resources that have
> been evaluated.
>
> If the evaluation needs to be replicable and allow synchronous or
> asynchronous comparisons (like monitoring) the evaluation sample must be
> generated by a uniform random procedure that is partly described by Richard
> in an earlier mail (see bottom of this message). Partly, because the
> situation for our uniform random procedure is a bit more complicated than
> with WCAG 1.0. There are some additional factors at work here that are
> described in the Scope section like accessibility support and use of
> different technologies and more. This is covered in WCAG 2.0 like also
> described by Alistair in an earlier mail but we will have to check if that
> is enough for the purpose of the evaluation report.
>
> Question: Can we make a list of what should minimally be in the core
> resource list (if available in the scope of the Website that is being
> evaluated)? We will discuss the size of the sample later.
>
> Using Richards list I come to:
>
> Home Page,
> Site Map,
> Section landing pages (is there a maximum?)
> Any sub-section landing pages (usually linked to from the section landing
> pages)
> Forms
> Data tables
> Multimedia (maybe we have to be more specific here)
>
> While reading, the following additions seem interesting to add:
> Help resource
> Contact information resource
> Search and extended search resources including resulting resources
> Distinct web technology pages (...)
> Pages with other programming languages
> CSS alternatives for mobile, (more..)
> Frames (are they still used?)
>
> Also:
> Resources representative of each category of resources having a
> substantially distinct “look and feel” (typically representative of
> distinct underlying site “templates”) (if identifiable).
> Resources describing accessibility features and / or the accessibility
> policy of the site (if any).
>
>
> The resource list as a whole should, as far as possible, collectively
> address all the applicable sampling objectives within the scope of the
> evaluation.
>
> Kindest regards,
>
> Eric
>
>
>
>
> ________________________________________
> Van: RichardWarren [richard.warren@userite.com]
> Verzonden: donderdag 15 december 2011 11:09
> Aan: Eval TF
> Onderwerp: Re: Sampling
>
> Hi
> Eric is correct that we are evaluating (and therefore sampling) at a moment
> in time.
>
> I presume that we will include documentation of pages visited etc. so that
> the process can be audited (we normally keep a checked list from the site
> map). This check list can be just as useful for Vivienne's monitoring.
>
> Richard
>
> -----Original Message-----
> From: Vivienne CONWAY
> Sent: Thursday, December 15, 2011 2:11 AM
> To: Velleman, Eric ; Boland Jr, Frederick E. ; RichardWarren ; Eval TF
> Subject: RE: Sampling
>
> Hi all
>
> For my current research project, I have used a targetted sampling method.
> As well as scanning the whole site with automated tools to check for
> trends,
> I have chosen 5 pages from each website to check manually according to the
> WCAG 2.0 guidelines.  These pages reflect as many WCAG 2.0 checkpoints as
> possible: home page, contact us page (or something with a form), page with
> photos, page with multimedia, and a page describing their services or other
> page with some complexity.
>
> As I check these websites repeatedly, I will check the same pages to see
> their progression (or degeneration) over time.
>
>
> Regards
>
> Vivienne L. Conway, B.IT(Hons)
> PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
> Director, Web Key IT Pty Ltd.
> v.conway@ecu.edu.au
> v.conway@webkeyit.com
> Mob: 0415 383 673
>
> This email is confidential and intended only for the use of the individual
> or entity named above. If you are not the intended recipient, you are
> notified that any dissemination, distribution or copying of this email is
> strictly prohibited. If you have received this email in error, please
> notify
> me immediately by return email or telephone and destroy the original
> message.
> ________________________________________
> From: Velleman, Eric [evelleman@bartimeus.nl]
> Sent: Thursday, 15 December 2011 7:22 AM
> To: Boland Jr, Frederick E.; RichardWarren; Eval TF
> Subject: RE: Sampling
>
> Frederick,
>
> your remark made me think of time lapse sampling:
>
> It could be that a website/webpage changes over time on purpose. For
> example: it could show images and text related to the time of the day or
> the
> feeling of the owner. If the owner feels bad at 8 am, then the page is dark
> but after his first cup of coffee, the page looks bright and happy...
> In that case it could be interesting to sample the same page at different
> times but I do not know how to put this into the methodology, maybe in a
> footnote?
>
> Eric
>
> ________________________________
> Van: Boland Jr, Frederick E. [frederick.boland@nist.gov]
> Verzonden: woensdag 14 december 2011 21:48
> Aan: RichardWarren; Eval TF
> Onderwerp: RE: Sampling
>
> We may also need to sample over time, since a site’s pages/content may
> change over time, which could affect WCAG2.0 conformance and/or resultant
> accessibility of the site.. or provide a date/time (required component) of
> evaluation for pages/site..  Questions that may arise in this regard are:
> how often to sample, etc. (for example,  do we just want to sample when
> there are major content changes, or do we just want to sample at regular
> intervals regardless of any perceived changes, or do we want to apply
> different strategies for different parts of a site?)
>
> Thanks Tim Boland
>
> From: RichardWarren [mailto:richard.warren@userite.com]
> Sent: Wednesday, December 14, 2011 2:32 PM
> To: Eval TF
> Subject: Sampling
>
> Hi,
>
> To reduce the effort of evaluating a large site I suggest that there are
> three methods of sampling the site’s pages/content which can be coupled
> with
> a barrier identification technique to avoid constant repetition.
>
> The three sampling techniques for manual evaluation are  :-
>
> 1) Random sampling  - selecting a number of pages at random. This can be
> done by making a random selection from the site map, or to take every tenth
> (or other suitable number) of links from the site map.
>
> 2) Structure sampling – selecting the higher level structural pages such as
> Home Page, Site Map, section landing pages (usually linked from the Home
> page within the main navigation bar), any sub-section landing pages
> (usually
> linked to from the section landing pages, plus (if not already found) a
> sample of pages containing elements such as forms, data tables and
> multi-media.
>
> 3) Task orientated sampling – Completing the key tasks on the site required
> to meet the site’s purpose. This might include tasks such as to source
> certain information, place an order or participate in a discussion.
>
> Barrier identification reduces effort further by noting examples of common
> failures in technique employed within the site and once identified and
> commented on we can ignore further occurrences within that section.
>
> We use a combination of all three sampling methods. We start with 2
> (structured sample) to explore the site and obtain an overview of its’
> purpose etc. Next we attempt the key tasks.  Then we do a random sample
> (skipping any pages already sampled).
>
> Richard
>
> This e-mail is confidential. If you are not the intended recipient you must
> not disclose or use the information contained within. If you have received
> it in error please return it to the sender via reply e-mail and delete any
> record of it from your system. The information contained within is not the
> opinion of Edith Cowan University in general and the University accepts no
> liability for the accuracy of the information provided.
>
> CRICOS IPC 00279B
>
> This e-mail is confidential. If you are not the intended recipient you
> must not disclose or use the information contained within. If you have
> received it in error please return it to the sender via reply e-mail and
> delete any record of it from your system. The information contained within
> is not the opinion of Edith Cowan University in general and the University
> accepts no liability for the accuracy of the information provided.
>
> CRICOS IPC 00279B
>
>


-- 
If you want to build a ship, don't drum up the people to gather wood,
divide the work, and give orders. Instead, teach them to yearn for the vast
and endless sea.
- Antoine De Saint-Exupéry, The Little Prince