W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > January 2012

Re: metadata for evaluation methodology?

From: Detlev Fischer <fischer@dias.de>
Date: Thu, 12 Jan 2012 10:25:36 +0100
Message-ID: <4F0EA710.2050008@dias.de>
To: public-wai-evaltf@w3.org
Hi Tim,

I think this is an excellent idea and it helps us structuring our effort.

I have quickly gone through the list of Test Metadata you have suplied 
and tried to map it on what metadata we might want to record. I think it 
is already quite a good fit.

Here are my thoughts:

----
2.1 Identifier
----
An identifier could be developed that might have a generic first part 
with, WCAG, version of techniques doc, unique ID of evaluating 
organization (such thing does not exist yet but is a possibility) may be 
more, and an evaluator-specified second part including a unique 
identifier of the test by that evaluating organization. Such hybrid 
Identifiers have worked quite well in the ATA numbering system in rthwe 
airline industry – but better schemes may exist…

----
2.2 Title
----
Name of the site, may be identifying the scope if several scopes are 
tested differently

----
2.3 Purpose
----
Seems more an aspect of the methodology itself than of any individual 
test/evaluation.

----
2.4 Description
----
This may contain a description of the context of the evaluation, 
especially regarding sections excluded (e.g., “web 2.0” sections with 
content submitted by users where full conformance to SC 1.3.1 may not be 
safeguarded systemically), or legacy content excluded, plans (if 
applicable) making such legacy content accessible in the future.

----
2.5 Status
----
This seems to be what I have described in section 2.10 Version further 
below – or it could be the current (latest) state of the series of 
states of one test.

----
2.6 SpecRef
----
Seems to map nicely onto the WCAG conformance level to be set per scope 
tested.

----
2.7 Preconditions
----
A precondition may rule to exclude sites of parts of sites from 
evaluation on the grounds of lacking accessibility support for the 
context of use. Example: WAI-ARIA and use of modern UA/AT can be 
safeguarded in an intranet context but such support is debatable if we 
think of a public site.
Another precondition may be a requirement for freezing the development 
of the site until the termination of the test (or testing a copy while 
development continues)

----
2.8 Inputs
----
Not sure what this might contain. (Provisional) URL of site/pages at the 
time of testing? A rationale of site owner for excluding parts of the 
site from scope?

----
2.9 ExpectedResults
----
For a binary version of the test, the expected result is that all SC on 
the chosen level of conformance are met. SC-based “tolerance metrics” 
such as “percentage of incidental errors permitted” would be documented. 
We need to determine whether the baseline for such metrics is the some 
average over the sample or each individual page tested (more likely).
A graded version may sit behind the conformance oriented binary result 
and indicate the level of conformance, e.g., in percent. This may simply 
be informal (or informational?) but could prove very useful. I 
personally do not believe a graded result can be based on a mere 
quantitative summing up fail and pass incidents per SC and page. At 
least it would need to recognize critical fails (e.g. image-based 
control without alt) overruling the quantitative count.

----
2.10 Version
----
This seems very useful. If the test is carried out in a DB-based 
application, it is quite easy to save the different states. Just to give 
an example, in the case of our test (BITV-Test) we now record

* Initial evaluator-selected page sample
* page sample signed off after quality check by the selected
   quality assurer who may ask evaluator for changes or additions
   before signing it off
* individual test instances after completion by evaluator
   (one or two)
* consensual test result after the arbitration phase involving
   both evaluators (only in tandem tests)
* final result after check by quality assurer and any changes
   requested before signing off the final test

A re-test would not qualify as version, it would be a new test. 2.13 
Grouping might link different tests of the same site carried out at 
different points in time

----
2.11 Contributor
----
For tests, this might have the components “site owner” (who is not 
necessarily the commissioner/customer of the test), “agency/agencies” 
(it is common to have several agencies contribute to a site design), and 
“evaluator/organisation” (evaluators may cooperate that may belong to 
more than one organisation).
If it applies to the evaluation results itself, separating out 
contributors  can get tricky when having several evaluators and a 
quality assurer working on a test and possibly modifying results 
(ratings and comments). But this may not be a real problem as long as 
there is consensual agreement on changes before they are finalised.

----
2.12 Rights
----
I doubt we will see copyright issues with evaluator reports but if used 
I guess the evaluator would have the right, several evaluators would 
have common rights (if such construct exists)?

----
2.13 Grouping
----
Could be useful for differentiating segments of a site if they conform 
to different levels of WCAG. Another possible use is to group pages in 
the sample that apply the full set of SC and (if this is the approach 
taken) those that apply a partial set (e.g. only SC relevant for data 
tables. Which would be applicable and which others can be safely ignored 
needs to be worked out. A subset for tables would apart from 1.3.1. 
certainly need to include contrast, reading order when linearized, 
scalability etc, but certainly not error identification, consistent 
navigation, multiple ways, bypass blocks, etc.

----
2.14 SeeAlso
----
Might be used for references to earlier tests or pre-tests of the same site


Am 11.01.2012 19:58, schrieb Boland Jr, Frederick E.:
> For example, date of evaluation, who is evaluating, what they’re
> evaluating (scope of evaluation, number of pages, etc.)), sampling
> method used, scoring method/error analysis used, version of WCAG2.0 used
> in evaluation, additional resources/data pertinent to evaluation, etc.)
> – that kind of thing – a little bit like:
>
> http://www.w3.org/TR/test-metadata/
>
> for tests, but in our case for evaluations. Some of the stuff in the
> methodology document, but in metadata form – coming to agreement on what
> should be in a good evaluation report besides the obvious (conformance
> to WCAG2.0) –
>
> sorry I wasn’t clear..
>
> Best, Tim
>
> *From:*Michael S Elledge [mailto:elledge@msu.edu]
> *Sent:* Wednesday, January 11, 2012 1:03 PM
> *To:* Boland Jr, Frederick E.
> *Cc:* Eval TF
> *Subject:* Re: metadata for evaluation methodology?
>
> Hi Tim--
>
> Can you expand a bit on this? I don't think I understand. It seems to me
> that the required components of an evaluation would be the WCAG 2.0
> success criteria, but I'll bet that isn't what you meant.
>
> Mike
>
> On 1/11/2012 8:38 AM, Boland Jr, Frederick E. wrote:
>
> I’m worried that we don’t seem to be compiling anywhere a list of
> (required/optional) (terms/components) for an evaluation. We’re having a
> lot of good discussion and document development, but I couldn’t find
> captured from that discussion anywhere some preliminary metadata for
> evaluations, like was done for Test Metadata for old W3C Quality
> Assurance Working Group and for old test samples development task force.
> Maybe it’s too early yet, but at least we should be brainstorming about
> such a list, at least for the template appendix?
>
> Just a thought..
>
> Thanks and best wishes
>
> Tim Boland NIST
>
> PS – if such a list exists, then my apologies..
>
Received on Thursday, 12 January 2012 09:26:16 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:12 GMT