Re: Accessibility Conformance Testing (ACT) Rules Format 1.0 Feedback

Hi Allistair,

I wanted to address your first point to hopefully provide some clarity on why it's entirely a non-issue that the structure of the output data from Alfa, aXe, and possibly other tools "diverge". The whole point of JSON-LD, RDF, and by extension EARL, is for data consumers to be agnostic to the exact structure of a piece of data, as long as the data is expressed in terms that are well understood by the consumer. Consumers do not need to care about whether a piece of data is structured as an expanded JSON-LD graph (what Alfa outputs) or an arbitrary JSON document with a JSON-LD context on top (what aXe outputs). 

What consumers are interested in are the terms expressed in the data, such as `earl:Assertion` or `earl:TestResult`. When consumers know what they're interested in, they can then frame (https://json-ld.org/spec/latest/json-ld-framing/) the data however they want to make it follow whatever structure their systems accept. As an example, consider this frame used when building the implementation reports for the ACT-R website: https://github.com/act-rules/act-rules.github.io/blob/develop/build/implementation-json-ld-frame.js. When applied to output from either Alfa or aXe, the resulting data has the exact same structure regardless of whether it came from Alfa or aXe.

In summary, the structure of the output data doesn't matter; what matters is that it's expressed in terms defined by EARL.

Kasper Isager
Product Owner, Accessibility

Siteimprove
Sankt Annæ Plads 28 | DK-1250 København K
kki@siteimprove.com | +45 31 23 42 44

> On 24 May 2019, at 10.43, Alistair Garrison <alistair.garrison@levelaccess.com> wrote:
> 
> Hi, 
>  
> Level Access’s feedback on Accessibility Conformance Testing (ACT) Rules Format 1.0, based on our experience / knowledge now gained from attempting its real-life implementation.
>  
> General
>  
> Suggestion 1: We need to tighten the outcome data format – possibly restricting it to one form.  The examples suggested for data-format are not trimmed to only those pieces of the format which are 100% necessary.  For example, we likely need the id of the coded test being run, the testing engine name + version, the id of the rule being claimed as supported, the id of the unit test and the outcome from the coded test.
> Reason: People are already starting to implement tools which read this outcome format.  It would be best to promote a single format ASAP, which of course could be a cut-down form of EARL.   You’ll notice, that both Alfa and aXe already report using different structural versions of EARL (and, Deque’s earl context seems different to the earl-act.json context) – with differences in the additional information presented too.  Without clarity more variants from more tool developers will follow. 
>  
> Under 3. ACT Rule Types
>  
> Suggestion 2: We might consider the removal of the sentence “Composite rules cannot contain other composite rules. Any time a nested composite rule would be needed, all of the relevant atomic rules can be combined directly into the new composite rule.”
> Reason: We might need a Composite rule which covers an entire SC, in order to generate outcomes as described under 4.4.1 Outcome Mapping.  What we are noticing is that our tests are way more atomic than the ACT-R rules.  Which means we already use a pseudo “Composite” rule structure to replicate most ACT-R rules.  If we were to present our collection of single, more atomic, rules formally to the ACT-TF as say a replacement for a larger over-arching ACT-R rule we would almost certainly end up having to have Composite rules of Composite rules when covering an entire SC.
>  
> Suggestion 3: We might consider asking authors to ensure their rules are as atomic as possible. 
> Reason: For example, the ACT-R rule “Buttons should have an accessible name” under our testing methods is actually composed of five atomic rules.
>  
> Under 4.5 Rule Input
>  
> Suggestion 4: We might consider extending the list under each example to include “ECMAScript” and “Shadow DOM”.
> Reason: Both must now be assumed to be present as default when testing.
>  
> Suggestion 5: Tighten the sentence “For input aspects that are not well defined, an ACT Rule MUST include either a detailed description of the aspect in question, or a reference to a well defined description.”
> Reason: We’ve noticed that the ACT-R Rules point to testing the Accessibility Tree.  There is no, as far as we understand, formal Accessibility Tree standard.  This means it’s implementation by each different tool developer might be a cause for divergence.  We might consider creating a detailed, shared understanding of some of these terms.
>  
> Alistair
>  
> ---
>  
> Alistair Garrison
> Director of Accessibility Research
> Level Access

Received on Friday, 24 May 2019 10:31:05 UTC