Re: [aria-testsuite] spec violations in ARIA testsuite

Hi James,

Thanks for the detailed reply. Some responses inline below.

James Craig <jcraig@apple.com>, 2013-08-19 15:43 -0700:

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/accessible-name-input/checkbox-label-embedded-slider.html":9:142:
> > error: Element “input” is missing required attribute “aria-orientation”.
> ...
> Error in the validator. aria-orientation has a default value of "horizontal" when it's not defined explicitly.
> http://www.w3.org/WAI/PF/aria/complete#aria-orientation
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/author-requirements/577.html":9:42:
> > error: Bad value “rowgroup” for attribute “role” on element “div”.
> ...
> Error in the validator. (The test case is also invalid but it's a different error.)
> http://www.w3.org/WAI/PF/aria/complete#rowgroup
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-gridcell-aria-required-false.html":10:65:
> > error: Attribute “aria-required” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#gridcell
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-radiogroup-aria-required-false.html":8:61:
> > error: Attribute “aria-required” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#radiogroup
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-separator-aria-orientation-horizontal.html":10:68:
> > error: Attribute “aria-orientation” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#separator
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-spinbutton-aria-required-false.html":8:61:
> > error: Attribute “aria-required” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#spinbutton
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-tablist-aria-level-1.html":8:51:
> > error: Attribute “aria-level” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#tablist
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported/roles-properties-supported-tree-aria-required-false.html":8:55:
> > error: Attribute “aria-required” not allowed on element “div” in this context.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#tree
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported-inherited/roles-properties-supported-inherited-columnheader-aria-required-false.html":12:68: error: Attribute “aria-required” not allowed on element “span” in this context.
> ...
> Error in the validator. Columnheader and rowheader inherit @aria-required from gridcell.
> http://www.w3.org/WAI/PF/aria/complete#columnheader
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-properties-supported-inherited/roles-properties-supported-inherited-treegrid-aria-required-false.html":10:59: error: Attribute “aria-required” not allowed on element “div” in this context.
> ...
> Error in the validator. Treegrid inherits @aria-required from tree.
> http://www.w3.org/WAI/PF/aria/complete#treegrid
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/namefromauthor-requ/883.html":10:35:
> > error: Bad value “columnheader” for attribute “role” on element “th”.
>...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#columnheader
>
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/namefromauthor-requ/884.html":8:77:
> > error: Bad value “scrollbar” for attribute “role” on element “div”.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#scrollbar
> ...
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/properties-global-norole/properties-global-norole-aria-dropeffect-link.html":8:44:
> > error: Bad value “link” for attribute “aria-dropeffect” on element “div”.
> ...
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#aria-dropeffect

Yep, thanks -- I've fixed all of the above in the validator now.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/host-language/implicit-semantics-checkbox-disparity.html":9:123:
> > error: Bad value “checkbox” for attribute “role” on element “input”.
> ...
> Michael Cooper, is this an error in the validator? The roles are the same. 
> <input type="checkbox" role="checkbox" checked="checked" aria-checked="false" id="test" value="Placeholder content">

That was validator error. I've fixed it in the validator now.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/presentation-role/496.html":8:89:
> > error: Bad value “button” for attribute “type” on element “input”.
> 
> Is this being flagged as an error because of the presentation role? input[type="button"] is definitely not an error.

Yes, it's being flagged because of the presentation role.  The HTML spec
currently doesn't allow the presentation role for input@type="button":

  http://www.w3.org/html/wg/drafts/html/master/dom.html#sec-implicit-aria-semantics

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/_functional/tree/ariatree.html":30:21:
> > error: Bad value “group” for attribute “role” on element “ul”.
> ...
> Error in the validator. The group role definitely exists and it's properly used in this tree example.
> http://www.w3.org/WAI/PF/aria/complete#group
> ...
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/presentation-role/528.html":9:23:
> > error: Bad value “checkbox” for attribute “role” on element “ul”.
> 
> Granted, it's a weird test case to set the role on the UL, but: Error in the validator.

The validator's in conformance with the HTML spec in reporting those as
errors, because the HTML spec currently doesn't allow ul@role="group" or 
ul@role="checkbox":

  http://www.w3.org/html/wg/drafts/html/master/dom.html#sec-implicit-aria-semantics

So allowing ul@role=group and ul@role="checkbox" would require a change to
the HTML spec.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/name-computation-input/548.html":9:59:
> > error: Bad value “menu” for attribute “role” on element “select”.
> ...
> Error in the validator.

The validator's in conformance with the HTML spec in reporting that as an
error, because the HTML spec currently doesn't allow select@role="menu":

  http://www.w3.org/html/wg/drafts/html/master/dom.html#sec-strong-native-semantics

As far as I can tell, the only role value allowed by the HTML spec for the
select element is select@role=listbox. So allowing select@role=menu would
require a change to the HTML spec.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/name-computation-input/548.html":10:60:
> > error: Bad value “menuitem” for attribute “role” on element “option”.
> ...
> Error in the validator.

The validator's in conformance with the HTML spec in reporting that as an
error, because the HTML spec currently doesn't allow option@role="menuitem":

  http://www.w3.org/html/wg/drafts/html/master/dom.html#sec-strong-native-semantics

The only role value allowed by the HTML spec for the option element is
option@role=option. So allowing option@role=menuitem would require a change
to the HTML spec.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/_/828.htm":1:59:
> > error: Bad value “text all” for attribute “aria-relevant” on element “div”.
> 
> Error in the validator. A value of "text all" is redundant (should just
> be "all") but it's not explicitly disallowed or invalid.

hmm.. The definition of the value of the aria-relevant attribute in the
ARIA spec currently reads:

  The attribute is represented as a space delimited list of the following
  values: additions, removals, text; or a single catch-all value all.

I read that as an either/or, with the spec requiring that it be only one of
the following two choices:

  A. A list of one or more of "additions", "removals", "text".

    OR

  B. A single value, "all" (instead of a list of values).

By that reading, "text all" would be invalid.

So if I'm misreading the spec here, and "text all" is actually intended to
be valid, then I'd think other readers are likely to be misreading the spec
in the same way, so the spec should be refined to make it more clear.

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/host-language/role-2tokens-first.html":8:39:
> > error: Bad value “dialog foo” for attribute “role” on element “div”.
> ...
> This should be a warning ("unrecognized role 'foo'"), not an error.

That current implementation of ARIA checking in the validator doesn't allow
checking of role values that contain multiple tokens. Making the validator
support role values containing multiple tokens would basically require me
to completely re-write the entirety of the ARIA-checking support in the
validator, in a much more complex way then the already-extremely-complex
way in which it's now implemented even just for the single-value-role case.
I'm not going to do that. See my comments here:

  http://lists.w3.org/Archives/Public/public-pfwg-comments/2013JulSep/0007.html

I'm not aware of any good reason why a conforming document would ever need
to contain something like role="dialog foo".

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/namefromauthor-requ/874.html":9:54:
> > error: Bad value “ marquee” for attribute “role” on element “div”.
> Error in the validator.
> http://www.w3.org/WAI/PF/aria/complete#marquee

The problem here is that the validator implementation currently doesn't
allow spaces around role names in the value of the role attribute. I guess
I can update it easily enough to just allow spaces around role names
(though not space-separated lists of multiple tokens), but I think the
better solution is instead for the ARIA spec to require that role
attributes not contain multiple tokens, nor any spaces.

> > https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/properties-global-norole/properties-global-norole-aria-invalid-false.html
> > was supposed to be invalid but was not.
> ...
> I'm not sure what this error message means. Why doesn't the validator
> trust the author's indication that the content is invalid?

That message is being emitted because I have the test runner in in the
validator configured to expect that any files with the string "-invalid-"
in their filenames are intentionally invalid. But I think what those
particular files in the ARIA testsuite are actually intended to test is the
"aria-invalid" attribute. So I guess I may need to come up with a different
filename string for identifying documents that are intentionally invalid.
Maybe "-not-valid-" instead?

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-plain-abstract/roles-plain-abstract-command.html":8:36:
> > error: Bad value “command” for attribute “role” on element “div”.
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/roles-plain-abstract/roles-plain-abstract-composite.html":8:38:
> > error: Bad value “composite” for attribute “role” on element “div”.
> ...
> These are intended to be invalid. It's testing the abstract roles.

OK, so it would be helpful to me and maybe to others as well if we could
follow some file-naming convention for documents that are intentionally
invalid. Maybe "-not-valid-"?

> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/_/554.htm":3:69:
> > error: Element “input” is missing required attribute “alt”.
> ...
> 
> This is an error in the test case, but the validator output should
> probably be more explicit in that @alt is required on input[type="image"]
> not just "input"…
> Should be ~: Element “input” with attribute “type” whose value is “image”
> must have non-empty attribute “alt”.
> ...
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/namefromauthor-requ/862.html":7:53:
> > error: Element “div” is missing required attribute “aria-checked”.
> ...
> Validator should be more explicit here. 
> Element “div” with attribute “role” whose value is “checkbox” must have
> non-empty attribute “aria-checked”.
> ...
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/namefromauthor-requ/885.html":8:74:
> > error: Element “div” is missing required attribute “aria-valuemax”.
> ...
> Validator message should be:
> Element “…” with attribute “role” whose value is “…” must have non-empty attribute “aria-…”.
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/presentation-role/530.html":8:55:
> > error: Element “li” is missing required attribute “role”.
> ...
> Validator message should be:
> Element “…” with attribute “aria-…” whose value is “…” must have non-empty attribute “role”.
> ...
> > "https://dvcs.w3.org/hg/pfwg/raw-file/tip/ARIA/1.0/tests/test-files/presentation-role/531.html":7:75:
> > error: Attribute “aria-checked” not allowed on element “ul” in this context.
> ...
> Validator message should be:
> Attribute “aria-checked” not allowed on element “ul” with attribute “role” whose value is “presentation”. 

I agree it'd be better for the error messages for those cases to be more
precise. I'll consider refining them. But it's actually not possible to
make them more precise if the checking for them continues to be handled in
the place where it's currently handled in the validator, in the RelaxNG
grammar. Making them more specific would require moving the checks out of the
RelaxNG grammar and instead writing some custom Java code to handle them.

  --Mike

-- 
Michael[tm] Smith http://people.w3.org/mike

Received on Tuesday, 20 August 2013 15:19:10 UTC