RE: UIA platform testing

Few notes:

- DPUB AAM - we currently don't implement those mappings, so this part can be skipped for EdgeHTML
- Things similar to "StyleId_Heading2" - this is about TextPattern and can be verified in Inspect.exe using Text Pattern Explorer (available on the <body> element).

-----Original Message-----
From: Joanmarie Diggs [mailto:jdiggs@igalia.com] 
Sent: Tuesday, May 23, 2017 10:24 AM
To: John Jansen <John.Jansen@microsoft.com>; Bogdan Brinza <bbrinza@microsoft.com>; Rich Schwerdtfeger <richschwer@gmail.com>
Cc: Shane McCarron <shane@spec-ops.io>; ARIA Working Group <public-aria@w3.org>
Subject: Re: UIA platform testing

Hi John.

<cough>Welcome to my world</cough>. ;)

If you go through the wiki history, you will find quite a few instances of my correcting statements. Automated tools have proven handy in this
regard: They flag a failure, you look into it and realize the bug is not in the implementation or your ATTA, but in the testable statement itself. :-/

Before you reach this as your definitive conclusion, it's important to check what is stated in the Core AAM, because I've also discovered bugs there (i.e. the testable statement is correct given what is in the Core AAM, which is wrong -- or at least highly suspect). If the bug is in the Core AAM, please file an issue against the spec. Bonus points for providing a patch which fixes it, should time permit and the fix be obvious.

But if in the end you are sure the fault lies 100% in the testable statement, and it's a statement for UIA, please just edit the wiki to reflect the correct assertion.

Thanks!
--joanie

On 05/23/2017 07:05 PM, John Jansen wrote:
> Thanks, Bogdan.
> 
>  
> 
> We have started testing, and I was actually going to send mail today 
> regarding some issues we've found with the tests that make it a bit 
> more challenging that I think it needs to be. I'm curious how the 
> automation system worked around these:
> 
>  
> 
>  1. Simple typo inconsistency: "Control Type", "control type",
>     "ControlType" and "Localized Control Type", "localizedControlType",
>     "LocalizedControlType".
>      1. This is manageable, but makes understanding the tests 
> challenging
> 
> *Q: Is there anyone going through to make these consistent? I guess 
> this is the result of multiple people writing test statements.*
> 
>  
> 
>  2. TestCases looking for aria-properties, ask in lots of different ways
>      1. Some tests say
> 
>                                                                i.     
> <testelement> /PROPERTY/ "AriaProperties" /IS/ "*_hidden:_*true"
> 
>                                                              ii.     
> <testelement> /PROPERTY /"AriaProperties*_.hidden_*" /IS/ "true"
> 
>                                                            iii.     
> <testelement> /PROPERTY/ "hidden" /IS /"true" (and just assumes you 
> know it wants ariaproperties)
> 
>                                                            iv.     
> Might be more.
> 
>      2. Some more ambiguous tests ask
> 
>                                                                i.     
> <testelement> /PROPERTY /"DataGrid.ColumnCount" /IS /"4" and seems to 
> really be wanting "AriaProperties.ColCount == 4"
> 
>              1. The element in inspect.exe shows no other fields showing
>                 value "4" so it must be AriaProperties.ColCount
>              2. I don't know what DataGrid has to do with anything
>      2. This is hard to handle... and probably should be fixed on the test
>         side. 
> 
> *Q: Should we just log bugs in the repo?*
> 
>  
> 
>  3. There is an entire family of tests asking for Pattern support, in
>     different ways seemingly:
>      1. <testelement> /PROPERTY /"controlPatterns" /CONTAINS
>         /"*ExpandCollapse*"
>      2. <testelement> /CONTROL PATTERN /"*ExpandCollapse*" /EXISTS/ "false"
> 
> *Q: is anyone working on this?*
> 
>  
> 
>  4. Some I cannot figure out what it's trying to verify at all,
>     inspect.exe shows nothing with a similar name, nothing with a value
>     matching the assertion value.
>      1. <testelement> /PROPERTY /"StyleId_Heading2" /IS/ "present" -
>         inspect shows nothing similar to StyleId_heading2.
>      2. Some are more ambiguous, some are less ambiguous.
> 
> *Q: Not sure what to do with these types of issues. What are your 
> thoughts?*
> 
>  
> 
>  
> 
>  
> 
>  
> 
> *From:* Bogdan Brinza
> *Sent:* Monday, May 22, 2017 10:02 PM
> *To:* Rich Schwerdtfeger <richschwer@gmail.com>; John Jansen 
> <John.Jansen@microsoft.com>
> *Cc:* Shane McCarron <shane@spec-ops.io>; Joanmarie Diggs 
> <jdiggs@igalia.com>; ARIA Working Group <public-aria@w3.org>
> *Subject:* Re: UIA platform testing
> 
>  
> 
> Adding John who drives UIA testing on our end.
> 
> ----------------------------------------------------------------------
> --
> 
> *From:*Rich Schwerdtfeger <richschwer@gmail.com 
> <mailto:richschwer@gmail.com>>
> *Sent:* Monday, May 22, 2017 12:28:01 PM
> *To:* Bogdan Brinza
> *Cc:* Shane McCarron; Joanmarie Diggs; ARIA Working Group
> *Subject:* UIA platform testing
> 
>  
> 
> Hi Bogdan,
> 
>  
> 
> Shane fixed the bug in the ATTA to enable manual testing to proceed. 
> 
>  
> 
> How close are you to being able to start manual testing?
> 
>  
> 
> Have you updated the wiki to reflect the UIA Core AAM changes you made?
> 
>  
> 
> Regards,
> 
> Rich
> 
>  
> 
> Rich Schwerdtfeger
> 
>  
> 
>  
> 
>  
>

Received on Tuesday, 23 May 2017 23:33:49 UTC