RE: web-platform-tests results

I also agree with the semantics that Adam is listing. Bunch of these negative test cases are not correct IMO. These needs to be more forgiving and that is what Edge is also doing.

I also don’t think spec should be held up by these tests.

From: Adam Langley <agl@google.com>
Sent: Wednesday, December 19, 2018 3:17 PM
To: Philippe Le Hégaret <plh@w3.org>
Cc: Anthony Nadalin <tonynad@microsoft.com>; W3C Web Authn WG <public-webauthn@w3.org>
Subject: Re: web-platform-tests results

On Wed, Dec 19, 2018 at 9:27 AM Philippe Le Hégaret <plh@w3.org<mailto:plh@w3.org>> wrote:
On 12/17/2018 8:23 AM, Philippe Le Hégaret wrote:
>> Also you should be able to test with safari now with their preview
>> announcement the other day.
>
> I don't have the proper computer for that unfortunately. If someone else
> has the preview, a key, and a python install, I'm happy to do remote
> assistance to get those results.

Sam and I managed to generate test results for safari tech preview 71.

Added in
  https://w3c.github.io/test-results/webauthn/less-than-2.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw3c.github.io%2Ftest-results%2Fwebauthn%2Fless-than-2.html&data=02%7C01%7CAkshay.Kumar%40microsoft.com%7C823bc80659884e1e239408d666084e46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636808583190983810&sdata=isyi4jkzBH8tQAP3P8keqQ1zo1ObFKia%2BcoyqUOB3FU%3D&reserved=0>
  https://w3c.github.io/test-results/webauthn/all.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw3c.github.io%2Ftest-results%2Fwebauthn%2Fall.html&data=02%7C01%7CAkshay.Kumar%40microsoft.com%7C823bc80659884e1e239408d666084e46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636808583190983810&sdata=%2BSMVyl%2FVpcPFOQ96KWckn1tuTxrGCPXIXmCEKHI4%2BOI%3D&reserved=0>

This is outside the scope of the spec but note that Safari doesn't
prompt the user to use a key and doesn't active the key if it has a pin.
We had to use a key that did not have a pin set.

I've reviewed every test that Chrome was listed as failing. Overall, I don't believe that the spec should be held up by these test results because I don't believe that the spec need be changed because of them.

There are a number of tests which are failing basically everywhere:

Bad AuthenticatorSelectionCriteria: authenticatorSelection is empty array
Bad AuthenticatorSelectionCriteria: authenticatorSelection is null
Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey is string
Bad rp: id is object
Bad rp: name is null
Bad rp: name is object
Bad user: displayName is null
Bad user: displayName is object
Bad user: name is null
Bad user: name is object
Bad extensions: extensions is null
Bad extensions: extensions is empty Array
Bad extensions: extensions is empty ArrayBuffer

These are putting values of the wrong type into various fields. However, the ECMAScript to WebIDL conversions are very forgiving and I believe these tests are invalid. For example, consider what values are valid for a WebIDL boolean<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fheycam.github.io%2Fwebidl%2F%23es-boolean&data=02%7C01%7CAkshay.Kumar%40microsoft.com%7C823bc80659884e1e239408d666084e46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636808583190993815&sdata=UKvwQmijciQIaB9TRwxAJ51BnpajXucSc9fApYtyECU%3D&reserved=0>. This references TC39 7.1.2<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftc39.github.io%2Fecma262%2F%23sec-toboolean&data=02%7C01%7CAkshay.Kumar%40microsoft.com%7C823bc80659884e1e239408d666084e46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636808583190993815&sdata=FHVRdvNMSyxN7zJ7BqRfxd7J%2Fh9KKTeyw6beqycBgUI%3D&reserved=0> which considers, amongst other things, both {} and [] to be "true".

Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey true
Chromium does not support resident keys until privacy issues are addressed, but this is a CTAP2 issue.

Bad rp: icon is null
Bad rp: icon is empty String
Bad rp: icon is object
Bad user: icon is empty String
Bad user: icon is null
Bad user: icon is object
These are arguably semantic errors due to the requirements of |icon|. However, Chrome does not currently implement icon.

Bad rp: name is empty String
Bad user: name is empty String
 I don't see a prohibition on it being empty.

Bad user: ArrayBuffer id is too long (65 bytes)
(and similar)
Bug in Chromium. Will be fixed by https://chromium-review.googlesource.com/c/chromium/src/+/1385104<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchromium-review.googlesource.com%2Fc%2Fchromium%2Fsrc%2F%2B%2F1385104&data=02%7C01%7CAkshay.Kumar%40microsoft.com%7C823bc80659884e1e239408d666084e46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636808583191003829&sdata=4oQ1mUl1qLD%2B3xifX4g7Zrj1%2B%2BO%2BRF69Okp78VB8UBE%3D&reserved=0>

Bad user: id is empty ArrayBuffer
We agreed on the call of 2018-12-19 that this test should be deleted.

exclude existing credential
I think the test is wrong. It's expecting "NotAllowedError" but the spec says it should be InvalidStateError by my reading.

Bad extensions: extensions is null
Bad extensions: extensions is empty Array
Bad extensions: extensions is empty ArrayBuffer
Bad extensions: malformatted JSON
Bad extensions: JavaScript object
Bad extensions: extension ID too long
Assumes that browsers are parsing all extensions and doing the generic transform to CBOR. I don't believe that any browser intends to do this. Arguably the over-long extension ID could be caught though.

passing credentials.create() with rpId (host and port)
passing credentials.get() with rpId (host and port)
I believe this test is incorrect because RP IDs do not include a port number. (See “It does not itself include a scheme or port” in the spec.)

DataView user id
Works in current Chrome build

Bad pubKeyCredParams: pubKeyCredParams is empty Array
Current Chrome fails this although it was marked as “passing”. It does appear that it's valid for this sequence to be empty. Thus I believe this is a Chrome bug.

Bad pubKeyCredParams: first param has bad alg (42)
Bad pubKeyCredParams: first param has bad alg (0)
I think this test is wrong. It's not invalid to specify an unknown COSEAlgorithmIdentifier: maybe the user will insert a token that understands it.

no credential specified
This failure ("Attempting list without defining credential to test") is coming from within the test itself (helper.js).


Cheers

AGL