Re: web-platform-tests results

On Wed, Dec 19, 2018 at 9:27 AM Philippe Le Hégaret <> wrote:

> On 12/17/2018 8:23 AM, Philippe Le Hégaret wrote:
> >> Also you should be able to test with safari now with their preview
> >> announcement the other day.
> >
> > I don't have the proper computer for that unfortunately. If someone else
> > has the preview, a key, and a python install, I'm happy to do remote
> > assistance to get those results.
> Sam and I managed to generate test results for safari tech preview 71.
> Added in
> This is outside the scope of the spec but note that Safari doesn't
> prompt the user to use a key and doesn't active the key if it has a pin.
> We had to use a key that did not have a pin set.

I've reviewed every test that Chrome was listed as failing. Overall, *I
don't believe that the spec should be held up by these test results*
because I don't believe that the spec need be changed because of them.

There are a number of tests which are failing basically everywhere:

Bad AuthenticatorSelectionCriteria: authenticatorSelection is empty array
> Bad AuthenticatorSelectionCriteria: authenticatorSelection is null
> Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey is
> string
> Bad rp: id is object
> Bad rp: name is null
> Bad rp: name is object
> Bad user: displayName is null
> Bad user: displayName is object
> Bad user: name is null
> Bad user: name is object
> Bad extensions: extensions is null
> Bad extensions: extensions is empty Array
> Bad extensions: extensions is empty ArrayBuffer

These are putting values of the wrong type into various fields. However,
the ECMAScript to WebIDL conversions are very forgiving and I believe these
tests are invalid. For example, consider what values are valid for a WebIDL
boolean <>. This references TC39
7.1.2 <> which considers,
amongst other things, both {} and [] to be "true".

Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey true

Chromium does not support resident keys until privacy issues are addressed,
but this is a CTAP2 issue.

Bad rp: icon is null
> Bad rp: icon is empty String
> Bad rp: icon is object
> Bad user: icon is empty String
> Bad user: icon is null
> Bad user: icon is object

These are arguably semantic errors due to the requirements of |icon|.
However, Chrome does not currently implement icon.

Bad rp: name is empty String

Bad user: name is empty String

 I don't see a prohibition on it being empty.

Bad user: ArrayBuffer id is too long (65 bytes)

(and similar)

Bug in Chromium. Will be fixed by

Bad user: id is empty ArrayBuffer

We agreed on the call of 2018-12-19 that this test should be deleted.

exclude existing credential

I think the test is wrong. It's expecting "NotAllowedError" but the spec
says it should be InvalidStateError by my reading.

Bad extensions: extensions is null
> Bad extensions: extensions is empty Array
> Bad extensions: extensions is empty ArrayBuffer
> Bad extensions: malformatted JSON
> Bad extensions: JavaScript object
> Bad extensions: extension ID too long

Assumes that browsers are parsing all extensions and doing the generic
transform to CBOR. I don't believe that any browser intends to do this.
Arguably the over-long extension ID could be caught though.

passing credentials.create() with rpId (host and port)
> passing credentials.get() with rpId (host and port)

I believe this test is incorrect because RP IDs do not include a port
number. (See “It does not itself include a scheme or port” in the spec.)

DataView user id

Works in current Chrome build

Bad pubKeyCredParams: pubKeyCredParams is empty Array

Current Chrome fails this although it was marked as “passing”. It does
appear that it's valid for this sequence to be empty. Thus I believe this
is a Chrome bug.

Bad pubKeyCredParams: first param has bad alg (42)
> Bad pubKeyCredParams: first param has bad alg (0)

I think this test is wrong. It's not invalid to specify an unknown
COSEAlgorithmIdentifier: maybe the user will insert a token that
understands it.

no credential specified

This failure ("Attempting list without defining credential to test") is
coming from within the test itself (helper.js).



Received on Wednesday, 19 December 2018 23:17:55 UTC