Re: Status report re: WebAuth extension interop reporting

Sam,
If a public thread is the best method for you, I am happy to continue our
dialog here.  But please confirm that you have the following individuals
subscribed to this list as they are critical to closing out all the current
open issues:

   - Yuriy Ackermann <yuriy@fidoalliance.org>
   - Rae Hayward <rae@fidoalliance.org>

I also want to be clear about one point I don't see covered in your
summary.  FIDO Alliance requests the extensions be published as normative,
not informative.  I just wanted to be clear about that point.  We stand by
to provide you with whatever documentation you still need to reach that
outcome, recognizing we are all in uncharted waters.  We have obviously
seen hundreds of products tested and publicly listed as having passed those
tests so this is simply a matter of providing the documentation in a format
you need to see it in, and perhaps that requires some leg work on our part
to get permission to share anything currently protected by test-event-NDAs.

Once you have Yuriy and Rae on this mailing list, would you send around a
specific example of the kind of test mapping documentation you are
accustomed to seeing?  That would be helpful.

Best regards,

*Brett McDowell* | Executive Director | FIDO Alliance
<https://fidoalliance.org>
brett@fidoalliance.org | M: +1.413.404.5593 | @FIDOalliance
<https://twitter.com/FIDOalliance>






On Thu, Nov 15, 2018 at 2:48 PM Samuel Weiler <weiler@w3.org> wrote:

> Colleagues,
>
> There are multiple threads going around with long - and different - CC
> lists re: interop testing for the WebAuth extensions.
>
> This has left many people - including this W3C Team Contact - feeling
> confused.  In the interest of improving matters, I'm starting a public
> thread.  Hopefully, to the extent that the matters are not covered by
> NDA, we can quit using CC lists that forget important players.
>
> Below are:
>
> 1) where I think we're at, and
>
> 2) some questions I sent to various people last week, edited to remove
> some context to protect the innocent.
>
> Feel free to correct my understanding as needed.
>
>
> Where I think we're at:
>
> The the extent that the extensions in the base WebAuth spec were
> implemented in UAF, Ralph has agreed to accept interop testing of those
> from the UAF context - rather than require new interop testing specific
> to WebAuth.
>
> Ralph remains willing to publish any or all of the extensions marked as
> informative (non-normative).  They could also be split into a separate
> doc and pushed through at a later time.
>
> W3C has received some documentation of a) which extensions have been
> implemented by multiple UAF devices and b) the names of certified UAF
> implementations.  We do not have a detailed mapping of which
> implementations were shown, though testing, to have interoperable
> versions of which extensions.
>
>
> I have asked for some more detail about the testing - or the
> certification criteria - to reassure us that the extensions have, in
> fact, been tested.
>
> I understand that FIDO, the W3C WG chairs, and others are assembling
> such details.
>
> I urge patience - I think we're in relatively uncharted territory here,
> partly because W3C proposes accepting interop testing based on another
> spec and, more significantly, because FIDO has not provided interop
> reports of the sort we're accustomed to seeing.
>
> Below are the clarifying questions I sent last week.
>
> -- Sam
>
>
> -------- Forwarded Message --------
> Date: Wed, 7 Nov 2018 08:40:48 -0500
> From: Samuel Weiler <weiler@w3.org>
>
> Colleagues,
>
> ...
>
> ... forwarded this thread (or at least a portion of it) and asked me to
> formulate some questions that may help clarify things:
>
> If a product had a non-interoperable implementation of one (or more) of
> these extensions, could it still have been certified by FIDO?
>
> I am concerned that while a product may advertise that it implements an
> extension, FIDO's specific certification requirements are unclear - for
> example, if a product supporting no optional extensions would be
> certified, I can imagine a certification program allowing that product
> to still be certified if it contained an "early" or "pre-release"
> extension implementation that was not (yet) interoperable.  (Perhaps
> related: if a product did not ask for certification re: a particular
> extension, did you test to make sure that extension was not present?)
>
> I think it would help to share specifics: e.g. "implementation X was
> shown to have an interoperable implementation of extension foo". Perhaps
> you have a chart of which implementations were shown to have
> interoperable implementations of which extensions?
>
>
> ...
>
> ... I expect an interop report to contain more detail.
>
> Here are some examples that look more like what I expect.  I'm not
> suggesting you mimic any one of these - they have their own flaws and,
> of course, their methodology may not be applicable - but perhaps you
> already have something more like this that you could share?
>
>
> https://datatracker.ietf.org/meeting/101/materials/slides-101-dots-ietf-101-hackathon-dots-interop-01
> https://tools.ietf.org/html/rfc6984
> https://tools.ietf.org/html/draft-rosen-megaco-interop-1-report-00
>
> -- Sam
>
>
>