Re: DID spec Test Suite

On Fri, Jun 7, 2019 at 7:05 AM Manu Sporny <msporny@digitalbazaar.com>
wrote:

> Hi all,
>
> Andrew Jones (cc'd) is currently working on a DID specification test
> suite and has made a full pass on the spec, pulling out every normative
> statement that needs to be tested.
>
> The good news is that we have a kernel of a set of tests that we can
> implement now and try to demonstrate interop between DID libraries that
> produce and validate DIDs and DID Documents. Hooray. We expect the test
> suite to be ready in two weeks with tests for 26 of the normative
> statements in the specification (roughly ~60 tests).
>
> The bad news is that there are an additional 83 normative statements
> that are not testable.


So you raise a question that may expose my ignorance about W3C specs. I was
not aware that all normative requirements have to be machine-testable. If
that's the case, no wonder there are 83 normative requirements that are not
"testable". There are a whole set (I would say half the specification or
more) that are normative requirements on DID method specifications. None of
those are machine-testable. But they are certainly human-testable (that's
the whole point).

What are we supposed to do about human-testable normative requirements?


> For example:
>
> """
> A DID is a new type of URL that SHOULD NOT require a centralized
> authority to register, resolve or deactivate, nor to update associated
> data.
> """
>
> So, how do you write a test to ensure that a Decentralized Identifier
> Registry is not centralized?
>

This is a good example of a human-testable requirement that we have
recently agreed may not be quite as clear as we'd hoped. So it may need to
change to point to the rubrics. Still, the vast majority of the rubrics are
human-testable as well. Does this mean they don't have value?


>
> There are other statements that have more to do with DID Resolution than
> the core DID Document and should probably be moved to the DID Resolution
> specification. For example:
>
> """
> Therefore DID resolvers and other client applications SHOULD validate
> that keys were not expired at the time they were used.
> """
>
> That statement is testable, but probably belongs in the DID Resolution
> specification.
>

Agreed.


>
> ... and other statements that have to do with requirements on DID Method
> specifications (and cannot be automatically tested, so there will be a
> strong argument to remove the normative statement from the specification
> when it goes standards-track). For example:
>
> """
> The DID method specification MUST specify how a client creates a DID and
> its associated DID Document on the Decentralized Identifier Registry,
> including all cryptographic operations necessary to establish proof of
> control.
> """
>

Wait a minute. There are whole bunch of specs that require that the authors
of other specs do certain things if they want to be conformant. One example
is requiring the registration of a MIME type with IANA.

In our case, one of the main purposes of the DID spec is to specify the
requirements for DID method specs.


> Just putting this out there so that others are aware of the current
> state of things related to the test suite.
>

Sounds like this bears discussion on the main CCG call, or at least on this
week's DID spec call.


>
> The mistake we made with the Verifiable Credentials test suite was
> waiting until we were in the W3C Standard Candidate Recommendation phase
> to put together the test suite. We are trying to fix this issue by
> making sure we have a test suite going into the DID Working Group.
>
> The challenge is going to be augmenting, moving, or rewording those 83
> normative statements in a timely fashion. During the early days of the
> VC WG, I went through and reworded/stripped many of those statements. My
> hope is that we can depend on the editors to do the right thing and not
> block them from making these changes. An extended discussion on 83
> normative statements will slow progress to a crawl, and we don't want
> that right now.
>
> To be clear, we can do this work when the DID WG is under way... but as
> with all things, it's easier to do these sorts of changes earlier than
> later.
>
> Just sharing this information with the group... we'll have to figure out
> how to manage this as the months roll on.
>

Agreed that the earlier we tackle, the better. Who at W3C can advise us on
this subject? Wendy?

=D


>
> --
> Manu Sporny (skype: msporny, twitter: manusporny)
> Founder/CEO - Digital Bazaar, Inc.
> blog: Veres One Decentralized Identifier Blockchain Launches
> https://tinyurl.com/veres-one-launches
>
>