Re: Announcing canivc.com

On Tue, Mar 19, 2024 at 8:38 AM John, Anil <anil.john@hq.dhs.gov> wrote:
> I also realize that this is a work in progress, so wanted to offer some (random) feedback.

Yes, very much a work in progress and we're seeking additional
feedback from the community.

As Benjamin noted, we've already spoken with around 40 of the people
in the Verifiable Credentials community, as well as organizations that
are deploying Verifiable Credentials. We're in the hundreds of hours
of community outreach to get us to where we are today with
canivc.com... and, we'll need more to ensure that this meets the needs
of organizations that are considering deploying this technology.

Prioritizing the types of people whom the site is supposed to serve is
important. At present, the priorities are, in order:

1. CTOs and other similar decision making executives that are
considering deploying Verifiable Credentials and need to make
technology selections based on what is actively supported in the
market.
2. Software developers that need to make a decision on what software
they need to deploy.
3. Software implementers that need to prove that their software does,
provably, follow the standards and are interoperable with other
software systems in the ecosystem.

In other words, this is about Verifiable Interoperability.

> I am not sure how to interpret the number of tests associated with each issuer and each verifier on the home page; I am unsure of the value provided by the total number of tests a particular entity has chosen to implement.

Yes, we've struggled with that aspect of the site. Just because an
implementer implements a lot of technologies/tests doesn't mean
they're the best implementation for a particular use case. If all you
(the one deploying a VC use case) cares about is a small sliver of the
ecosystem, the only implementers that matter to you are the ones that
are implementing that sliver of the ecosystem.

Our hope is that, in time, we'll be able to get to that level of
granularity. A lot of the data is there in the tests today, but we'll
have to iterate on packaging each one of those tests up into a logical
"feature" or "capability" set.

> I am also wondering if the distinction should be passing/failing a test (if that test is of a feature in a spec that is required) and an entity making a conscious decision that there is a particular option in the standard that they are supporting/not-supporting.

We do have a "Not implemented" test outcome where a particular
implementation can signal that they have not implemented an optional
feature. We don't have many (any?) examples of that today because the
sorts of tests we're writing for the W3C test suites are mostly for
mandatory features in specifications that either one implements fully,
or not at all.

We do need to put more thinking and work into optional feature sets,
but from what I gather today, those "optional" feature sets are entire
specifications that an implementer either chooses to implement or does
not. We only test "MUST" statements today, and don't test "SHOULD"
statements. There is a desire to implement the SHOULD statements,
which would start reporting more of the "optional" parts of the
specifications, but given limited time and funding, those are
currently a secondary priority.

> An example of this in the VCDM 2.0 is that you are required to either support an embedded proof model (Data Integrity) or an enveloped proof model (JOSE/COSE).

Yes, in this case, the spider chart will show a particular implementer
as either supporting one, or the other, or both. From the perspective
of this sort of optionality, it shows up as implementing less or more
tests.

Perhaps the way to talk about this is the granularity in which
"optional" testing happens. At present, the granularity is at the
specification level -- you either implement all the mandatory features
in a specification, or you don't. That is, you either implement a
specific Data Integrity cryptosuite, or you don't.

In the future, we can support optional features within a
specification. Arguably, we can do a variation of this today (via the
"tags" one uses in an implementation):

https://github.com/w3c/vc-test-suite-implementations/blob/main/implementations/DigitalBazaar.json#L28
https://github.com/w3c/vc-test-suite-implementations/blob/main/implementations/DanubeTech.json#L15
https://github.com/w3c/vc-test-suite-implementations/blob/main/implementations/SpruceID.json#L17

There is always room to improve this in the future, though.

> An entity making a choice to support one option only for reasons that are their own is a perfectly rational one and should not come with any negative implications.

Yes, agreed.

Ultimately, we've modeled canivc.com off of a site called caniuse.com,
which is a popular website used by web developers to determine if they
can use a particular feature in a web browser. I will note that each
browser engine does have a score on the main page and that web
developers have found that useful over time (for a variety of
reasons):

https://caniuse.com/

Fundamentally, though, what you are saying makes sense -- the only
technology choices that are relevant to an organization deploying this
technology are the ones that support their use case. It is, also,
useful to understand if one cared about "broader support", what the
other choices might be. Picking a solution that implements a narrow
selection of features could also box an organization into a technology
vertical that is inflexible as the market changes.

> Some of the ways that I've seen people game tests is to (1) self-select a specific set of tests/features that they implement so that every test shows as passing and (2) add on a whole bunch of tests beyond what is in the standard/spec test suite itself to show how wonderful they are with everything they support in their product (more is not necessarily better)

Yes, exactly.

> Ability to mitigate these types of gaming would be welcome with a possible future outcome whereby I have the ability to select the particular test suites that are of interest to me, and have the ability run them against a (set of) implementers.

Yes, that is the goal, and there is an aspect to this that is a cat
and mouse game... vendors game test suites to win business, and
knowing that going into this is of utmost importance. We need to
decide, as a community, how to convey this information to ensure that
organizations deploying this technology have a good experience (and,
as a result, grow the ecosystem).

The other aspect to the test suite that I believe is vital is the
"continuous testing" methodology. Just because a software
implementation passed the test suite a month ago, doesn't mean it
passes it today. Continuous testing, against production software
products and libraries, AND cross-testing against vendor
implementations, is a goal of this set of interoperability testing
that is currently lacking in many other testing frameworks.

The browser test suites are run on a nightly basis, against the latest
builds of all of the browsers. They are an effective smoke test for
all developers to understand how a particular implementation is doing
TODAY... not one, three, or six months ago (which is how a lot of
certifications are performed).

> Going forward, I would love to be able to use these tests to find answers to questions like the following:
>
> Show me the subset of implementers that can issue VCDM 2.0 + Data Integrity credentials + proof sets
> Show me the sub-set of implementers that can issue VCDM 2.0 + JOSE/COSE credentials
>
> Show me the intersection between (1) and (2)
>
> Show me the entities that if I choose to go down (1) or (2), have products that can verify those credential/proof combos.

Yep, getting answers to questions like that is an explicit design goal
of the canivc.com site. It'll take us a while to get there, but the
most important step -- having a framework and raw data to work from,
is the accomplishment Benjamin was announcing this week.

> Needless to say, this is a great and useful and I am looking forward to the incorporation of additional test suites to provide a more comprehensive picture of decentralized identity implementations.

+1, we look forward to others contributing to the testing architecture
as it's decentralized. The site doesn't care who wrote the tests or
where they are run, it reads in a test result file, from anywhere on
the Web, and integrates it into the dashboard views and spiderchart
views. We believe that this allows other communities to write their
own tests and test output and get integrated into a larger
multi-community view of the state of interoperability among Verifiable
Credentials.

-- manu

-- 
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
https://www.digitalbazaar.com/

Received on Tuesday, 19 March 2024 13:35:43 UTC