VCWG Test Suites

Hi all,

As we march towards the Candidate Recommendation phase, it's important
for us to have a believable test suite story. The Test Suite we used
in VC 1.0 and 1.1 left much to be desired, a few of those things
being:

* The v1.x test suite was only run a handful of times by implementers
and then not used thereafter. This resulted in implementations
drifting in their conformance over time. If we had to do this again,
we'd adopt a continuous integration / testing approach where the test
suite would be run against all current implementations on a regular
basis.

* It didn't test the "securing" mechanism for VCs, and focused on just
the normative data model statements. We need to test the securing
mechanisms this time around.

* It required implementers to submit test reports manually, even if
they had an automated process for generating the test reports.

Fast forward a few years and many of us have participated in plugfests
that provided better interoperability testing infrastructure that
overcame all of the shortcomings listed above. For example, the recent
Jobs for the Future plugfest demonstrated interoperability between 17
different issuers with an updated test suite infrastructure (see
slides 4 and 5):

https://docs.google.com/presentation/d/19GmJ3bLMrbVadesnkmsWaaUr-U71Y9Kr775tZvgs-xI/edit

Thanks to Andrew Jones and Charles Lehner, we now have a VC Data Model
2.0 test suite that tests 22 of the 44 normative statements in the VC
Data Model 2.0 specification (see example conformance report attached
to this email):

https://github.com/digitalbazaar/vc-data-model-2-test-suite

We also have test suites that can test the VC Data Integrity base
specification as well as each cryptosuite specification:

https://w3c-ccg.github.io/di-ed25519-test-suite/

While most of the implementations today integrate using the VC API,
the test suites are flexible enough that other "drivers" could be
added to support Docker-based workflows as well. These need not be the
only test suites that the WG employs, if others want to contribute
test suites, I imagine those would be welcomed discussion.

VCWG Chairs, this is a request for some call time (main call or
special topic call) to consider the adoption of at least the
vc-data-model-2-test-suite and the di-ed25519-test-suite listed above.

-- manu

-- 
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
News: Digital Bazaar Announces New Case Studies (2021)
https://www.digitalbazaar.com/