Re: VCWG Test Suites

> Implementers that had proprietary implementations were not willing
to send a Docker image that included their implementation code outside
of their organizations.

Perhaps I’m missing something, but I take this to mean that there is a requirement that the test suite must be runnable by someone other than the vendor under test? Is that agreed upon? Because if not then I don’t think this is a concern?

> A number of implementers were not familiar with it and were not
willing to learn / set up Docker just to demonstrate compliance to a
W3C test suite.

Just highlighting that there is an assumption being made here that more implementers would find wrapping their implementation to expose an agreed upon HTTP interface less burdensome than a docker style test harness which I’m personally not convinced is true.

Thanks,
[MATTR website]<https://mattr.global/>

Tobias Looker
MATTR
+64 273 780 461
tobias.looker@mattr.global<mailto:first.last@mattr.global>
[MATTR website]<https://mattr.global/>
[MATTR on LinkedIn]<https://www.linkedin.com/company/mattrglobal>
[MATTR on Twitter]<https://twitter.com/mattrglobal>
[MATTR on Github]<https://github.com/mattrglobal>

This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it – please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

From: Manu Sporny <msporny@digitalbazaar.com>
Date: Monday, 17 April 2023 at 6:15 AM
To: W3C Verifiable Credentials Working Group <public-vc-wg@w3.org>
Subject: Re: VCWG Test Suites
EXTERNAL EMAIL: This email originated outside of our organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.


On Thu, Apr 13, 2023 at 10:31 AM Orie Steele <orie@transmute.industries> wrote:
> I don't think there should be any HTTP requirements for such a test suite, and I think docker & github pages should be all that is needed.

I'm making the following comments w/o my Editor hat on.

I'm merely commenting that this approach has been attempted a few
times before and it didn't scale well. The issue we hit with running
Docker for test suites at that time was:

* A number of implementers were not familiar with it and were not
willing to learn / set up Docker just to demonstrate compliance to a
W3C test suite.

* Those that rejected Docker (most implementations), requested that we
provide a command-line interface, which required us to create a
bespoke command line interface just for the W3C test suite.

* Implementers that had proprietary implementations were not willing
to send a Docker image that included their implementation code outside
of their organizations.

* Since we had to fall back to a command line interface, re-creating
the build environments for each implementation (in the automation
environment) became a non-starter, so we had to back off to just
accepting a standardized report format from implementers.

* Since implementers had to keep their implementations tracking the
W3C test suite, and because the interface into the W3C test suite was
bespoke, implementers stopped demonstrating interop after we were
through the Candidate Recommendation phase.

Things might be different this time, just noting that we've run this
experiment before and it didn't turn out well. I expect that Digital
Bazaar almost certainly won't provide a Docker-based interface to our
software for some of the reasons listed above; it's not a sustainable
practice.

> Regarding the core data model test suite, I don't believe HTTP should be a requirement of that test suite either.

As mentioned here[1], using HTTP isn't a requirement. There are
abstraction points in the vc-data-model-test-suite (and the Data
Integrity test suites) that allow for docker or postman/neuman. The
HTTP interface is how 12+ implementations[2] have chosen to integrate
with each other during the last Jobs For The Future Plugfest, and we'd
rather start the test suite process knowing that we have 12
implementations that are already integrated.

Food for thought.

-- manu

[1]https://github.com/w3c-ccg/community/issues/241#issuecomment-1448921606

[2]https://docs.google.com/presentation/d/19GmJ3bLMrbVadesnkmsWaaUr-U71Y9Kr775tZvgs-xI/edit?pli=1#slide=id.g18a979873b4_2_50


--
Manu Sporny - https://www.linkedin.com/in/manusporny/

Founder/CEO - Digital Bazaar, Inc.
https://www.digitalbazaar.com/