Re: VCWG Test Suites

On Thu, Apr 13, 2023 at 10:31 AM Orie Steele <orie@transmute.industries> wrote:
> I don't think there should be any HTTP requirements for such a test suite, and I think docker & github pages should be all that is needed.

I'm making the following comments w/o my Editor hat on.

I'm merely commenting that this approach has been attempted a few
times before and it didn't scale well. The issue we hit with running
Docker for test suites at that time was:

* A number of implementers were not familiar with it and were not
willing to learn / set up Docker just to demonstrate compliance to a
W3C test suite.

* Those that rejected Docker (most implementations), requested that we
provide a command-line interface, which required us to create a
bespoke command line interface just for the W3C test suite.

* Implementers that had proprietary implementations were not willing
to send a Docker image that included their implementation code outside
of their organizations.

* Since we had to fall back to a command line interface, re-creating
the build environments for each implementation (in the automation
environment) became a non-starter, so we had to back off to just
accepting a standardized report format from implementers.

* Since implementers had to keep their implementations tracking the
W3C test suite, and because the interface into the W3C test suite was
bespoke, implementers stopped demonstrating interop after we were
through the Candidate Recommendation phase.

Things might be different this time, just noting that we've run this
experiment before and it didn't turn out well. I expect that Digital
Bazaar almost certainly won't provide a Docker-based interface to our
software for some of the reasons listed above; it's not a sustainable
practice.

> Regarding the core data model test suite, I don't believe HTTP should be a requirement of that test suite either.

As mentioned here[1], using HTTP isn't a requirement. There are
abstraction points in the vc-data-model-test-suite (and the Data
Integrity test suites) that allow for docker or postman/neuman. The
HTTP interface is how 12+ implementations[2] have chosen to integrate
with each other during the last Jobs For The Future Plugfest, and we'd
rather start the test suite process knowing that we have 12
implementations that are already integrated.

Food for thought.

-- manu

[1]https://github.com/w3c-ccg/community/issues/241#issuecomment-1448921606
[2]https://docs.google.com/presentation/d/19GmJ3bLMrbVadesnkmsWaaUr-U71Y9Kr775tZvgs-xI/edit?pli=1#slide=id.g18a979873b4_2_50

-- 
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
https://www.digitalbazaar.com/

Received on Monday, 17 April 2023 13:15:17 UTC