Re: VCWG Test Suites

Yes I should further clarify, I too don’t think the test suite needs to require the system under test to expose any specific type of interface e.g docker or HTTP, instead mainly house a set of test vectors and expected results that can be used to test a system, what I understand the JSONLD 1.1 test suite does which I think is a good approach to follow.

Thanks,
[MATTR website]<https://mattr.global/>

Tobias Looker
MATTR
+64 273 780 461
tobias.looker@mattr.global<mailto:first.last@mattr.global>
[MATTR website]<https://mattr.global/>
[MATTR on LinkedIn]<https://www.linkedin.com/company/mattrglobal>
[MATTR on Twitter]<https://twitter.com/mattrglobal>
[MATTR on Github]<https://github.com/mattrglobal>

This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it – please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

From: Orie Steele <orie@transmute.industries>
Date: Monday, 17 April 2023 at 10:19 AM
To: Filip Kolarik <filip26@gmail.com>
Cc: Tobias Looker <tobias.looker@mattr.global>, Manu Sporny <msporny@digitalbazaar.com>, W3C Verifiable Credentials Working Group <public-vc-wg@w3.org>
Subject: Re: VCWG Test Suites
EXTERNAL EMAIL: This email originated outside of our organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.

Indeed,

For VC-JWT, you can submit test report results, even if you don't use docker to generate them.

The idea is that they are "file based", so if you can make "files" you are capable of demonstrating conformance.

OS

On Mon, Apr 17, 2023 at 12:11 PM Filip Kolarik <filip26@gmail.com<mailto:filip26@gmail.com>> wrote:
Hi, reacting to Docker vs HTTP API to test an implementation.

As an implementer, I don't want to deal with Docker nor with HTTP API when developing/maintaining an implementation. I want to easily setup a testing suite as a part of the implementation, a suite that could be easily run every build, even locally.

e.g. Titanium JSON-LD<https://github.com/filip26/titanium-json-ld> is backed by more than 1500 tests. The JSON-LD 1.1. test suite is the best and most complete suite I've ever seen. The testing suite allowed me to finish the implementation in less than two months, taking all the advantage of TTD.

As a provider of an implementation I could be interested to prove a compliance

1) by a declaration - self submitted report
2) by exposing HTTP API providing an access to test to 3rd party

Perhaps, there could be two levels of compliance. Declared, self-reported, - potentially out-of-date if not updated. And "verified" compliance, run by 3rd party at any time, using simple HTTP API.

Best,
Filip






On Mon, Apr 17, 2023 at 6:33 PM Tobias Looker <tobias.looker@mattr.global> wrote:
> Implementers that had proprietary implementations were not willing
to send a Docker image that included their implementation code outside
of their organizations.

Perhaps I’m missing something, but I take this to mean that there is a requirement that the test suite must be runnable by someone other than the vendor under test? Is that agreed upon? Because if not then I don’t think this is a concern?

> A number of implementers were not familiar with it and were not
willing to learn / set up Docker just to demonstrate compliance to a
W3C test suite.

Just highlighting that there is an assumption being made here that more implementers would find wrapping their implementation to expose an agreed upon HTTP interface less burdensome than a docker style test harness which I’m personally not convinced is true.

Thanks,
[MATTR website]<https://mattr.global/>

Tobias Looker
MATTR
+64 273 780 461
tobias.looker@mattr.global<mailto:first.last@mattr.global>
[MATTR website]<https://mattr.global/>
[MATTR on LinkedIn]<https://www.linkedin.com/company/mattrglobal>
[MATTR on Twitter]<https://twitter.com/mattrglobal>
[MATTR on Github]<https://github.com/mattrglobal>

This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it – please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

From: Manu Sporny <msporny@digitalbazaar.com<mailto:msporny@digitalbazaar.com>>
Date: Monday, 17 April 2023 at 6:15 AM
To: W3C Verifiable Credentials Working Group <public-vc-wg@w3.org<mailto:public-vc-wg@w3.org>>
Subject: Re: VCWG Test Suites
EXTERNAL EMAIL: This email originated outside of our organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.


On Thu, Apr 13, 2023 at 10:31 AM Orie Steele <orie@transmute.industries> wrote:
> I don't think there should be any HTTP requirements for such a test suite, and I think docker & github pages should be all that is needed.

I'm making the following comments w/o my Editor hat on.

I'm merely commenting that this approach has been attempted a few
times before and it didn't scale well. The issue we hit with running
Docker for test suites at that time was:

* A number of implementers were not familiar with it and were not
willing to learn / set up Docker just to demonstrate compliance to a
W3C test suite.

* Those that rejected Docker (most implementations), requested that we
provide a command-line interface, which required us to create a
bespoke command line interface just for the W3C test suite.

* Implementers that had proprietary implementations were not willing
to send a Docker image that included their implementation code outside
of their organizations.

* Since we had to fall back to a command line interface, re-creating
the build environments for each implementation (in the automation
environment) became a non-starter, so we had to back off to just
accepting a standardized report format from implementers.

* Since implementers had to keep their implementations tracking the
W3C test suite, and because the interface into the W3C test suite was
bespoke, implementers stopped demonstrating interop after we were
through the Candidate Recommendation phase.

Things might be different this time, just noting that we've run this
experiment before and it didn't turn out well. I expect that Digital
Bazaar almost certainly won't provide a Docker-based interface to our
software for some of the reasons listed above; it's not a sustainable
practice.

> Regarding the core data model test suite, I don't believe HTTP should be a requirement of that test suite either.

As mentioned here[1], using HTTP isn't a requirement. There are
abstraction points in the vc-data-model-test-suite (and the Data
Integrity test suites) that allow for docker or postman/neuman. The
HTTP interface is how 12+ implementations[2] have chosen to integrate
with each other during the last Jobs For The Future Plugfest, and we'd
rather start the test suite process knowing that we have 12
implementations that are already integrated.

Food for thought.

-- manu

[1]https://github.com/w3c-ccg/community/issues/241#issuecomment-1448921606

[2]https://docs.google.com/presentation/d/19GmJ3bLMrbVadesnkmsWaaUr-U71Y9Kr775tZvgs-xI/edit?pli=1#slide=id.g18a979873b4_2_50


--
Manu Sporny - https://www.linkedin.com/in/manusporny/

Founder/CEO - Digital Bazaar, Inc.
https://www.digitalbazaar.com/



--
ORIE STEELE
Chief Technical Officer
www.transmute.industries

[Image removed by sender.]<https://www.transmute.industries/>