Re: VCWG Test Suites

Hi, reacting to Docker vs HTTP API to test an implementation.

As an implementer, I don't want to deal with Docker nor with HTTP API when
developing/maintaining an implementation. I want to easily setup a testing
suite as a part of the implementation, a suite that could be easily run
every build, even locally.

e.g. Titanium JSON-LD <https://github.com/filip26/titanium-json-ld> is
backed by more than 1500 tests. The JSON-LD 1.1. test suite is the best and
most complete suite I've ever seen. The testing suite allowed me to finish
the implementation in less than two months, taking all the advantage of
TTD.

As a provider of an implementation I could be interested to prove a
compliance

1) by a declaration - self submitted report
2) by exposing HTTP API providing an access to test to 3rd party

Perhaps, there could be two levels of compliance. Declared, self-reported,
- potentially out-of-date if not updated. And "verified" compliance, run by
3rd party at any time, using simple HTTP API.

Best,
Filip






On Mon, Apr 17, 2023 at 6:33 PM Tobias Looker <tobias.looker@mattr.global>
wrote:

> > Implementers that had proprietary implementations were not willing
> to send a Docker image that included their implementation code outside
> of their organizations.
>
>
>
> Perhaps I’m missing something, but I take this to mean that there is a
> requirement that the test suite must be runnable by someone other than the
> vendor under test? Is that agreed upon? Because if not then I don’t think
> this is a concern?
>
>
>
> > A number of implementers were not familiar with it and were not
> willing to learn / set up Docker just to demonstrate compliance to a
> W3C test suite.
>
>
>
> Just highlighting that there is an assumption being made here that more
> implementers would find wrapping their implementation to expose an agreed
> upon HTTP interface less burdensome than a docker style test harness which
> I’m personally not convinced is true.
>
>
>
> Thanks,
>
> [image: MATTR website] <https://mattr.global/>
>
>
>
> *Tobias Looker*
>
> MATTR
>
> +64 273 780 461
> tobias.looker@mattr.global <first.last@mattr.global>
>
> [image: MATTR website] <https://mattr.global/>
>
> [image: MATTR on LinkedIn] <https://www.linkedin.com/company/mattrglobal>
>
> [image: MATTR on Twitter] <https://twitter.com/mattrglobal>
>
> [image: MATTR on Github] <https://github.com/mattrglobal>
>
>
> This communication, including any attachments, is confidential. If you are
> not the intended recipient, you should not read it – please contact me
> immediately, destroy it, and do not copy or use any part of this
> communication or disclose anything about it. Thank you. Please note that
> this communication does not designate an information system for the
> purposes of the Electronic Transactions Act 2002.
>
>
>
> *From: *Manu Sporny <msporny@digitalbazaar.com>
> *Date: *Monday, 17 April 2023 at 6:15 AM
> *To: *W3C Verifiable Credentials Working Group <public-vc-wg@w3.org>
> *Subject: *Re: VCWG Test Suites
>
> EXTERNAL EMAIL: This email originated outside of our organisation. Do not
> click links or open attachments unless you recognise the sender and know
> the content is safe.
>
>
> On Thu, Apr 13, 2023 at 10:31 AM Orie Steele <orie@transmute.industries>
> wrote:
> > I don't think there should be any HTTP requirements for such a test
> suite, and I think docker & github pages should be all that is needed.
>
> I'm making the following comments w/o my Editor hat on.
>
> I'm merely commenting that this approach has been attempted a few
> times before and it didn't scale well. The issue we hit with running
> Docker for test suites at that time was:
>
> * A number of implementers were not familiar with it and were not
> willing to learn / set up Docker just to demonstrate compliance to a
> W3C test suite.
>
> * Those that rejected Docker (most implementations), requested that we
> provide a command-line interface, which required us to create a
> bespoke command line interface just for the W3C test suite.
>
> * Implementers that had proprietary implementations were not willing
> to send a Docker image that included their implementation code outside
> of their organizations.
>
> * Since we had to fall back to a command line interface, re-creating
> the build environments for each implementation (in the automation
> environment) became a non-starter, so we had to back off to just
> accepting a standardized report format from implementers.
>
> * Since implementers had to keep their implementations tracking the
> W3C test suite, and because the interface into the W3C test suite was
> bespoke, implementers stopped demonstrating interop after we were
> through the Candidate Recommendation phase.
>
> Things might be different this time, just noting that we've run this
> experiment before and it didn't turn out well. I expect that Digital
> Bazaar almost certainly won't provide a Docker-based interface to our
> software for some of the reasons listed above; it's not a sustainable
> practice.
>
> > Regarding the core data model test suite, I don't believe HTTP should be
> a requirement of that test suite either.
>
> As mentioned here[1], using HTTP isn't a requirement. There are
> abstraction points in the vc-data-model-test-suite (and the Data
> Integrity test suites) that allow for docker or postman/neuman. The
> HTTP interface is how 12+ implementations[2] have chosen to integrate
> with each other during the last Jobs For The Future Plugfest, and we'd
> rather start the test suite process knowing that we have 12
> implementations that are already integrated.
>
> Food for thought.
>
> -- manu
>
> [1]https://github.com/w3c-ccg/community/issues/241#issuecomment-1448921606
> [2]
> https://docs.google.com/presentation/d/19GmJ3bLMrbVadesnkmsWaaUr-U71Y9Kr775tZvgs-xI/edit?pli=1#slide=id.g18a979873b4_2_50
>
> --
> Manu Sporny - https://www.linkedin.com/in/manusporny/
> Founder/CEO - Digital Bazaar, Inc.
> https://www.digitalbazaar.com/
>