Re: Test reports and interop (was Re: We've moved on from proposing the "Solid" WG (PUMPKIN WG))

Kjetil
The work you refer to is still ongoing. There are over 600 tests but
coverage is still not complete - we were hoping for more community
contributions to this.
* There is a QA spec defining how we manage and report on tests:
https://solidproject.org/ED/qa
* The link you were looking at is for results from one of the test suites,
the other test suite is here:
https://github.com/solid-contrib/specification-tests/
* This produces a coverage report which includes the protocol spec and
access control elements common to both the WAC and ACP specs:
https://solid-contrib.github.io/specification-tests/coverage
Pete

On Mon, 25 Mar 2024 at 12:38, Kjetil Kjernsmo <kjetil@kjernsmo.net> wrote:

> On mandag 25. mars 2024 11:16:21 CET Sjoerd van Groning wrote:
> > > Got reference to authoritative/verifiable/reproducible test reports?
> >
> > https://solidservers.org/
>
> I have to admit, this is not where I hoped Solid would have gone. Back in
> the
> day, the vision was to have anchors in the specs for every requirement, so
> that interested parties would have a full coverage report, so that it
> would be
> easy to verify that every requirement had at least a test, and so that it
> would be easy to see what requirement a test belonged to. In addition,
> there
> would be a test suite that tested things that weren't in the spec, like
> HTTP
> interpretations.
>
> Just having a test suite that ticks a box when all tests pass does not
> have
> much to offer for people wishing to see evidence of actual interop before
> they
> dive in, then, they would have to first write their app, and then see if
> it
> works with all servers. I wouldn't have made that investment to be frank.
>
> I would like to see coverage reports, what parts of the spec are covered
> and
> what are not covered, I would like to see linked a full test report where
> each
> test is explained for every server. That would be actually transparent.
>
> And I note that we did have this more or less implemented for the protocol
> spec. Coverage wasn't great, and things were so much in flux back then
> that it
> was too early to publish full reports, but the functionality was more or
> less
> there.
>
> Kind regards,
>
> Kjetil
>
>
>
>

-- 
This e-mail, and any attachments thereto, is intended only for use by the 
addressee(s) named herein and may contain legally privileged, confidential 
and/or proprietary information. If you are not the intended recipient of 
this e-mail (or the person responsible for delivering this document to the 
intended recipient), please do not disseminate, distribute, print or copy 
this e-mail, or any attachment thereto. If you have received this e-mail in 
error, please respond to the individual sending the message, and 
permanently delete the email.