Test reports and interop (was Re: We've moved on from proposing the "Solid" WG (PUMPKIN WG))

On mandag 25. mars 2024 11:16:21 CET Sjoerd van Groning wrote:
> > Got reference to authoritative/verifiable/reproducible test reports?
> 
> https://solidservers.org/

I have to admit, this is not where I hoped Solid would have gone. Back in the 
day, the vision was to have anchors in the specs for every requirement, so 
that interested parties would have a full coverage report, so that it would be 
easy to verify that every requirement had at least a test, and so that it 
would be easy to see what requirement a test belonged to. In addition, there 
would be a test suite that tested things that weren't in the spec, like HTTP 
interpretations.

Just having a test suite that ticks a box when all tests pass does not have 
much to offer for people wishing to see evidence of actual interop before they 
dive in, then, they would have to first write their app, and then see if it 
works with all servers. I wouldn't have made that investment to be frank.

I would like to see coverage reports, what parts of the spec are covered and 
what are not covered, I would like to see linked a full test report where each 
test is explained for every server. That would be actually transparent.

And I note that we did have this more or less implemented for the protocol 
spec. Coverage wasn't great, and things were so much in flux back then that it 
was too early to publish full reports, but the functionality was more or less 
there.

Kind regards,

Kjetil

Received on Monday, 25 March 2024 12:38:17 UTC