Re: ActivityPub Test Suite?

It could be a useful exercise to imagine… If someone came along selling you an ‘activitypub’ app or service— or a ‘fediverse’ or social ‘web’ thingo— In an open market where people lie, cheat, or bend the truth, how would *you* or *any of us* try to figure out if it was compliant and interoperable and actually interoperating? fraud detection is a hard problem, and seems to have relevance not just to swicg here but other groups at w3c like the credentials community group.

Example of ambiguities in a test suite: Someone is reasonably going to offer an activitypub service that only checks signatures via the (not normatively required in any social w3c TR afaict) http signatures. That’s awesome that they implemented that and it works and protects from MITM. But someone else is going to come along who only creates signatures using jsonld proofs (not nor actively required in any social w3c TR afaict either), and say “well hey, you didn’t *fully* implement the spec in a way we could inter operate, and I did everything required in the TRs”. This is an inevitable consequence of AP et al not being too picky about the crypto and the authorization checks.

That’s not the only layer at which ambiguities arise, and at which some open open sense making could support.

It seems to me that there is much room for elaboration and consensus-building and forking and rejoining on ambiguities like that. These are discussions and experiments that can, and perhaps should, be completely divorced from past test suites, implementation reports, specific programming languages or impermanently resolvable dns names, etc. The output of each little thread here and the knot at each part of shaving the yak might become inputs to the creation of a plurality of test suites, conformance profiles, test suites of those conformance profiles, forks, experiments, etc.
The path to publishing new recommendations on top of the approved TRs is long, and can be established and reinforced brick by brick.

There’s value in the human-readable text of the individual test cases. Maybe even more than any particular ‘test suite’ (any single good test is a better test suite than many inaccurate or flaky tests).
There are many ways to publish on the web.
So shameless call to action: publish some individually-addressable and repayable-via-activitypub test cases or use cases or Q&A on the network itself. :) (each email thread on this list ain’t too bad, it gets an https URI via the mailman archives website)

I wonder if many actors in parallel could propose individual test cases in a human readable, if not yet machine readable or executable, format. Might that be a good way of generating many parallel threads? and for people to get ideas of where to start either debating or documenting or implementing or otherwise interoperating and socializing

This doesn’t need to require central coordination or permission.
we can aggregate test cases across all channels via (#activitystreams or #activitypub) and #test hashtags and/or as2:tags.
(it would be useful to have a test in the test suite asserting that entering hashtags in GUIs result or suggest in adding corresponding as2:tags in an as2 representation).

> On Mar 24, 2023, at 11:17, Bob Wyman <bob@wyman.us> wrote:
> 
> 
> The ActivityPub spec says there is supposed to be a test suite at: https://test.activitypub.rocks/ However, that is no longer the case.
> Does the test suite still exist? If so, where is it?
> Is/Was the test suite's coverage, effectiveness, operation, etc. satisfactory? If not, why not?
> Would it be useful to either improve or replace the old test suite? If so, in what ways?
> Implementation reports, based on the old test suite are provided at: https://activitypub.rocks/implementation-report/ . Should this list, or a copy, be maintained within the SWICG GitHub repository? If not, why not?
> Was there ever a test suite for ActivityStreams?
> Is anyone willing to work on developing a new/improved AP test suite?
> If a new or improved test suite were to be developed, what non-obvious requirements should it satisfy?
> bob wyman
>