Re: activitypub-testing: a test runner for ActivityPub Actors

On Mon, Jan 15, 2024 at 8:09 PM Johannes Ernst <johannes.ernst@gmail.com>
wrote:

> Hey Ben,
>
> Would it be accurate to describe what you announced here is a set of
> structural correctness tests on the Actor document?
>

Sounds accurate for a subset of what is published, e.g. for
actor-objects-must-have-inbox-outbox-properties
<https://activitypub-testing-website.socialweb.coop/test-cases/actor-objects-must-have-inbox-outbox-properties>

On the other hand, it is an incomplete description of other subsets of what
I shared. For example, I wouldn't say your description is accurate for the
test outbox-wraps-object-with-create-checked-using-get-location
<https://activitypub-testing-website.socialweb.coop/test-cases/outbox-wraps-object-with-create-checked-using-get-location>,
which unlike the last example is testing conformance requirements about the
behavior of an outbox server handling an HTTP Post request, not the
structure of the Actor document.

I wanted to try out a few approaches on the *easiest* to test requirements
first, and ship something useful for those ASAP and to prototype/learn
while building out shared tooling e.g. test case website.
I floated the proposal via blog "Focus on the 'easy to test' ActivityPub
server requirements and improve their testability."
<https://bengo.is/blogging/easy-to-test-activitypub-requirements/>
including rationale,
and then I presented it at the 2023-10-18 SocialCG Testing TF meeting.


> So for example, it seems you are looking whether the Actor document has an
> inbox, and if so, that the inbox has the right type.
>
> But you are not testing that, say, if I post to your inbox, you actually
> receive my Note. (You might be testing 201 Created?)
>

There are a lot of ways of interpreting 'actually receive' here. I'm not
getting into that.

Primary objective right now is to have one test case for each of the ~50
conformance requirements. I think there are at least 10 of those
requirements that aren't relevant to servers, so there may be less than 50.

Here is a web page that groups test cases by requirement into 'requirements
with test cases' and 'requirements that need test cases'.
https://activitypub-testing-website.socialweb.coop/conformance/requirements/tests/

Some requirements may end up with test rules that involve asserting
something like what you mean above. I don't have anything like that to
share right now. Try out other tools for that (see helge's using ngrok) and
i'll let you know when we implement tests that involve that.

activitypub-testing, as a test *runner*, is interested here in the shape of
these test rules and test specifications and related requirements more than
their contents, which the test runner abstracts along. There will
inevitably be a whole landscape of things to focus on across various
dimensions of
1) which behaviors to test and
2) how to interpret the specified behavior into testable assertions and then
3) test rule specification for that interpretation
(see https://www.w3.org/TR/test-methodology/)

>
> My apologies if I’m getting this wrong, I have not had yet the time to
> read your source code in detail, but I’d like to think about it the right
> way.
>
Keep reading and sleep on it. Try it out on a public URL and share the test
results and blog about it so others can help interpret them.

>
> Secondly, how would you compare this to what Helge created at
> https://funfedi.dev/testing_tools/verify_actor/ ?
>
> As much as possible, I wouldn't compare them. I'm a big fan of Helge and
the things I've seen from that effort and others too.

Off the cuff of 5min research, some differences appear to be:

* activitypub-testing is in JavaScript and type-checked with tsc.
pasture_verify_actor is in python and README uses docker
* activitypub-testing runs in a web browser + anywhere node.js runs, and
pasture_verify_actor doesn't
* pasture_verify_actor, to my knowledge, does not run activitypub
conformance tests or have an explicit goal of having 100% conformance
requirement coverage, activitypub-testing does
* activitypub-testing can be used as a library to operate on requirements
and test cases as data and/or run(input) the test, and
activitypub-testing-website demos how to do that. pasture_verify_actor
doesn't seem to have a goal to be useful as a library vs. an application
* pasture_verify_actor doesn't have an html element
* there are a bunch of other things pasture_verify_actor does that
activitypub-testing doesn't do and won't do any time soon (e.g. some auth
signatures that aren't in ActivityPub conformance requirements)
* probably a good idea for any given project to try out both/many/yes/and.
e.g. in your CI pipeline you can run your activitypub server, get a URL for
it, then use that as input to start two jobs: i.e. one job each for
pasture_verify_actor and `activitypub-testing test actor` against the same
URL. They are testing different things. If an implementation passes either,
it's more likely to interop with other impls that pass that test suite. The
more the merrier.

Thanks,
>
🙏

>
> Johannes.
>
>
> Johannes Ernst
>
>
>
>
> On Jan 15, 2024, at 14:17, Benjamin Goering <ben@bengo.co> wrote:
>
> Hey everyone. I hope the new year has been good to everyone.
>
> I'd like to announce
> <https://socialweb.coop/activitypub-testing/announcement/> the
> activitypub-testing <https://socialweb.coop/activitypub-testing/> project.
>
> activitypub-testing is a test runner for ActivityPub Actors.
>
> The goal of this release is to bring the project to the attention of
> anyone who is interested in testing ActivityPub Actors so they can follow
> the project, solicit light testing of the tools and software, and start to
> collect issues about how it works, when it doesn't work, and what other
> ways people might like it to work.
>
> The project homepage has a Getting Started
> <https://socialweb.coop/activitypub-testing/#getting-started> section
> with usage and installation instructions for the `activitypub-testing`
> command line interface (aka 'CLI'), which I expect may be the most common
> interface for ActivityPub implementation developers, shell scripters, or
> anyone else who feels comfortable on a posix command
> <https://en.wikipedia.org/wiki/Command_(computing)> line.
>
> There is also a web UI for activitypub-testing. The Web UI is rendered by
> an HTML Custom Element called `activitypub-actor-tester
> <https://bengo.is/activitypub/projects/activitypub-actor-tester/README/>`.
> You can see a demo of this at
> https://socialweb.coop/activitypub/actor/tester/ which has HTML like
> `<activitypub-actor-tester test="true"
> actor="/"></activitypub-actor-tester>`.
>
> activitypub-testing-website is a subpackage within activitypub-testing
> that is a proof of concept of using activitypub-testing as a library in a
> website, e.g. to make a web page for each test case or conformance
> requirement. An example deployment of activitypub-testing-website is hosted
> at https://activitypub-testing-website.socialweb.coop/
>
> Find source code mirrors for the activitypub-testing subprojects at
> https://codeberg.org/socialweb.coop/
>
> My hope for testing in the ActivityPub ecosystem is that all kinds of
> persons or other actors can be ActivityPub testers, e.g. developers of an
> ActivityPub implementation, everyday end-users who control an Actor hosted
> by a service provider they expect to conform to the protocol, or
> professional conformance testers doing ongoing conformance assessment
> <https://www.nist.gov/standardsgov/conformity-assessment-basics>.
> Furthermore, I hope that there is abundant experimentation, competition,
> consistency, and interoperability amongst many testing tools. We should be
> inspired by the WAI Conformance Evaluation
> <https://www.w3.org/WAI/test-evaluate/conformance/> ecosystem and others.
> <https://www.w3.org/WAI/test-evaluate/conformance/>
>
> -- bengo
>
> Thanks to bumblefudge <bumblefudge@learningproof.xyz> and codenamedmitri <
> dzagidulin@gmail.com> for cooperating on this.
> Thanks to evanpro <evan@prodromou.name> for offering support while
> testing onepage.pub + activitypub-testing.
> Thanks to Christine Lemmer-Webber for amongst other things, building and
> hosting an initial testing tool 5+ years ago and for convening
> <https://socialhub.activitypub.rocks/t/the-activitypub-test-suite/290>
> many helpful discussions about its evolution.
>
>
>