Re: [Testing TF] First Meet-Up 18Oct, 15:00UTC

As stewards of AP and AS2, I think the SocialCG have a /responsibility/ 
to pick an official way to test the specs. Automated testing is 
important. No other entity has our status to do so.

It is nice that so many people are working on testing systems, but 
that's probably in part because the original testing systems for AS2 and 
AP are offline. At least for AS2, I know that the as2.rocks site only 
provided a parse of input documents, which wasn't a great test of AS2 
processors in the first place.

One fully-baked cake is better than 5 half-baked cakes. I'd rather have 
one good, working testing system than a handful of unfinished projects, 
even if they are made by nice people with good intentions.

I would like to see the Testing TF make a list of deliverables for 
itself, even if those deliverables are a recommendation for what the CG 
should make. But, ideally for me, I'd love to see the TF take on:

  * A public Web site where you can submit a server for review and get
    an automated score (0 to 100) on spec compliance
  * (Optional) A command-line testing system that can be used in
    development environments or CI

I know that's a lot, but it's what we need.

Evan

P.S. There's a little bit about recording meetings in the W3C process doc:

https://www.w3.org/2023/Process-20230612/#meeting-recording

...which seems in line with your thoughts.


On 2023-10-09 12:50 p.m., Bumblefudge von CASA wrote:
> Dear SocialCG:
>
> I would like to put an event on the SocialCG calendar, and invite to 
> it all interested parties[^1] reading this. The event will be a 
> "meet-up" of the Testing Task Force, with two invited guests. It will 
> be held at 3pm UTC [^2] on Wednesday, 18 October in the 
> recording-friendly[^3] this jitsi room:
> https://meet.w3c-ccg.org/socialcg
>
> For our first meeting, I wanted to hear from longtime fediverse 
> community dev @Helge, who leads the Bovine project and is working on 
> testing tooling in the codeberg project `funfedidev` [^4]. As with all 
> invited guests, I would also like to hear in detail what issues and 
> merge-requests Helge would appreciate in the short term and in the 
> medium term, as he continues to build out his set of tools, hopefully 
> with significant help, inspiration, and review from the rest of us.
>
> In the second half of the hour, I wanted @Helge to share his 
> experiences, challenges, and successes translating specified behaviors 
> (both AP behaviors, FEP behaviors, and other specs from other sources) 
> into plain-language test-cases, a great example of which you can see 
> here [^5]. @Bengo also wanted to share some ideas for systematically 
> identifying which of the AP spec's behaviors[^6] are easier and harder 
> to design plain-language test-cases for; in future meetings, these 
> criteria may help us jump into each other's git repositories 
> constructively, and understand the strengths and weaknesses of 
> complementary approaches.
>
> People have been asking in various places and private messages whether 
> the Task Force has a gameplan, a timeline, a roadmap, a leaderboard. I 
> wanted to assure people that I have none of those things, and that I 
> see my role purely as "convener" of a productive exchange between 
> independent projects in the community, whether funded or unfunded[^7]. 
> I am not here to convince anyone (or be convinced by anyone) of the 
> **best** way to test ActivityPub, or its adjacent specs and 
> extensions. Nor am I interested in helping the W3C pick **one 
> official** test suite, although I do hope to capture helpful feedback 
> on the ambiguities and weak spots of all the specs the CG defines and 
> profiles in the process. My primary goal for this task force is to 
> support and promote helpful testing work in the open source and in the 
> CG's interest, and to provide a platform for people to help each other 
> achieve "rough consensus and running code" on multiple different kinds 
> of testing tools that test whatever they want to test. Please message 
> me if you think those goals are not being met by these meetings and 
> related async channels[^8], or if you think you could help us keep the 
> sync and async work productive.
>
> Sorry for the magnum opus, but most of this can be moved into a primer 
> once this task force has a repo somewhere!
>
> Conveniently yours,
> __@bumblefudge
>
> ----------
> [^1]: Note that this is a meeting of the Social CG and thus is 
> operated within the W3C's IP policies, so if you would like to 
> participate or make any contributions (even offhand opinions!) that 
> could remotely be considered "intellectual property," please 
> officially join the CG before the meeting by creating a w3c.org 
> account and using it to join the Social CG by clicking "join" at 
> https://www.w3.org/community/SocialCG/ . If you cannot join the CG for 
> any reason, still feel invited to attend, but err on the side of 
> silence if you're not sure where your IP boundary is.
> [^2]: That's 8am PST, 11am EST, 5pm CEST this time of year.
> [^3]: For meetings of this task force, I would propose the capacity to 
> record be made available to all guests who *prefer* to have their 
> demos and presentations recorded, with no pressure on those who prefer 
> the opposite. Q&A needs not be recorded and can be more conventionally 
> scribed as the group decides; anyone attending who is uncomfortable 
> being recorded is encouraged to request recording be paused or stopped 
> before speaking, or use jitsi chat instead.
> [^4]: https://helge.codeberg.page/funfedidev/
> [^5]: 
> https://codeberg.org/helge/fediverse-features/src/branch/main/w3c/ap_follow.feature
> [^6]: https://socialweb.coop/activitypub/behaviors/
> [^7]: I am, full disclosure, contributing to one set of testing 
> implements; here is a little background on that project's philosophy 
> for the curious: https://socialweb.coop/blog/additive-testing/
> [^8]: @activitypubtestsuite@venera.social is worth following for 
> announcements and codeberg/gitlab/etc links!
>
>