Re: Call for Agenda: Testing Task Force Sync Call, 30Jan

Hi,

Are there any minutes from this meeting please, as I inadvertently missed
it due to changed location and schedules.

Aaron

On Tue, 30 Jan 2024 at 10:39, Benjamin Goering <ben@bengo.co> wrote:

> All of the above sounds great to discuss.
>
> This doesn't need sync agenda time, and is probably too long to digest
> before the meeting. That's cool, it's async-friendly and a status report
> video activitypub-testing overview 2024-01-29
> <https://www.youtube.com/watch?v=fRicTYlTHFk> and has ~5min videos for
> each of 5 parts of how activitypub-testing
> <https://codeberg.org/socialweb.coop/activitypub-testing> and
> activitypub-testing-website
> <https://activitypub-testing-website.socialweb.coop/> works.
> i hope the video transcodes to hd fully w/in an hour or so, sorry it it
> doesn't but I'm going to sleep :P Even without full res you can get the
> gist of most demos and can hear the narration of how things fit together.
> (big caveat: lots is still WIP. I'm sharing here since its relevant
> activity since last TF call, not because anything mentioned is recommended
> for wide use right now)
>
> I'm ready for a break from all that though, and I'm excited to learn from
> other testing approaches on the call
> tty in a few hours
>
>
>
>
>
> On Mon, Jan 29, 2024 at 9:51 PM Bumblefudge <bumblefudge@learningproof.xyz>
> wrote:
>
>> Reminder: in 10.5 hours this call is happening!
>> On 1/24/2024 6:27 PM, Johannes Ernst wrote:
>>
>> Bumblefudge <bumblefudge@learningproof.xyz>
>> <bumblefudge@learningproof.xyz> wrote:
>>
>> Please reply-all to the list if you would like to put something on the
>> agenda.
>>
>>
>> 1. I can provide a brief update on the status of FediTest. There is some
>> running code, but still experimental and a bit too early to show-and-tell.
>>
>> Excellent!
>>
>> 2. I could also provide an overview over the responses to my survey on
>> developers and their development setup and virtualization. There are 44
>> responses so far, some of it is unexpected (to me). (A number that
>> surprised me!) Survey is here:
>> https://apps.dazzlelabs.net/nextcloud/apps/forms/s/ed2WBPzrrWFcWKjT5a9MwMGp
>>
>> Nice!
>>
>>
>> More importantly, echo’ing some of what Marcus said:
>>
>> On Jan 23, 2024, at 01:17, Marcus Rohrmoser <me.swicg@mro.name>
>> <me.swicg@mro.name> wrote:
>>
>> Yes, however I am unsure how to phrase it.
>>
>> 1. IMO the tests should benefit the netizens, operators and developers
>> and therefore should be easy to operate and friendly to ad-hoc deploy
>> (rather than 24/7) for unprevileged (non-root) netizens. Without vendor
>> lock in. And tomorrow and the week after as well, without permanent
>> underlying framework changes to be integrated (upwards compatibility. I
>> know it's hard). So I advocate thinking from the end towards the means, not
>> vice versa. What options are there aside php and cgis?
>>
>> I'd love an unpacking of this paragraph to be agenda item, since I don't
>> understand it. I love the idea that netizens and operators/admins have as
>> much at stake as developers, but I'm not sure I can picture what state of
>> affairs would benefit them equally, much less work backwards from it?
>> Should be a fun discussion!
>>
>> 2. Many ActivityPub processes are asynchronous in nature and are hard to
>> follow and therefore test. IMO we should encourage a 'friendly feedback'
>> policy to
>> a. immediately report back the effect in a machine readable manner, evtl.
>> with an url to track the progress,
>> b, if asynchronous, then call back once there is a result,
>> c. never silently discard requests.
>>
>>
>> This sounds like something socialweb.coop has been grappling with
>> lately: many of the "harder to test" requirements in AP require results
>> other than pass/fail, and complex layered results. Would be good to sanity
>> check the tentative approach we've been working on, love the name "friendly
>> feedback" :D
>>
>> 3. I think it would be useful to spend some time on the requirements side
>> of testing for the Fediverse.
>>
>> We sometimes wave “testing!!!” around as some kind of magic wand, but as
>> we can see from the various projects, I don’t think we quite agree on what
>> testing should be done, and in particular Why. Also: Who needs it and what
>> does it need to look like so they can get the maximum benefit out of a
>> testing environment and the results of testing?
>>
>> The only magic wands I wave around are the word "composability" and
>> "friendly fork"😉
>>
>>
>> 4. Another important subject would be: just where exactly do those tests
>> come from? Who decides what is and isn’t a valid test? That’s particularly
>> important because AS and AP are so flexible. Example:
>>
>> * if anybody gets to do anything allowed by AS and AP, hopes of
>> out-of-the-box interop as low, and testing tells developers who want
>> real-world interoperability fairly little.
>>
>> * if “passing all tests” is supposed to mean “will interoperate with 90%
>> of the installed fediverse base”, then many tests have to be defined that
>> do not have a root in a W3C or other standards document, but test for
>> conventions deployed by the leading implementations.
>>
>> (Personally I believe we need to have both, and tests needs to be
>> organized in a very modular fashion based on the “authority” from which
>> they are derived.)
>>
>> Totally agree! The user running the tests decides which tests are valid,
>> and will ignore any tests they disagree with unless we make it very easy
>> for them to turn them off and replace them with tests they find valid.
>> Ideally, if we serve those users well, they might even "upstream" their
>> modifications and make our simple test suites a borgesian garden of forking
>> paths and optionality😅
>>
>> I don't think the scope of this CG's testing task force is or should be
>> limited to "writing tests for this CG's ratified specifications", and I
>> want to support people writing profiles of IETF specs or community/living
>> documents not rooted in SDO authority. The only thing that strikes me as
>> out-of-scope (at least of tomorrow's meeting) would be giving the CG's
>> implicit blessing to, or donating its finite bandwidth to, specific
>> implementations or platforms (even if non-commercial!). Maybe I'm being
>> more catholic than the pope, though? We could always scope calls with a
>> single-implementation/platform API focus and declare that scope before
>> scheduling, if there's demand for it? I'm just the facilitator and
>> note-taker, the users and contributors set the agenda here.
>>
>>
>> So plenty to talk about from my perspective :-)
>>
>> Cheers,
>>
>>
>>
>> Johannes.
>>
>> Johannes Ernst
>>
>> Fediforum <https://fediforum.org/>
>> Dazzle Labs <https://dazzlelabs.net/>
>>
>>

-- 
Aaron Gray - @AaronNGray@fosstodon.org

Independent Open Source Software Engineer, Computer Language Researcher,
Information Theorist, and Computer Scientist.