Re: Call for Agenda: Testing Task Force Sync Call, 30Jan

Bumblefudge,

Wonderful thanks, if you can get the video link/uploaded too that would be
great.

So I can put names/faces to voices.

Aaron

On Mon, 12 Feb 2024 at 19:31, Bumblefudge <bumblefudge@learningproof.xyz>
wrote:

> Hey Aaron:
>
> Sorry for the delayed response.  Minutes for this testing TF call (and
> hopefully all future ones as well, unless I forget one) are here:
> https://github.com/swicg/meetings
>
> I'm not sure what happened with the transcoded video bengo mentioned, but
> I can throw the video or a link to it up there in the relevant meeting's
> directory/minutes doc when I get my hands on it.
>
> Thanks!
> __bumble
> On 2/8/2024 5:05 PM, Aaron Gray wrote:
>
> Hi,
>
> Are there any minutes from this meeting please, as I inadvertently missed
> it due to changed location and schedules.
>
> Aaron
>
> On Tue, 30 Jan 2024 at 10:39, Benjamin Goering <ben@bengo.co> wrote:
>
>> All of the above sounds great to discuss.
>>
>> This doesn't need sync agenda time, and is probably too long to digest
>> before the meeting. That's cool, it's async-friendly and a status report
>> video activitypub-testing overview 2024-01-29
>> <https://www.youtube.com/watch?v=fRicTYlTHFk> and has ~5min videos for
>> each of 5 parts of how activitypub-testing
>> <https://codeberg.org/socialweb.coop/activitypub-testing> and
>> activitypub-testing-website
>> <https://activitypub-testing-website.socialweb.coop/> works.
>> i hope the video transcodes to hd fully w/in an hour or so, sorry it it
>> doesn't but I'm going to sleep :P Even without full res you can get the
>> gist of most demos and can hear the narration of how things fit together.
>> (big caveat: lots is still WIP. I'm sharing here since its relevant
>> activity since last TF call, not because anything mentioned is recommended
>> for wide use right now)
>>
>> I'm ready for a break from all that though, and I'm excited to learn from
>> other testing approaches on the call
>> tty in a few hours
>>
>>
>>
>>
>>
>> On Mon, Jan 29, 2024 at 9:51 PM Bumblefudge <
>> bumblefudge@learningproof.xyz> wrote:
>>
>>> Reminder: in 10.5 hours this call is happening!
>>> On 1/24/2024 6:27 PM, Johannes Ernst wrote:
>>>
>>> Bumblefudge <bumblefudge@learningproof.xyz>
>>> <bumblefudge@learningproof.xyz> wrote:
>>>
>>> Please reply-all to the list if you would like to put something on the
>>> agenda.
>>>
>>>
>>> 1. I can provide a brief update on the status of FediTest. There is some
>>> running code, but still experimental and a bit too early to show-and-tell.
>>>
>>> Excellent!
>>>
>>> 2. I could also provide an overview over the responses to my survey on
>>> developers and their development setup and virtualization. There are 44
>>> responses so far, some of it is unexpected (to me). (A number that
>>> surprised me!) Survey is here:
>>> https://apps.dazzlelabs.net/nextcloud/apps/forms/s/ed2WBPzrrWFcWKjT5a9MwMGp
>>>
>>> Nice!
>>>
>>>
>>> More importantly, echo’ing some of what Marcus said:
>>>
>>> On Jan 23, 2024, at 01:17, Marcus Rohrmoser <me.swicg@mro.name>
>>> <me.swicg@mro.name> wrote:
>>>
>>> Yes, however I am unsure how to phrase it.
>>>
>>> 1. IMO the tests should benefit the netizens, operators and developers
>>> and therefore should be easy to operate and friendly to ad-hoc deploy
>>> (rather than 24/7) for unprevileged (non-root) netizens. Without vendor
>>> lock in. And tomorrow and the week after as well, without permanent
>>> underlying framework changes to be integrated (upwards compatibility. I
>>> know it's hard). So I advocate thinking from the end towards the means, not
>>> vice versa. What options are there aside php and cgis?
>>>
>>> I'd love an unpacking of this paragraph to be agenda item, since I don't
>>> understand it. I love the idea that netizens and operators/admins have as
>>> much at stake as developers, but I'm not sure I can picture what state of
>>> affairs would benefit them equally, much less work backwards from it?
>>> Should be a fun discussion!
>>>
>>> 2. Many ActivityPub processes are asynchronous in nature and are hard to
>>> follow and therefore test. IMO we should encourage a 'friendly feedback'
>>> policy to
>>> a. immediately report back the effect in a machine readable manner,
>>> evtl. with an url to track the progress,
>>> b, if asynchronous, then call back once there is a result,
>>> c. never silently discard requests.
>>>
>>>
>>> This sounds like something socialweb.coop has been grappling with
>>> lately: many of the "harder to test" requirements in AP require results
>>> other than pass/fail, and complex layered results. Would be good to sanity
>>> check the tentative approach we've been working on, love the name "friendly
>>> feedback" :D
>>>
>>> 3. I think it would be useful to spend some time on the requirements
>>> side of testing for the Fediverse.
>>>
>>> We sometimes wave “testing!!!” around as some kind of magic wand, but as
>>> we can see from the various projects, I don’t think we quite agree on what
>>> testing should be done, and in particular Why. Also: Who needs it and what
>>> does it need to look like so they can get the maximum benefit out of a
>>> testing environment and the results of testing?
>>>
>>> The only magic wands I wave around are the word "composability" and
>>> "friendly fork"😉
>>>
>>>
>>> 4. Another important subject would be: just where exactly do those tests
>>> come from? Who decides what is and isn’t a valid test? That’s particularly
>>> important because AS and AP are so flexible. Example:
>>>
>>> * if anybody gets to do anything allowed by AS and AP, hopes of
>>> out-of-the-box interop as low, and testing tells developers who want
>>> real-world interoperability fairly little.
>>>
>>> * if “passing all tests” is supposed to mean “will interoperate with 90%
>>> of the installed fediverse base”, then many tests have to be defined that
>>> do not have a root in a W3C or other standards document, but test for
>>> conventions deployed by the leading implementations.
>>>
>>> (Personally I believe we need to have both, and tests needs to be
>>> organized in a very modular fashion based on the “authority” from which
>>> they are derived.)
>>>
>>> Totally agree! The user running the tests decides which tests are valid,
>>> and will ignore any tests they disagree with unless we make it very easy
>>> for them to turn them off and replace them with tests they find valid.
>>> Ideally, if we serve those users well, they might even "upstream" their
>>> modifications and make our simple test suites a borgesian garden of forking
>>> paths and optionality😅
>>>
>>> I don't think the scope of this CG's testing task force is or should be
>>> limited to "writing tests for this CG's ratified specifications", and I
>>> want to support people writing profiles of IETF specs or community/living
>>> documents not rooted in SDO authority. The only thing that strikes me as
>>> out-of-scope (at least of tomorrow's meeting) would be giving the CG's
>>> implicit blessing to, or donating its finite bandwidth to, specific
>>> implementations or platforms (even if non-commercial!). Maybe I'm being
>>> more catholic than the pope, though? We could always scope calls with a
>>> single-implementation/platform API focus and declare that scope before
>>> scheduling, if there's demand for it? I'm just the facilitator and
>>> note-taker, the users and contributors set the agenda here.
>>>
>>>
>>> So plenty to talk about from my perspective :-)
>>>
>>> Cheers,
>>>
>>>
>>>
>>> Johannes.
>>>
>>> Johannes Ernst
>>>
>>> Fediforum <https://fediforum.org/>
>>> Dazzle Labs <https://dazzlelabs.net/>
>>>
>>>
>
> --
> Aaron Gray - @AaronNGray@fosstodon.org
>
> Independent Open Source Software Engineer, Computer Language Researcher,
> Information Theorist, and Computer Scientist.
>
>

-- 
Aaron Gray - @AaronNGray@fosstodon.org

Independent Open Source Software Engineer, Computer Language Researcher,
Information Theorist, and Computer Scientist.