Re: Measuring conformance to agent protocols, and a proposed group

Tima,

Thanks for the support, and for the pointer.

The RFC 8785 story is the whole thing. Four divergences found, canonicalizer
fixed, corpus green 10/10, vectors kept as regression tests, then confirmed
against a live endpoint. That is a complete evidence chain and almost
nobody publishes one. If the lab makes that reproducible and reportable,
that is most of the value right there.

Yes to contributing the run-report format and the corpus. I would rather we
converge on one envelope than end up with two, so the same run report works
for us!

One thing I can bring the other way: we keep paired fixtures. A clean tree
that deliberately contains the shapes naive checkers get wrong, and a
dirty tree with the real versions of the same things. Every fix has to keep
passing both. It proves a suite discriminates, not just that it repeats.

Happy to contribute those patterns. I will go through aps-conformance-suite
this week and come back on the list
with specifics.

*Julian*

On Tue, Aug 25, 2026 at 4:37 PM Tima Pidlisnyi <tima@aeoess.com> wrote:

> Hi Julian, all,
>
> I've supported the group.
>
> The reporting-format work overlaps with something we're building in Agent
> Authority Conformance, an LF Decentralized Trust lab for reproducible test
> vectors and run reports around agent authority.
>
> One useful example came out of this CG's #44 thread. Another
> implementation ran our 10 RFC 8785 canonical-byte vectors against its
> production signing path and found four divergences. They fixed the
> canonicalizer, reran the corpus 10/10, added the vectors as regression
> tests, and then confirmed the repaired signing path against their live
> endpoint. That's exactly the kind of cross-implementation evidence I'd like
> the lab to make easy to reproduce and report.
>
> I'm formalizing the run-report format now and would be happy to contribute
> it, along with the corpus, as input to this group.
>
> Repo: https://github.com/Agent-Authority-Conformance/aps-conformance-suite
>
> Tymofii Pidlisnyi
> Agent Passport System (APS)
>
> On Aug 25, 2026, at 10:48 AM, Julian Joseph <julian@thruliquid.com> wrote:
>
> Hi all,
>
> Just joined. One offer and one ask.
>
> The offer: we run a deterministic audit of what web resources actually
> expose to a non-browser agent. 36 checks, no model in the loop, evidence
> recorded for each one. Across 30 real production sites the median scored
> 61.75 out of 100. A protocol is only as good as what deployments actually
> do with it, and we can measure that. Happy to run the cohort against
> anything this group specifies.
>
> The ask: I have proposed a Community Group for the measurement side, Agent
> Conformance and Benchmarking. Conformance reporting formats, runnable test
> suites with adversarial fixtures, versioned corpora. Built to consume this
> group's output, not compete with it.
>
>
> https://www.google.com/url?q=https://www.w3.org/community/blog/2026/08/25/proposed-group-agent-conformance-and-benchmarking-community-group/&source=gmail&ust=1787766514570000&sa=E
>
> It needs five supporters to launch. Support button:
> https://www.google.com/url?q=https://www.w3.org/community/groups/proposed/%23agent-conformance&source=gmail&ust=1787766514570000&sa=E
>
> Julian Joseph
> ApexClaw
>
>
>

-- 
*Julz*
Lead: Global Rev. & Client Success | ThruLiquid
julian@thruliquid.com | https://thruliquid.com/
c: 416-898-5537
"Driving Your Digital Growth"
*Book my calendly <https://calendly.com/julz-/30min>*