Re: Measuring conformance to agent protocols, and a proposed group

Hi Julian, all,

I've supported the group.

The reporting-format work overlaps with something we're building in Agent Authority Conformance, an LF Decentralized Trust lab for reproducible test vectors and run reports around agent authority.

One useful example came out of this CG's #44 thread. Another implementation ran our 10 RFC 8785 canonical-byte vectors against its production signing path and found four divergences. They fixed the canonicalizer, reran the corpus 10/10, added the vectors as regression tests, and then confirmed the repaired signing path against their live endpoint. That's exactly the kind of cross-implementation evidence I'd like the lab to make easy to reproduce and report.

I'm formalizing the run-report format now and would be happy to contribute it, along with the corpus, as input to this group.

Repo: https://github.com/Agent-Authority-Conformance/aps-conformance-suite

Tymofii Pidlisnyi
Agent Passport System (APS)

> On Aug 25, 2026, at 10:48 AM, Julian Joseph <julian@thruliquid.com> wrote:
> 
> Hi all,
> 
> Just joined. One offer and one ask.
> 
> The offer: we run a deterministic audit of what web resources actually expose to a non-browser agent. 36 checks, no model in the loop, evidence recorded for each one. Across 30 real production sites the median scored 61.75 out of 100. A protocol is only as good as what deployments actually do with it, and we can measure that. Happy to run the cohort against anything this group specifies.
> 
> The ask: I have proposed a Community Group for the measurement side, Agent Conformance and Benchmarking. Conformance reporting formats, runnable test suites with adversarial fixtures, versioned corpora. Built to consume this group's output, not compete with it.
> 
> https://www.google.com/url?q=https://www.w3.org/community/blog/2026/08/25/proposed-group-agent-conformance-and-benchmarking-community-group/&source=gmail&ust=1787766514570000&sa=E
> 
> It needs five supporters to launch. Support button: https://www.google.com/url?q=https://www.w3.org/community/groups/proposed/%23agent-conformance&source=gmail&ust=1787766514570000&sa=E
> 
> Julian Joseph
> ApexClaw