- From: Evgenii Arsentev <arsevgen22@gmail.com>
- Date: Mon, 14 Sep 2026 11:13:09 +0300
- To: Kenne Ives <kenne@agentavow.com>
- Cc: Nicolas Rocchia <nicolasrocchia@gmail.com>, public-agent-conformance@w3.org
- Message-ID: <CAFF7QGk6QmnoWTZkKND1qvE2knhpGTxb=KbKuLmT=V9bYBEiVw@mail.gmail.com>
Nicolas, Your question first, because it deserves a straight answer and the honest one is partly against me. The pass count was able to differ. The suite is the project's own test suite, run at the end of every run against the code the agent actually wrote. The tasks and their order were identical in every condition and nothing in the suite depends on the session policy, so the suite is not a function of the variable I was manipulating. A run that dropped a task or broke an interface would have produced failures; "the cheaper policy also did less work" is precisely the failure the gate was there to catch, because a cost comparison is worthless without it. The failing artifact is constructible. So: saturated, not foreclosed. Now the part I have to concede, because your mutation check has it and my report does not. I did not prove it. I inferred falsifiability from the construction of the harness rather than demonstrating it by relaxing something and watching a vector flip. You ran that experiment on your corpus. I did not run it on mine. If the format is going to carry this distinction, it should expose that gap for my report too, and not only for the cases that are obviously foreclosed. A reader of my paper has my word that the gate could have failed. That is worth less than your two-halves proof, and the format should be able to say so. Which is why I think the distinction belongs as a qualifier on the verdict, not as a member of the cause vocabulary. State and cause, as you and Roel have them, answer "what may a consumer conclude" and "why is there no verdict". Every entry in the CAP-1 list and every change you propose to it describes a unit that produced no verdict, or produced one that cannot be trusted. Falsifiability is a different question asked of a unit that did produce one: this check passed, and how much is that pass worth. Folding "could not have failed" into cause would put a qualifier of pass into a vocabulary built for the absence of pass, and the aggregation rule would immediately get strange: a consumer counting causes would be counting passes and non-passes in the same column. So a third field, next to state and cause, with a closed vocabulary of its own. The three-way split I would start from is close to yours: foreclosed no artifact that would fail this check can exist under the constraints in force unproved the check could fail in principle, but nothing here demonstrates it proved accompanied by the artifact that demonstrates it, your two-halves run being exactly that My own result is "unproved" under that vocabulary, and it should be. Yours, for R3 and R8, is "foreclosed", and the whole point of the field is that those two are not the same thing and neither is a green tick. One caution about the default, drawn from a different corner of this field. A field that a submitter fills in about the rigour of their own evidence converges on the flattering value. If "proved" can be asserted, it will be asserted, and within a year the field carries no information, which is the same disease one level up. So "proved" should be admissible only when the report carries the artifact that proves it, and the default must be "unproved" rather than "proved" or blank. CAP-1's reasoning for refusing free text, that free text does not aggregate, applies here with equal force to unverifiable booleans. The other thing I would add is one level above the check. In my data every individual state was pass and every individual state was, I believe, correct. The failure of information was not in any single check but in the suite's power to discriminate between the conditions being compared: 4,086 tests, zero failures, in all six conditions and all thirty-six runs, while cost over those same conditions moved from $2.680 per run to $2.138, a difference of 25.3% with an exact permutation p of 0.0022. A consumer reading thirty-six green reports concludes the six configurations are equivalent. They differ by a quarter of the bill. No per-check field can express that, because the property belongs to the comparison and not to the check. A result-set level statement of whether any check changed state across the compared conditions would have caught it, and it costs one boolean plus the identity of the checks that moved. On the work. I would like to take the run-reporting section of the fifth deliverable and bring measurable fields into it from draft-arsentev-agent-run-metrics-00: the run, the step and the model invocation as distinct units, input separated by provenance from generated output, and the attribution rule for delegated sub-runs so that two collectors cannot count the same work twice. contextburn (DOI 10.5281/zenodo.22712985, on PyPI) already emits that shape, so the section would be written against a working implementation rather than against an intention. If the group prefers the section to start from scratch, that is a fine answer and I will still write it; the draft is unencumbered either way. I will read 0000, 0008, 0020 and 0021 properly before proposing any text, and I will send an outline to the list rather than a finished section, so the shape can be argued before the wording is. The study is at DOI 10.5281/zenodo.22699668, with the run-level counters and the harness released alongside it, in case anyone wants to check the claim I just made about it. Best regards, Evgenii Arsentev, PhD ARSENTEV.AI - https://arsentev.ai ORCID 0000-0002-9120-7298 пн, 14 сент. 2026 г. в 01:56, Kenne Ives <kenne@agentavow.com>: > Welcome, Evgenii. Your 4,086-tests-zero-failures result and the R3/R8 gap > Nicolás described are two forms of the same problem, and the reporting > format has to tell them apart. > > Yours is a saturated gate: the suite could have failed and didn't, so the > pass carries no information about the conditions you varied. Someone can > still write a failing test for it. R3/R8 is worse. There the failing > artefact can't be constructed at all, because the schema rejects the > condition the rule tests before the rule ever runs. A coverage count sees a > vector and stops, so it never notices. > > The consequence for the format is the same either way: a pass is only > evidence if you also report whether the check could have failed. Roel and I > landed on representing this as two fields, state and cause, checked against > each other so impossible pairs are rejected. That covers your saturation > case (checked, could-fail, passed) and the foreclosed case (not-run, > precondition-unsatisfiable) in one schema. > > Nicolás already put the key question to you, and your answer decides which > case your study is. The format needs a cell for each. > > Kenne > > On Sun, Sep 13, 2026 at 6:18 AM Nicolas Rocchia <nicolasrocchia@gmail.com> > wrote: > >> Evgenii, >> >> Welcome, and the conformance point is the right one to lead with. >> >> Some of it is already on this list, and the message numbers are the >> fastest way in. On 8 September I ran a mutation check over the 89 >> conformance vectors of my own corpus: relax one rule, re-run every vector, >> compare verdict and fired rule labels against what was recorded (0013). For >> twelve of the fourteen rules, the vectors that flip are exactly the ones >> that name them. Two, R3 and R8, have no MUST-FAIL vector at all, so a >> verifier that never implements them passes the whole suite and the exact >> comparison cannot see it, because there is nothing to compare. Kenne called >> it the real gap (0014). >> >> Then it got worse, and that is the part I would put in front of you. The >> two missing vectors cannot be written. All three schema versions already >> forbid the exact condition each rule tests, and the validator returns shape >> errors without reaching the rules. The requirement is stated twice, in the >> schema and in a rule, and only the schema can ever be observed (0016). >> Kenne's sharpening: "That is worse than untested: untested is at least a >> visible gap, and this isn't visible at all" (0017). >> >> That is the distinction I would put next to your 4,086. A saturated suite >> can fail in principle: someone can write the failing test. A foreclosed >> check cannot fail in any artifact that is allowed to exist. Both look like >> evidence and are not. Only the second is invisible to a coverage count, >> because the count sees a vector and stops. >> >> So a question about your own data, and I ask it because the answer may >> strengthen your result rather than weaken it. Across the six conditions, >> was the pass count able to differ, or was it constant by construction? If >> the suite could have failed and did not, you have a saturated gate. If >> nothing in those conditions could have made it fail, you have the second >> case, and that is a harder finding than the one you reported. >> >> What came out of this here is a third outcome for a coverage census, next >> to "has a vector" and "missing": a proved exemption. Each run rebuilds the >> artifact that would fire the rule and requires both halves, that the schema >> rejects it, and that the rule still fires with the schema layer skipped. >> Relax the schema and the first half fails. Delete the rule and the second >> does. It runs in two implementations, Python and TypeScript, from one >> shared declaration and with no shared proof code (0018). >> >> On the fifth deliverable. The survey of how existing formats represent a >> check that did not run is on this list in two versions (0000, 0008): ten >> rows, five questions, every row read in the primary source, and every >> participant's row left blank for its author to fill. Every row whose author >> is on this list has since been answered by its author, and Anton corrected >> his own rather than confirming it. The last exchange, Roel and Kenne on 10 >> September (0020, 0021), converged on a concrete design decision for the >> format: state and cause as two fields, checked against each other, with >> impossible pairs rejected by the validator. >> >> That is already the first piece of the reporting format, and three people >> who disagreed on the way there wrote it. So there is a third starting point >> besides your draft and a blank page, and I would rather co-edit that >> deliverable with you than pick between the other two. >> >> Independent of any title: I will write up the two-field decision with the >> survey rows as its evidence base and post it here within a week, for you to >> take apart. If the group wants a single editor, I will still write that >> piece. >> >> Honest scope, unchanged since my introduction: one maintainer, no outside >> adoption, and both implementations of the schema are mine. The independent >> reproduction in the archive (Aug/0006) is one cold run by one person, not >> adoption. >> >> Nicolás >> >> >> Nicolás Rocchia >> disensor.dev · San Francisco, Córdoba, Argentina >> >> El dom, 13 sept 2026, 09:51, Evgenii Arsentev <arsevgen22@gmail.com> >> escribió: >> >>> Hello all, >>> >>> I joined the group today, so this is an introduction with something >>> attached to it rather than a wave. >>> >>> I measure what language-model agents actually consume, and I publish the >>> run-level data rather than the >>> summary. Two pieces bear directly on what this group says it wants to do. >>> >>> The first is a controlled study of session policy in coding agents: 36 >>> runs across six context-clearing >>> policies, six replicates each, holding the model, the tasks, their order >>> and the verification suite >>> constant. Cost is not monotone in session length - $2.68 per run when >>> the context is cleared after every >>> task, $2.14 at every third task, $2.47 when it is never cleared, with >>> exact permutation tests giving >>> p = 0.0022 and p = 0.0108 against the two endpoints. The run-level token >>> counters and the harness are >>> released with it. DOI 10.5281/zenodo.22699668 >>> >>> The second is a mining study of the execution journals of one agent >>> runtime: 722 runs, 150,902 model >>> invocations, 34.6 billion tokens. More than half of all model >>> invocations occurred in delegated sub-runs >>> rather than in the runs that started them, which turns "how much did >>> this run cost" into a definitional >>> question before it is a counting one. DOI 10.5281/zenodo.22688706 >>> >>> One finding is worth putting in front of this group specifically, >>> because it is about conformance rather >>> than about cost. In that first study the verification suite reported >>> 4,086 tests and zero failures - in >>> every one of the six conditions. Quality was identical everywhere, so >>> the gate carried no information at >>> all about the differences between conditions, while cost over those same >>> conditions differed by up to 25%. >>> A suite that never fails looks like evidence and is not. Whatever rubric >>> this group lands on, the pass rate >>> of a check needs to be reported alongside the pass, or saturated checks >>> will quietly accumulate inside >>> every published result. >>> >>> On the deliverables: of the five listed, the fifth - "a reporting format >>> so results are comparable across >>> implementations and vendors" - is the one I have already had to build, >>> and I would like to serve as its >>> editor. >>> >>> What I bring is a running start rather than an intention. I am the >>> author of two IETF Internet-Drafts >>> published this month: >>> >>> draft-arsentev-agent-run-metrics-00 - Agent Run Metrics, a JSON >>> interchange format for the resource >>> accounting of language-model agent runs. It defines the run, the step >>> and the model invocation as >>> distinct units, separates input by provenance (fresh, cache-read, >>> cache-write) from generated output, >>> and specifies how the consumption of a delegated sub-run is attributed >>> so that two collectors cannot >>> count the same work twice. That delegation rule exists because the >>> 722-run dataset broke every simpler >>> approach I tried. >>> https://datatracker.ietf.org/doc/draft-arsentev-agent-run-metrics/ >>> >>> draft-arsentev-llm-context-discovery-00 - discovery and retrieval of >>> publisher-curated context files, >>> built on a fifteen-day log study: 331,758 logged requests to one >>> origin, of which 44,005 came from >>> model-provider crawlers, and zero of those retrieved the curated >>> context file that was sitting there >>> waiting for them. >>> https://datatracker.ietf.org/doc/draft-arsentev-llm-context-discovery/ >>> >>> There is also a working implementation rather than a slide: contextburn, >>> an open-source run-efficiency >>> meter for coding agents, on PyPI. DOI 10.5281/zenodo.22712985 . The >>> group's reproducibility line - that a >>> published result must name the engine version, the corpus version and >>> the date - is exactly the constraint >>> that tool had to satisfy to be useful, and I would write the reporting >>> format around that requirement >>> rather than bolt it on afterwards. >>> >>> If the group would rather the reporting format started from scratch than >>> from an existing draft, that is a >>> fine answer and I will still do the work. The drafts are unencumbered >>> either way. >>> >>> How I work: asynchronously, and in writing - drafts, pull requests, >>> review of other people's text, >>> comments on this list. I do not attend calls, so please do not hold a >>> slot for me. If the group's rhythm is >>> a weekly meeting, I will read the minutes and have text ready between >>> them, on a deadline, which is >>> usually the part that matters. >>> >>> Chair and all: if you want me as editor of the reporting format >>> deliverable, say so on this list and I >>> will bring a first outline within a week. >>> >>> Best regards, >>> >>> Evgenii Arsentev, PhD >>> ARSENTEV.AI - https://arsentev.ai >>> ORCID 0000-0002-9120-7298 >>> GitHub https://github.com/arsentev-ai >>> >>> -- С уважением, Арсентьев Евгений
Received on Monday, 14 September 2026 08:13:27 UTC