Re: Introduction: Evgenii Arsentev, and an offer to edit the reporting format deliverable

Nicolas,

Your question first, because it deserves a straight answer and the honest
one is partly against me.

The pass count was able to differ. The suite is the project's own test
suite, run at the end of every run
against the code the agent actually wrote. The tasks and their order were
identical in every condition and
nothing in the suite depends on the session policy, so the suite is not a
function of the variable I was
manipulating. A run that dropped a task or broke an interface would have
produced failures; "the cheaper
policy also did less work" is precisely the failure the gate was there to
catch, because a cost comparison
is worthless without it. The failing artifact is constructible. So:
saturated, not foreclosed.

Now the part I have to concede, because your mutation check has it and my
report does not. I did not prove
it. I inferred falsifiability from the construction of the harness rather
than demonstrating it by relaxing
something and watching a vector flip. You ran that experiment on your
corpus. I did not run it on mine. If
the format is going to carry this distinction, it should expose that gap
for my report too, and not only
for the cases that are obviously foreclosed. A reader of my paper has my
word that the gate could have
failed. That is worth less than your two-halves proof, and the format
should be able to say so.

Which is why I think the distinction belongs as a qualifier on the verdict,
not as a member of the cause
vocabulary.

State and cause, as you and Roel have them, answer "what may a consumer
conclude" and "why is there no
verdict". Every entry in the CAP-1 list and every change you propose to it
describes a unit that produced
no verdict, or produced one that cannot be trusted. Falsifiability is a
different question asked of a unit
that did produce one: this check passed, and how much is that pass worth.
Folding "could not have failed"
into cause would put a qualifier of pass into a vocabulary built for the
absence of pass, and the
aggregation rule would immediately get strange: a consumer counting causes
would be counting passes and
non-passes in the same column.

So a third field, next to state and cause, with a closed vocabulary of its
own. The three-way split I would
start from is close to yours:

  foreclosed no artifact that would fail this check can exist under the
constraints in force
  unproved the check could fail in principle, but nothing here demonstrates
it
  proved accompanied by the artifact that demonstrates it, your two-halves
run being exactly that

My own result is "unproved" under that vocabulary, and it should be. Yours,
for R3 and R8, is "foreclosed",
and the whole point of the field is that those two are not the same thing
and neither is a green tick.

One caution about the default, drawn from a different corner of this field.
A field that a submitter fills
in about the rigour of their own evidence converges on the flattering
value. If "proved" can be asserted,
it will be asserted, and within a year the field carries no information,
which is the same disease one
level up. So "proved" should be admissible only when the report carries the
artifact that proves it, and
the default must be "unproved" rather than "proved" or blank. CAP-1's
reasoning for refusing free text,
that free text does not aggregate, applies here with equal force to
unverifiable booleans.

The other thing I would add is one level above the check. In my data every
individual state was pass and
every individual state was, I believe, correct. The failure of information
was not in any single check but
in the suite's power to discriminate between the conditions being compared:
4,086 tests, zero failures, in
all six conditions and all thirty-six runs, while cost over those same
conditions moved from $2.680 per run
to $2.138, a difference of 25.3% with an exact permutation p of 0.0022. A
consumer reading thirty-six green
reports concludes the six configurations are equivalent. They differ by a
quarter of the bill. No per-check
field can express that, because the property belongs to the comparison and
not to the check. A result-set
level statement of whether any check changed state across the compared
conditions would have caught it, and
it costs one boolean plus the identity of the checks that moved.

On the work. I would like to take the run-reporting section of the fifth
deliverable and bring measurable
fields into it from draft-arsentev-agent-run-metrics-00: the run, the step
and the model invocation as
distinct units, input separated by provenance from generated output, and
the attribution rule for delegated
sub-runs so that two collectors cannot count the same work twice.
contextburn
(DOI 10.5281/zenodo.22712985, on PyPI) already emits that shape, so the
section would be written against a
working implementation rather than against an intention. If the group
prefers the section to start from
scratch, that is a fine answer and I will still write it; the draft is
unencumbered either way.

I will read 0000, 0008, 0020 and 0021 properly before proposing any text,
and I will send an outline to the
list rather than a finished section, so the shape can be argued before the
wording is.

The study is at DOI 10.5281/zenodo.22699668, with the run-level counters
and the harness released alongside
it, in case anyone wants to check the claim I just made about it.

Best regards,

Evgenii Arsentev, PhD
ARSENTEV.AI - https://arsentev.ai
ORCID 0000-0002-9120-7298


пн, 14 сент. 2026 г. в 01:56, Kenne Ives <kenne@agentavow.com>:

> Welcome, Evgenii. Your 4,086-tests-zero-failures result and the R3/R8 gap
> Nicolás described are two forms of the same problem, and the reporting
> format has to tell them apart.
>
> Yours is a saturated gate: the suite could have failed and didn't, so the
> pass carries no information about the conditions you varied. Someone can
> still write a failing test for it. R3/R8 is worse. There the failing
> artefact can't be constructed at all, because the schema rejects the
> condition the rule tests before the rule ever runs. A coverage count sees a
> vector and stops, so it never notices.
>
> The consequence for the format is the same either way: a pass is only
> evidence if you also report whether the check could have failed. Roel and I
> landed on representing this as two fields, state and cause, checked against
> each other so impossible pairs are rejected. That covers your saturation
> case (checked, could-fail, passed) and the foreclosed case (not-run,
> precondition-unsatisfiable) in one schema.
>
> Nicolás already put the key question to you, and your answer decides which
> case your study is. The format needs a cell for each.
>
> Kenne
>
> On Sun, Sep 13, 2026 at 6:18 AM Nicolas Rocchia <nicolasrocchia@gmail.com>
> wrote:
>
>> Evgenii,
>>
>> Welcome, and the conformance point is the right one to lead with.
>>
>> Some of it is already on this list, and the message numbers are the
>> fastest way in. On 8 September I ran a mutation check over the 89
>> conformance vectors of my own corpus: relax one rule, re-run every vector,
>> compare verdict and fired rule labels against what was recorded (0013). For
>> twelve of the fourteen rules, the vectors that flip are exactly the ones
>> that name them. Two, R3 and R8, have no MUST-FAIL vector at all, so a
>> verifier that never implements them passes the whole suite and the exact
>> comparison cannot see it, because there is nothing to compare. Kenne called
>> it the real gap (0014).
>>
>> Then it got worse, and that is the part I would put in front of you. The
>> two missing vectors cannot be written. All three schema versions already
>> forbid the exact condition each rule tests, and the validator returns shape
>> errors without reaching the rules. The requirement is stated twice, in the
>> schema and in a rule, and only the schema can ever be observed (0016).
>> Kenne's sharpening: "That is worse than untested: untested is at least a
>> visible gap, and this isn't visible at all" (0017).
>>
>> That is the distinction I would put next to your 4,086. A saturated suite
>> can fail in principle: someone can write the failing test. A foreclosed
>> check cannot fail in any artifact that is allowed to exist. Both look like
>> evidence and are not. Only the second is invisible to a coverage count,
>> because the count sees a vector and stops.
>>
>> So a question about your own data, and I ask it because the answer may
>> strengthen your result rather than weaken it. Across the six conditions,
>> was the pass count able to differ, or was it constant by construction? If
>> the suite could have failed and did not, you have a saturated gate. If
>> nothing in those conditions could have made it fail, you have the second
>> case, and that is a harder finding than the one you reported.
>>
>> What came out of this here is a third outcome for a coverage census, next
>> to "has a vector" and "missing": a proved exemption. Each run rebuilds the
>> artifact that would fire the rule and requires both halves, that the schema
>> rejects it, and that the rule still fires with the schema layer skipped.
>> Relax the schema and the first half fails. Delete the rule and the second
>> does. It runs in two implementations, Python and TypeScript, from one
>> shared declaration and with no shared proof code (0018).
>>
>> On the fifth deliverable. The survey of how existing formats represent a
>> check that did not run is on this list in two versions (0000, 0008): ten
>> rows, five questions, every row read in the primary source, and every
>> participant's row left blank for its author to fill. Every row whose author
>> is on this list has since been answered by its author, and Anton corrected
>> his own rather than confirming it. The last exchange, Roel and Kenne on 10
>> September (0020, 0021), converged on a concrete design decision for the
>> format: state and cause as two fields, checked against each other, with
>> impossible pairs rejected by the validator.
>>
>> That is already the first piece of the reporting format, and three people
>> who disagreed on the way there wrote it. So there is a third starting point
>> besides your draft and a blank page, and I would rather co-edit that
>> deliverable with you than pick between the other two.
>>
>> Independent of any title: I will write up the two-field decision with the
>> survey rows as its evidence base and post it here within a week, for you to
>> take apart. If the group wants a single editor, I will still write that
>> piece.
>>
>> Honest scope, unchanged since my introduction: one maintainer, no outside
>> adoption, and both implementations of the schema are mine. The independent
>> reproduction in the archive (Aug/0006) is one cold run by one person, not
>> adoption.
>>
>> Nicolás
>>
>>
>> Nicolás Rocchia
>> disensor.dev · San Francisco, Córdoba, Argentina
>>
>> El dom, 13 sept 2026, 09:51, Evgenii Arsentev <arsevgen22@gmail.com>
>> escribió:
>>
>>> Hello all,
>>>
>>> I joined the group today, so this is an introduction with something
>>> attached to it rather than a wave.
>>>
>>> I measure what language-model agents actually consume, and I publish the
>>> run-level data rather than the
>>> summary. Two pieces bear directly on what this group says it wants to do.
>>>
>>> The first is a controlled study of session policy in coding agents: 36
>>> runs across six context-clearing
>>> policies, six replicates each, holding the model, the tasks, their order
>>> and the verification suite
>>> constant. Cost is not monotone in session length - $2.68 per run when
>>> the context is cleared after every
>>> task, $2.14 at every third task, $2.47 when it is never cleared, with
>>> exact permutation tests giving
>>> p = 0.0022 and p = 0.0108 against the two endpoints. The run-level token
>>> counters and the harness are
>>> released with it. DOI 10.5281/zenodo.22699668
>>>
>>> The second is a mining study of the execution journals of one agent
>>> runtime: 722 runs, 150,902 model
>>> invocations, 34.6 billion tokens. More than half of all model
>>> invocations occurred in delegated sub-runs
>>> rather than in the runs that started them, which turns "how much did
>>> this run cost" into a definitional
>>> question before it is a counting one. DOI 10.5281/zenodo.22688706
>>>
>>> One finding is worth putting in front of this group specifically,
>>> because it is about conformance rather
>>> than about cost. In that first study the verification suite reported
>>> 4,086 tests and zero failures - in
>>> every one of the six conditions. Quality was identical everywhere, so
>>> the gate carried no information at
>>> all about the differences between conditions, while cost over those same
>>> conditions differed by up to 25%.
>>> A suite that never fails looks like evidence and is not. Whatever rubric
>>> this group lands on, the pass rate
>>> of a check needs to be reported alongside the pass, or saturated checks
>>> will quietly accumulate inside
>>> every published result.
>>>
>>> On the deliverables: of the five listed, the fifth - "a reporting format
>>> so results are comparable across
>>> implementations and vendors" - is the one I have already had to build,
>>> and I would like to serve as its
>>> editor.
>>>
>>> What I bring is a running start rather than an intention. I am the
>>> author of two IETF Internet-Drafts
>>> published this month:
>>>
>>>   draft-arsentev-agent-run-metrics-00 - Agent Run Metrics, a JSON
>>> interchange format for the resource
>>>   accounting of language-model agent runs. It defines the run, the step
>>> and the model invocation as
>>>   distinct units, separates input by provenance (fresh, cache-read,
>>> cache-write) from generated output,
>>>   and specifies how the consumption of a delegated sub-run is attributed
>>> so that two collectors cannot
>>>   count the same work twice. That delegation rule exists because the
>>> 722-run dataset broke every simpler
>>>   approach I tried.
>>>   https://datatracker.ietf.org/doc/draft-arsentev-agent-run-metrics/
>>>
>>>   draft-arsentev-llm-context-discovery-00 - discovery and retrieval of
>>> publisher-curated context files,
>>>   built on a fifteen-day log study: 331,758 logged requests to one
>>> origin, of which 44,005 came from
>>>   model-provider crawlers, and zero of those retrieved the curated
>>> context file that was sitting there
>>>   waiting for them.
>>>   https://datatracker.ietf.org/doc/draft-arsentev-llm-context-discovery/
>>>
>>> There is also a working implementation rather than a slide: contextburn,
>>> an open-source run-efficiency
>>> meter for coding agents, on PyPI. DOI 10.5281/zenodo.22712985 . The
>>> group's reproducibility line - that a
>>> published result must name the engine version, the corpus version and
>>> the date - is exactly the constraint
>>> that tool had to satisfy to be useful, and I would write the reporting
>>> format around that requirement
>>> rather than bolt it on afterwards.
>>>
>>> If the group would rather the reporting format started from scratch than
>>> from an existing draft, that is a
>>> fine answer and I will still do the work. The drafts are unencumbered
>>> either way.
>>>
>>> How I work: asynchronously, and in writing - drafts, pull requests,
>>> review of other people's text,
>>> comments on this list. I do not attend calls, so please do not hold a
>>> slot for me. If the group's rhythm is
>>> a weekly meeting, I will read the minutes and have text ready between
>>> them, on a deadline, which is
>>> usually the part that matters.
>>>
>>> Chair and all: if you want me as editor of the reporting format
>>> deliverable, say so on this list and I
>>> will bring a first outline within a week.
>>>
>>> Best regards,
>>>
>>> Evgenii Arsentev, PhD
>>> ARSENTEV.AI - https://arsentev.ai
>>> ORCID 0000-0002-9120-7298
>>> GitHub https://github.com/arsentev-ai
>>>
>>>

-- 
С уважением,
Арсентьев Евгений

Received on Monday, 14 September 2026 08:13:27 UTC