Research note

When Better IT Agents Cost More Per Run: An ITBench Cost Recalculation

Four paired ITBench results reveal a cost distinction that matters for autonomous incident response: an agent can become more expensive to run, yet cheaper relative to its diagnostic score.

The Null Institute | Independent Research Note | Updated 18 November 2026
By Ilya Nikitin, Ph.D. | Secondary calculation from published IBM results; not peer-reviewed; no new agent executions.

Our finding: In four matched-model comparisons reported by IBM Research, the graph-guided EoG agent uses more input tokens than ReAct for three models. At an illustrative 4:1 output-to-input token price ratio, those three EoG runs would cost 2.26×, 7.18× and 2.51× as much in reported tokens. Yet across all four models, EoG consumes less measured token expenditure per point of published majority@3 root-cause-entity F1, yielding cost-per-F1 ratios of 0.23× to 0.86× relative to ReAct. We further calculate the exact output-token price premiums required for EoG to break even per run: approximately 139×, 895× and 235× for the three models whose input usage rose. These are our arithmetic results, not new agent evaluations or observed dollar costs.

What we used, and what IBM measured

IBM’s Jha et al., Think Locally, Explain Globally, arXiv:2601.17915v2 (29 January 2026), Table 2, PDF page 8, compares ReAct with Explanations over Graphs (EoG). Its test covers 35 ITBench Kubernetes incident scenarios, with three runs per configuration. EoG uses graph-guided reasoning and a deterministic controller. The paper reports two related root-cause-entity F1 measures, including majority@3, the score for answers supported by repeated runs. The authors conducted the experiments. We independently transcribed four matched model pairs from their published table and calculated the additional economic comparisons below. We did not reproduce the original model runs.

Model ReAct input / output
(thousand tokens)
EoG input / output
(thousand tokens)
Majority@3 entity F1
ReAct → EoG
GPT-5.1 779 / 9.3 1,837 / 1.7 8.6 → 86.1
GPT-OSS-120B 230 / 3.8 1,751 / 2.1 5.7 → 79.4
Gemini 3 Flash 5,376 / 21.2 3,390 / 2.0 74.3 → 92.1
Kimi K2 1,021 / 8.8 2,641 / 1.9 28.6 → 83.0

All input, output and F1 values above are from IBM’s Table 2, not measurements by The Null Institute. The paper reports mean token counts per trial per scenario and explicitly excludes reasoning tokens, which are not consistently reported by providers.

Independent calculation 1: what price ratio would make EoG cheaper per run?

Let I and O be the reported input and output token counts, and r be the output-to-input token price ratio. With the same prices for ReAct and EoG within each model, the measured-token cost proxy is:

C(r) = I + r × O, in input-token price units.

For a model whose EoG version uses more input but less output, equal per-run cost occurs at:

r* = (IEoG − IReAct) / (OReAct − OEoG).

For GPT-5.1, for instance, EoG adds 1,058,000 input tokens while saving 7,600 output tokens. Thus r* = 1,058 / 7.6 = 139.21. Only at an output-token premium above that threshold would the recorded-token cost become lower for EoG. Gemini 3 Flash needs no threshold: its EoG version uses fewer tokens in both categories.

Model Extra EoG input tokens Per-run break-even
output:input price
EoG/ReAct cost
at illustrative r=4
GPT-5.1 +136% 139.21:1 2.26×
GPT-OSS-120B +661% 894.71:1 7.18×
Gemini 3 Flash −37% EoG lower for any positive ratio 0.62×
Kimi K2 +159% 234.78:1 2.51×

The 4:1 price ratio is a hypothetical sensitivity setting, not a claim about current model-provider rates. We do not compare actual money spent between different model providers or report full inference costs.

Independent calculation 2: change the denominator

One run is not necessarily the right unit for an incident-response programme. The architecture also changes the reported diagnostic score. To expose this trade-off without pretending to know exact incident-success rates, we divided each architecture’s measured-token cost proxy by its own majority@3 entity F1 score in percentage points:

Relative cost per F1 point = (CEoG/F1EoG) / (CReAct/F1ReAct).

Model EoG/ReAct reported-token cost per F1 point, at r=4
GPT-5.1 0.23×
GPT-OSS-120B 0.52×
Gemini 3 Flash 0.50×
Kimi K2 0.86×

This is not merely a result of choosing 4:1. In all four matched comparisons, EoG uses fewer recorded input tokens per majority@3 F1 point and fewer recorded output tokens per majority@3 F1 point. Therefore, for any positive input and output token prices that apply equally to the two architectures within a model, their positive weighted sum is also lower per F1 point for EoG. That is a price-ratio-independent property of these reported aggregates.

Crucial caveat: an F1 point is not an incident solved. F1 can reward partially correct root-cause identification. Dividing by F1 is a descriptive efficiency index, not an estimate of dollars per successfully resolved outage or a linear model of operational value. The two costs answer different questions: How expensive is this attempt? and How much benchmark score does the measured token expenditure buy? The data do not answer a third question, How much would an enterprise save on real incidents?

What a CIO can use

A purchase or deployment decision should not be made from an agent’s headline accuracy or its token count alone. A useful pilot should compare cost per attempted investigation, consistency across repeats, exact correctness of the root cause, time to diagnosis, human review and the risks of automated remediation. The IBM result makes graph-guided investigation worth testing, but it does not establish the cost of production deployments. Keep the first rollout read-only and require authorised checks before production-changing actions. The right team combines SRE or operations expertise with measurement, tool orchestration and access-control skills.

Reproduce our result

All input numbers are visible in the first table. For each model, insert the source values into r_star = (I_eog-I_react)/(O_react-O_eog) and cost_ratio = (I_eog+4*O_eog)/(I_react+4*O_react). Then divide that ratio by F1_eog/F1_react to obtain relative cost per F1 point. Here is a complete standard-library Python check:

rows = [
 ("GPT-5.1",779,9.3,8.6,1837,1.7,86.1),
 ("GPT-OSS-120B",230,3.8,5.7,1751,2.1,79.4),
 ("Gemini 3 Flash",5376,21.2,74.3,3390,2.0,92.1),
 ("Kimi K2",1021,8.8,28.6,2641,1.9,83.0),
]
for name,ir,orr,fr,ie,oe,fe in rows:
    threshold = (ie-ir)/(orr-oe) if ie>ir else None
    ratio = (ie+4*oe)/(ir+4*orr)
    efficiency = ratio*fr/fe
    assert ie/fe < ir/fr and oe/fe < orr/fr
    print(name, threshold, round(ratio,3),
          round(efficiency,3))

Checks and limitations: The source dataset contains 35 diagnostic scenarios with only three runs per configuration; the paper’s Table 2 provides aggregate results rather than the run-level values needed for our own confidence intervals. The comparisons combine multiple architectural changes. Most importantly, the published input/output token figures exclude reasoning tokens, as well as any infrastructure overhead, caching effects, tool costs, latency, human review or production remediation. Actual prices may differ by provider, model and contract. We make no claims about statistical significance, causal effect of the controller alone, or exact production incident success.

Source: Jha et al. (2026), arXiv:2601.17915v2, Table 2 on PDF p. 8. The original observations, model comparisons and root-cause F1 scores are IBM’s; the price thresholds, conditional cost ratios and F1-normalized comparisons are The Null Institute’s independent secondary calculations.

Revision note (18 November 2026): Earlier versions of this article focused first on hypothetical retry probabilities and then on the pass-to-majority F1 gap. This revision makes the incremental cost audit explicit, retains the original URL, and distinguishes per-run cost from a metric-normalized cost proxy. Previous hypothetical retry numbers are not presented as empirical results.