Qualifying Claude Opus 5.5 on the incident queue with pre-deployment gates and trace replay

The claim

In report 01 and report 02, we showed across eight models that an LLM's self-reported confidence cannot gate an incident queue. Because raw model confidence is uncalibrated, our production agent enforces deterministic verification rules in code after the model answers. Those rules assign the status chip (grounded, supported, or inconclusive) shown in report 02's Figure 5. This report shows how our pre-deployment evaluation and trace-replay harness qualified Opus 5.5, rejected a schema-level retry candidate, and verified a three-tier evidence gate.

We evaluated upgrading our Claude shortlist leg from Opus 4.6 to Opus 5.5 on 47 Support-closed incidents (2 repeats each, 188 judged cells). Usable root-cause accuracy rose from 83% to 95% (+11.7 points, 90% interval +5.3 to +19.2). List-price cost per cell fell to 0.66 times Opus 4.6 ($1.58 vs $2.40). Our co-primary safety endpoint ("confidently wrong", a final grounded chip on a MISS) dropped from 6% to 0% (-6.4 points).

Three engineering findings underneath that headline shaped our production release. First, trace auditing separated raw model claims from our deterministic post-processing gate. The existing analyzer-link gate downgraded grounded on 71 of 94 Opus 5.5 answers (81 raw claims down to 10 final chips). Opus 5.5 diagnosed those incidents directly from DuckDB log queries rather than pre-canned signatures. Sixty-nine of those 71 downgraded answers (97%) were right.

Second, we ran a pre-deployment ablation on a candidate prompt-and-retry modification: a one-shot validator retry plus stricter JSON schema field descriptions. Tested across 94 new cells, our acceptance gate rejected the candidate before it could reach production. The grounded rate rose only +7.4 points (against +25 required), while usable accuracy fell by 6.4 points (95% to 88%, interval -11.7 to -2.1). Tightening the schema description induced upfront hedging in 45 of 94 runs, and full-JSON retries interacted with downstream verification rules.

Third, we replayed 818 archived runs at $0 LLM cost to verify gate logic in deterministic code alone. That replay refined word-boundary entity matching ("stream" vs "downstream"), corrected one benchmark answer key, and verified our schema-invisible supported tier (SkipJsonSchema).

Lessons for teams running agents

1. Treat schema text and retries as treatments. Changing two Pydantic field descriptions cost 6.4 points of diagnostic accuracy (Sculley et al., 2015). We caught the regression only because the fix went through the same blinded benchmark as the model swap.
2. Audit raw model outputs alongside post-processed labels. Aggregate metrics scored our post-processed chip, not the model. Keeping raw traces separated what the model claimed (81 of 94 grounded) from what our code allowed (10 of 94). That check also corrected a finding in report 01.
3. Replay stored traces before spending tokens. Rebuilding the evidence ledger from traces reproduced 818 of 818 historical labels (100%) at $0. We falsified a looser gate across four models in minutes.
4. Enforce automated spend caps and audit logs on evaluation runs. Our pre-flight cost gate halted the initial smoke run at 4 cells when projected spend exceeded budget. Every protocol amendment was recorded with UTC run-log and git timestamps.

I lead Applied AI and Support at Yugabyte. The agent in this report is an internal tool that reads diagnostic bundles for our support engineers. All models run through our standard cloud accounts with no vendor sponsorship.

How the model swap was measured

Cases. The test used the 47 Support-closed incidents from report 01 whose root cause does not appear in linked escalation records. Those are the same 47 Support-closed tickets evaluated alongside 42 escalated tickets in report 02.

Arms. The two arms differed only in the model (Opus 4.6 vs Opus 5.5). Both ran the same agent build, prompt, tools, and prompt caching (report 04). Each incident ran twice per arm, giving 94 cells per arm and 188 cells in total.

Judging. A blinded panel of two judge models (Claude Opus 5.5 and Gemini 3.8 Flash), three passes each, scored every shuffled answer against the human root cause. Scores were MATCH, PARTIAL, or MISS. Usable (MATCH or PARTIAL) was pre-registered as the primary endpoint before any judging (evals/forensic_metrics.py). Two readings were pre-registered: the full two-judge panel and Gemini 3.8 Flash alone, which removes any self-preference from Opus 5.5 judging its own arm.

Pre-registered decision rule. The primary usable endpoint had a non-inferiority margin of -10 points (90% lower bound above -10 points). The co-primary confidently-wrong endpoint required a 90% upper bound below +5 points. Intervals come from a paired case bootstrap over the 47 incidents (10,000 resamples, Efron & Tibshirani, 1993). The two repeats are averaged inside each case.

Automated spend gates and audit timeline

The pre-registered plan started with 1 repeat (94 cells) and a hard cost gate. After a 4-cell smoke check, projected spend had to stay at or below $170. After 4 cells ($8.00 spent), the projection hit $216.96 ($194.91 naive). The gate fired and stopped the run before any answer was judged.

Before resuming, we wrote Amendment A1, moving to 2 repeats per incident (188 cells) under a $450 cap ($440 hard stop) to measure run-to-run stability. We also recorded Deviation D1 at checkpoint 1 (of 3 checkpoints), removing an analyst-added clause about shrinking run length. Total spend across all 188 cells was $389.12 ($374.21 for agent runs and $14.91 across 1,128 judge calls, or $0.08 per judged cell). Running 47 new cases under the same design would cost $389.12 ($8.28 per case).

To maintain a complete audit trail, the table below records both the run-log timestamps and the git commit timestamps.

The initial registration entered git 22 minutes after the 4-cell smoke check launched, and eight hours before judging began. Amendments A1, D1, and the schema-ablation registration were timestamped in the protocol files during execution. They were committed alongside the run artifacts, placing this round at claim level L2. Auditing these September commit timestamps led to our formal change-control rule requiring pre-registration commits to land in Git before benchmark execution starts.

Act 1: the model swap and the hidden gate

Opus 5.5 scored higher on usable root causes in 8 incidents and lower in 0. On the first repeat alone, the usable gain was +10.6 points (90% interval +4.3 to +19.2). On exact match, the 95% interval ran from -2.1 to +26.6 points (14 incidents up, 7 down). Under a symmetric tie-break, exact match was 29% vs 33% (+4.3 points, 95% interval -7.5 to +14.9).

Why did Opus 5.5 advance from 60% (59.6%) exact match in late September to 87% (87.2% exact, 96% usable) in report 02? Three harness upgrades occurred between the two rounds. First, we expanded our MCP diagnostic tools to include ticket comments and lower query latency. Second, trace inspection identified an upstream pydantic-ai default of 4,096 max_tokens on claude-opus-5-5 (19 of 19 draws hit truncation), which we raised to 16,000 tokens. Third, cross-model auditing refined one reference answer key in the 47-ticket benchmark (detailed in Act 3 below).

Judge overlap, stability, and cost

To check whether Opus 5.5 favored its own arm, we measured the Opus 5.5 judge gap minus the Gemini judge gap. On the usable scale that difference was 0.0 points (90% interval -2.1 to +2.1). On the 0-to-2 score scale it was -0.096 (interval -0.19 to 0.00), meaning the Opus 5.5 judge graded its own family slightly harder.

Repeat agreement was much higher than our power model assumed. We had budgeted for a 36% case flip rate based on earlier Gemini runs. In practice, repeats 1 and 2 agreed on 91% of incidents for Opus 4.6 (4 flips) and 98% for Opus 5.5 (1 flip).

At Vertex list price on GENERATION spans, Opus 5.5 cost $1.58 per cell ($1.55 median, $148.97 total) against $2.40 for Opus 4.6 ($2.33 median, $225.24 total). That is a cost ratio of 0.66. Opus 5.5 was cheaper on 47 of 47 incidents, using 15.7 turns against 20.2 (11.6 vs 0.0 thinking parts) and 20.2 tool calls against 35.2. Wall time dropped from 7.1 minutes (426.6 s mean, 330.1 s median) to 4.3 minutes (259.9 s mean, 190.1 s median), a ratio of 0.61.

Separating model claims from the deterministic link gate on 71 of 94 runs

The 0% confidently-wrong rate on Opus 5.5 reflected both model accuracy and our deterministic post-processing gate. Auditing the raw model outputs in our self-hosted internal Langfuse trace store across all 188 cells separated the two effects.

Since 2026-06-16, our post-processing function (calibrate_tldr) has enforced a deterministic rule. It keeps a grounded chip only when symptom_to_finding_links contains a root_cause link to a non-config analyzer finding. Without that link, the rule rewrites the chip to inconclusive.

In the raw outputs, Opus 4.6 claimed grounded in 91 of 94 cells (97%). The link gate kept 46 (49%) and stripped 40 (4 had zero links; 43 had no root_cause link). Opus 5.5 claimed grounded in 81 of 94 cells (86%). The link gate kept only 10 (11%) and stripped 71 (61 had zero links; 84 had no root_cause link). Thus 89% of Opus 5.5 notes were marked inconclusive, against 51% for Opus 4.6.

Why did Opus 5.5 omit analyzer links? Pre-computed analyzer rules cover only known signatures. Opus 5.5 found root causes through ad hoc DuckDB queries and log greps (67 of 94 cells had bundle evidence; 54 had no analyzer finding). Among the 71 stripped Opus 5.5 answers, 69 were usable (97.2%, 97% rounded). On Opus 4.6, only 77.8% (78% rounded) of non-grounded raw claims were usable. On the older model the gate filtered out wrong answers; on the stronger model it suppressed right answers. Even on raw claims before the gate, Opus 5.5's raw confidently-wrong rate was 2% vs 17% for Opus 4.6 (-14.9 points, 90% interval -23.4 to -7.4).

Trace-replay attribution for Report 01

Report 01 ran agent library version 0.7.19 (dated 2026-08-14). Replaying that version's deterministic gate on Report 01's Wave 2 traces reproduced 102 of 102 stored labels (100%). Report 01 noted that the Claude arm "under-claimed: it said grounded on 43% of incidents and was exactly right on 48%". That 43% measured the combined Opus 4.6 + link gate system (91 of 94 raw claims filtered to 46), confirming that our code enforced the bound.

Act 2: how pre-deployment testing blocked a 6.4-point schema-and-retry regression

Before deploying Opus 5.5, we evaluated a candidate modification (F55) designed to increase grounded link citations. First, when a grounded answer lacked a root_cause link, a one-shot pydantic-ai ModelRetry validator prompted the model once to attach the link or select inconclusive. Second, we tightened the JSON schema descriptions on symptom_to_finding_links and grounding_status to state that grounded requires an analyzer root_cause link.

We pre-registered F55 against the A55 control (94 cells per arm, 188 judged cells, $165.15 total spend including $7.22 judge cost, $1.68 per F55 cell). To pass, F55 had to raise grounded by at least +25 points (90% lower bound above +15 points) while maintaining usable non-inferiority (>-10 points).

Our pre-registered acceptance gate rejected the candidate on both endpoints.

Usable accuracy fell by 6.4 points (95% to 88%, exact match 61% to 56%), and MISS cells rose from 5 to 11 (5 cases down, 0 up). Repeat agreement also fell from 98% to 89%. Inspecting the 94 F55 traces isolated two mechanisms behind the regression.

1. Schema descriptions acted as a hedging prompt (Tam et al., 2024). All 11 F55 MISS cells had 0 validator retries. Every cell where a validator retry fired (31 of 31, 100%) was judged usable. Instead, reading the stricter schema description upfront caused Opus 5.5's raw grounded claims to fall from 81 to 37. In 45 of 94 F55 cells (against 4 of 94 in A55), the blinded answer echoed the new schema wording ("no analyzer finding directly explains... thus inconclusive"). Those 45 hedged answers scored 84.4% usable (38 of 45) and held 7 of the 11 MISSes.
2. Full-JSON retries hit a second gate. The link retry fired in 19 of 94 cells (+$0.18 and +1.42 turns per retried cell). Those retries rewrote the blinded prose in 16 of 19 cells. Yet it saved the grounded badge in only 1 of 19 cells (6 still lacked a link, and 12 hit a downstream gate). Because calibrate_tldr ran a cascade of gates, satisfying the step-2 link gate exposed 20 cells to downstream gates. Eleven were downgraded by the step-3 differential gate and 1 by the step-4 entity gate.

Because our pre-deployment benchmark caught the 6.4-point accuracy regression before release, we rejected F55 and promoted baseline Opus 5.5 to production on 2026-09-27 (15:57Z). Post-deployment telemetry (Clause 9.1) immediately flagged the 4,096 max_tokens truncation and 89% inconclusive rate, triggering the 16,000 max_tokens and supported-tier corrective release (report 02).

Act 3: $0 trace replay, "downstream", and the supported tier

Next, we asked whether we could fix the gate in deterministic Python code alone (gate_v2), without touching the prompt or schema. Under gate_v2, an answer kept grounded if its winning hypothesis cited either (a) an analyzer finding or (b) a verified bundle-inspection tool call.

Rather than running new live calls, we rebuilt the evidence ledger from archived traces. We replayed gate_v2 across 818 historical cells (282 Claude and 536 Gemini) at $0 LLM cost. Replaying each run's original gate version reproduced 282 of 282 Claude labels and 536 of 536 Gemini labels (100%). It also matched 24 of 24 live confirmation cells.

That $0 replay surfaced three engineering insights before any production change.

1. Word-boundary precision in the entity-recovery gate. On Opus 5.5, gate_v2 raised grounded from 11% to 36% (36.2%, 34 of 94 cells, 97.1% precision). Why only 36% when 67 of 94 cells had verified bundle evidence? Lifting the link gate isolated the step-4 entity gate. That rule withheld grounded on 28 Opus 5.5 cells (28 of 28 usable, 100%) and 17 Opus 4.6 cells (15 of 17 usable). The rule checked whether the cause text contained terms like "stream", "partition", or templated <*> without populating recovered_entities. Across the 282 Claude cells, substring matching triggered on "downstream" (20 cells), "upstream" (8), or "secure_stream" (6) in 34 cells. In another 28 cells, <*> matched a quoted log template where <*> represented a process ID or duration.
2. Why grounded must stay distinct from tool-call verification across model tiers. On Opus 4.6, promoting bundle-evidence answers to grounded raised the confidently-wrong rate from 6.4% to 12.8% (+6.4 points). It also raised confidently-wrong answers by +5.4 points on Gemini 3.7 Flash and +3.3 points on Gemini 3.8 Flash. Calling a log tool proves the model inspected the bundle, not that its interpretation was sound.
3. Cross-model consensus as a benchmark-label audit. We audited the 4 recurring confidently-wrong cases across all models. On 1 incident in our 47-ticket set, all 9 runs across 3 model families converged on the same root cause. Auditing the ticket confirmed the models had identified the true customer resolution, whereas the benchmark key referenced an earlier timestamp test. Blinded re-judging of all 11 cells on that ticket ($0.84) scored 9 MATCH, 1 PARTIAL, and 1 MISS.

To recover bundle-backed answers safely, we added a middle post-processing tier: supported ("supported by bundle evidence", capped at medium confidence). By annotating the Python type with pydantic.SkipJsonSchema, the JSON schema sent to the model stays 100% byte-identical, preventing any hedging shift. Combined with word-boundary entity matching, supported captured 24 of the 94 Opus 5.5 cells (25.5%, 26% rounded, 24 of 24 usable = 100%). Meanwhile, grounded stayed identical across all 818 replayed cells. After lifting the 4,096 max_tokens cap to 16,000 in October (report 02), Opus 5.5 settled at 22% grounded, 38% supported, and 37% inconclusive on these 47 tickets.

Verification gates and methodology checks

• Judge self-preference. The Gemini-only reading (+10.6 points) and the self-preference indicator (0.0 points on usable, -0.096 on score) ruled out self-favoritism.
• First-repeat analysis. Using only repeat 1 (the original 94-cell plan), the usable gain was +10.6 points (90% interval +4.3 to +19.2), passing the same rule.
• Gate mutation. Comparing raw and post-gate text across all 188 cells confirmed the blinded text was identical in 188 of 188 cells.
• Retry vs schema text. In F55, 31 of 31 retried cells were usable (0 MISSes), while the 45 schema-hedged cells held 7 of 11 MISSes.
• Trace replay fidelity. Before evaluating gate_v2, deterministic replay was verified against stored labels on 818 of 818 historical cells (100%) and 24 of 24 live runs.

Hypotheses falsified by the benchmark

• Treating the 0% confidently-wrong score as evidence that Opus 5.5 was better calibrated (71 of 94 badges had been stripped by the link gate).
• Report 01's attribution of the 43% Claude grounded rate to model self-restraint alone (102 of 102 Wave 2 labels were post-gate).
• The F55 prompt-and-retry fix (+7.4 points grounded vs +25 required; -6.4 points usable accuracy).
• The gate_v2 plan to merge bundle tool calls into grounded (doubled Opus 4.6 confidently-wrong answers from 6.4% to 12.8%).
• The entity gate's un-anchored "stream" and <*> substring checks (34 "downstream"/"upstream" false positives and 28 PID/duration false positives).

Pressure tests

What this does not show

It does not measure engineer handling time saved on live tickets. The F55 fix test combined the validator retry and the schema text change in one arm on the following day. Separating them relies on observational stratification (0 retries on all 11 MISSes) rather than a two-by-two factorial. Finally, the pre-registration files for the September runs were committed after launch or alongside results, so their timestamps are self-attested.

Reproduce

The study repository stays private because it holds customer diagnostic bundles. This bundle ships the report, the four figures, and numbers.json. The file records the exact source file, key, and provenance for every printed value (0.0 maximum discrepancy against endpoints.json).

References

• Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. DOI.
• Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Proceedings of NeurIPS 2015.
• Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-Y., & Chen, Y.-N. (2024). Let Me Speak Freely? EMNLP 2024 Industry Track. DOI.