Using a single-pass 4B model for decision making

The claim

In our support stack, the same generative language models that write root-cause drafts were also making discrete decisions. Those decisions included assigning a ticket to one of 9 categories, scoring customer dissatisfaction risk, ranking diagnostic rules, and judging candidate answers.

Generative models handle open-ended text well and discrete probability poorly. Across 4,000 production classification traces, our Gemini classifier saw 793 distinct ticket inputs at least twice. It changed its assigned category on 161 of those 793 repeated inputs (20.3%). Across 12 generative models on 533 CSAT-rated tickets (Vidigal, 2026), verbalized confidence collapsed onto 5 to 20 discrete numbers (Xiong et al., 2024).

We tested moving those discrete decisions to a 4B single-pass decision model (h2o-lightning-4b, 0 output tokens). The model runs 24 causal Gated DeltaNet layers followed by 8 bidirectional full-attention layers (32 layers total). Its top 8 layers attend both ways across the input and the candidate options. One prefill pass returns a float32 probability distribution (Kadavath et al., 2022). On a local GPU, median latency is 29.0 ms (280.3 ms over HTTPS keep-alive), with 0 flips across 400 sequential repeat calls.

The model is fast and repeatable. Qualifying it for production across five workflows required solving five deployment boundary conditions that standard benchmarks miss. Putting historical rule fire rates inside the prompt reversed the diagnostic ranking (0.667 to 0.402 AUROC), which we solved via post-hoc log-odds fusion. A hosted endpoint flag (noul_floor = 0.801) clamped probabilities between 0.199 and 0.801 (6 of 6 uncertain escalations), which we bypassed using a 2-option choice head. Passing 3 multi-paragraph root-cause reports into one choice call shifted winners on 42.9% to 81.5% of cases when slot order changed. We eliminated that order sensitivity with a 3-stage cyclic pairwise runoff. Quantizing weights to 4-bit flipped 5.4% to 8.7% of borderline safety verdicts, establishing our 8-bit production floor. And splitting compound agent turns at sentence boundaries prevented opening status lines from masking durable rules.

I lead Applied AI and Support at Yugabyte. The systems tested here are our internal support triage, our automated cluster-health engine, MCP privacy, and evaluation pipelines (report 01 and report 02). Nothing here is a product for sale.

What to check before deploying a small decision model

1. Keep base rates out of the prompt. When historical frequencies sit in the prompt, the model anchors on the number (Tversky & Kahneman, 1974). Combine model probabilities with empirical base rates in log-odds space after the call (Saerens et al., 2002).
2. Probe your endpoint for probability clamping. Send borderline inputs and check whether probabilities between 0.20 and 0.80 snap to fixed floors. If they do, query a 2-option choice head instead of a binary head.
3. Permute option order when candidates are long. Single-pass choice is order-invariant on short category labels. On multi-paragraph candidate reports, run a cyclic permutation and a forward-and-reverse (A, B) / (B, A) pairwise tie-break (Pezeshkpour & Hruschka, 2024).
4. Test 4-bit vs 8-bit quantization on boundary flips, not net accuracy. Moving from BF16 to 4-bit moved net accuracy by less than 0.3 points while flipping up to 8.7% of borderline safety decisions.

Production benchmarks across classification, CSAT calibration, and redaction

1. Replaying 793 ticket classifications

Our production classifier assigns each incoming ticket to one of 9 categories (Bug Report, Performance Issue, Configuration Help, Usage Question, and 5 others). We pulled 4,000 recent classification traces and isolated the 793 distinct input texts that Gemini had classified 2 or more times.

Gemini gave the same category on every run for 632 inputs (79.7%) and contradicted itself on 161 inputs (20.3%). In an offline shadow-mode evaluation, we replayed all 793 distinct inputs through h2o-lightning-4b using the production category definitions (type: "choice").

Across all 793 inputs, the 4B model agreed with Gemini's majority vote on 62.2% (68.0% on the 632 inputs where Gemini agreed with itself). Agreement tracked the 4B model's probability across all four bands (Guo et al., 2017; Murphy, 1973). On the 255 tickets (32.2% of the queue) where 4B confidence reached 0.90 or higher, agreement with Gemini's majority reached 82.7%.

Gemini's majority vote is not ground truth. Under our ISO/IEC 42001 Clause 9.1 human calibration rule, production cutover is gated on completing our 130-ticket blind annotation benchmark (issue-category-gold-v1). It oversamples the 32 Gemini self-contradictions and the 16 high-confidence (>=0.90) disagreements (plus 45 unsure disagreements, 25 controls, and 12 confident agreements). Engineers see only the ticket text.

2. Continuous probability resolution on 533 CSAT-rated tickets

Our ACL 2026 study evaluated all 533 CSAT-rated support tickets over 5 years (26 negative CSAT, 4.88% base rate, or 4.9% rounded). Across those 533 tickets, 12 generative models emitted only 5 to 20 distinct verbalized confidence values (Vidigal, 2026). Claude Opus 4.6 ($67.50 per 1,000 tickets) predicted "Neutral" on 41.3% of tickets and scored a Matthews Correlation Coefficient (MCC) of 0.165. Gemini 3 Flash scored 0.347 MCC at $0.60 per 1,000 tickets.

Running the 533 tickets through h2o-lightning-4b (0 output tokens, $0.028 to $0.032 per 1,000 tickets, 280.3 ms median latency) produced 525 distinct float32 probabilities (98.5% unique, and 531 after out-of-fold Platt scaling). Across 40 tickets repeated 10 times (400 calls), the choice head had 0 verdict flips. It ran 51.7x faster than Gemini 3 Flash and cost 2,437x less than Opus 4.6. Expected Calibration Error (ECE, Guo et al., 2017) dropped to 0.0325 under prior-shift correction and 0.0046 under out-of-fold Platt scaling.

3. Contextual redaction at 29 ms on GPU (323 MCP tool outputs)

On a 77-item synthetic red-team suite for our 53-tool MCP server, standard word-boundary regexes missed 27 of 33 compound secret patterns. Those misses included all 27 prefixed or camelCase keys (such as ldapbindpasswd=). Google Cloud DLP masked 18 of 18 database identifiers as phone numbers or credit cards. Under its 100-RPM quota, Cloud DLP also stalled for 57 to 59 seconds on 10.3% of calls.

We paired our 33-rule deterministic redactor with the local 8-bit 4B model under a monotonic OR rule. Local GPU median latency is 29.0 ms (29.1 ms in the 323-output firewall replay in report 10, 331.9 ms over HTTPS). The 4B model can add redactions but can never unmask a deterministic match. Across 77 red-team items (49 secrets and PII, 28 technical IDs), the gate caught 100.0% of secrets with 0.0% false positives on 18 database IDs. On 323 replayed production tool outputs, it reached 99.4% recall at a 0.42% false-positive rate (F1 = 0.995) with 0% quota stalls.

Five production engineering controls established by our verification gates

Control 1: post-hoc log-odds fusion vs in-prompt base-rate anchoring

In our automated cluster-health engine, some deterministic rules fire on over 70% of support bundles (environmental noise). Rare rules carry much higher diagnostic value. Under Bayes' rule, a rule's positive likelihood ratio scales inversely with its background fire rate $f_r$.

Initially, we passed each rule's historical fleet fire rate $f_r$ inside the prompt so the 4B model could weigh rarity. Across all 110 evaluated rules, the model did the opposite of Bayes' rule. It anchored on the number in the prompt (Tversky & Kahneman, 1974). Predicted actionability shifted up by +8.3 points (+0.083) when f_r > 70% and down by -12.8 points (-0.128) on rare rules (Spearman rho = +0.75).

On the 29 human-labelled rules (12 positive), putting f_r in the prompt pushed AUROC from 0.667 down to 0.402 (0.4020, under the 0.50 random baseline). We then removed all frequencies from the prompt and added logit(1 - f_r) in log-odds space (Saerens et al., 2002; Zhao et al., 2021). That raised AUROC to 0.843 (+44.1 points over the prompted prior). It also cut selective risk (AURC, Geifman & El-Yaniv, 2017) by 49.4% (0.626 to 0.317).

Notice the third row in the table above. Deterministic rarity alone ($1 - f_r$, without the 4B model) already reaches 0.816 AUROC (0.288 AURC). On rule priors alone ($n=29$), the 4B log-odds fusion adds +2.7 AUROC points over rarity alone with wide overlapping intervals.

The 4B model earns its place on case-specific context (83 case-finding pairs and 18 fault-injection bundles). On the 83 pairs, scalar Platt scaling collapsed to the base rate (0.194 Brier vs 0.195). Applying case-level (hierarchical log-odds fusion) calibration cut case Brier by 30.5% (0.240 to 0.167, ECE -44.6%, E-AURC -36.0%). On the 18 fault-injection trials, it lifted Top-1 root-cause recall from 0% (0.138 MRR) to 100% (18 of 18 trials, 1.0 MRR).

Control 2: 2-option choice routing to bypass noul_floor = 0.801 probability clamping on 6 of 6 escalations

Our legacy PR quality gate needed 15 generative judge calls per ticket (750 calls per 50-ticket PR, taking >18.5 minutes). With the 4B model, 1 call per ticket (50 calls) finishes in 9.44 seconds at 8-way concurrency (Cohen's kappa = 0.918). To guard borderline cases under Learn-Then-Test risk control (Angelopoulos et al., 2025), we set an abstention band of (0.20, 0.80) at threshold tau = 0.80.

We queried the hosted endpoint's binary type: "noul" head across 30 rubric decisions (6 cases, 5 dimensions, 10 repeats). Zero of the 30 decisions landed between 0.20 and 0.80. Every uncertain item was stamped at 0.801 or 0.199.

The hosted server ran with a default post-softmax clamp (noul_floor = 0.801). Any probability in $[0.50, 0.801)$ was pushed up to 0.801, and any probability in $(0.199, 0.50)$ was pushed down to 0.199. That created a 0.602 jump discontinuity at 0.50, suppressing 100.0% (6 of 6 down to 0 of 6) of uncertain escalations at tau = 0.80. It also caused 8 verdict flips across 324 repeat checks (2.5%, or 2.47% exact) on items near 0.50.

The fix required zero server changes. Querying the exact same binary predicate on the exact same server as a 2-option type: "choice" (pass vs fail) bypassed the noul clamp completely. It returned continuous probabilities (0.448 to 0.771 on the 6 uncertain items) and preserved all 6 of 6 escalations (20.0% escalation rate). It achieved 100.0% accuracy (24 of 24) on non-escalated decisions and produced 0 flips across 324 repeat checks (0.0%).

Control 3: enforcing an 8-bit quantization floor after 4-bit flipped 5.4% to 8.7% of boundary verdicts

We tested local deployment of the 4B gatekeeper under 4-bit weight quantization (Q4_K_M in GGUF and 4-bit in MLX). Net benchmark accuracy changed by less than 0.3 points. Across items near the decision threshold, 4-bit quantization silently flipped 5.4% (Q4_K_M) to 8.7% (MLX 4-bit) of verdicts compared to BF16. Running at 8-bit (Q8_0, 4.48 GB VRAM) cut boundary flips to 1.4% (98.6% agreement with BF16). For safety and compliance gates, we ban 4-bit quantization and require an 8-bit floor.

Control 4: 3-stage cyclic pairwise runoff to eliminate 42.9% to 81.5% slot-order sensitivity

In production, we run a 3-model ensemble on support bundles (Claude Opus 4.6 plus two Gemini legs). On Wave 1 (21 incidents) and Wave 2 (92 incidents), the 3-model oracle ceiling was 81.0% (17 of 21) and 64.1% (59 of 92). The best single model (Opus 4.6) scored 66.7% (14 of 21) and 47.8% (44 of 92). Majority voting (consensus_selector) dropped to 52.4% (11 of 21) and 37.0% (34 of 92) due to shared errors between the cheaper models.

We tested using h2o-lightning-4b (type: "choice") to select the best report among the 3 candidates (A, B, C). Single-pass choice was order-invariant on short category names. Passing 3 multi-paragraph RCA reports into one choice call failed our permutation check (Zheng et al., 2023; Pezeshkpour & Hruschka, 2024).

Permuting the 3 reports across all 6 slot orders flipped the winner on 42.9% (9/21), 52.4% (11/21), and 81.5% (75/92) of cases. On Wave 2, unpermuted single-pass choice scored 37.0% (34 of 92), matching majority vote.

To remove order sensitivity, we built a 3-stage cyclic pairwise runoff selector. Stage 1 scores each report's evidence density in isolation. Stage 2 averages choice probabilities across all 3 cyclic Latin-square shifts ((A,B,C), (B,C,A), (C,A,B)). Stage 3 runs a symmetrized binary runoff between the top 2 contenders, averaging forward (A, B) and reverse (B, A) log-odds. That reduced positional flips to 0.0% (0 of 21 and 0 of 92, position consistency PC = 1.0). Exact match rose to 71.4% on Wave 1 (76.2%, or 16/21, with cyclic choice alone). On Wave 2, it reached 50.0% (46/92, +13.0 points over unpermuted choice).

On our 787-run RCA calibration benchmark (368 Wave 2 runs), we also combined our leakage-free runtime evidence ledger with the 4B gate. Usable AUROC rose from 0.463 to 0.725 (0.723 ledger alone, 0.588 4B alone). Confidently-wrong false positives fell from 16.8% (16.85%) to 2.7% (2.72%). On 6 healthy negative-control bundles, it abstained on 6 of 6 (100.0%), whereas verbalized confidence over-claimed on 4 of 6.

Control 5: batch-size invariance and sentence-boundary splitting for agent memory

Two additional runtime controls emerged from our RAG and agent-memory verification suite:

1. Dynamic-batching bfloat16 jitter (150 RAG calls). At batch_size = 1, repeated calls are bit-for-bit identical (0.0 variance). When 5 concurrent workers graded 15 query-chunk pairs across 10 repeats (150 calls), dynamic batching changed bfloat16 reduction order (max probability delta 0.106 on 4-level grades, 0.040 on binary). Binary verdicts had 0 of 150 flips. One borderline distractor chunk (p_0 = 0.45 vs p_1 = 0.48) flipped its 4-level grade on 3 of 10 runs (2.0% overall). Strict boundary gates need batch_size = 1 or batch-invariant kernels.
2. Compound-turn dilution in memory gating (94 calls). When filtering agent memory writes, whole-turn scoring rejected 20 of 20 noise items (100.0%). Agents also wrote compound turns mixing a status prefix with a rule ("Translated InvoiceRecord.cob to InvoiceRecord.java. Used BigDecimal with scale 2..."). Whole-turn scoring rejected all 4 of 4 compound turns (noul 0.088 to 0.199) because the opening status clause dominated bidirectional attention. Splitting on sentence boundaries (". ") rejected 4 of 4 status clauses (noul <= 0.076) and kept the BigDecimal convention (0.878 noul, 0.936 choice). Preflight retrieval hits rose from 0% to 100.0% (4/4), and Tier-3 escalation handoff net tokens dropped by 12.7% (52.0% fewer setup tokens across 20 turns).

Combining operational telemetry with text to catch polite B2B churn

On the 533 CSAT-rated tickets, running the text-only 4B model on customer text alone beat Claude Opus 4.6 (0.259 vs 0.165 MCC). It also cut "Neutral" hedging in half (41.3% to 21.2%) with 50.0% precision (4 false positives). However, recall on dissatisfied tickets sat at 15.4% (23.1% with Saerens prior shift, 26.1% on the 497-ticket subset). Gemini 3 Flash reached 65.4% recall (0.347 MCC).

Why did a 4B single-pass reader miss those tickets on text alone? In B2B support, 10 of the 26 dissatisfied accounts wrote polite, neutral prose (0 of 10 caught by the text-only 4B model or Opus 4.6). Meanwhile, Gemini 3 Flash suffered 54 false positives (76.1% false-discovery rate). Urgent P1 outages read as angry in text yet receive 5-star CSAT after fast recovery.

We then passed structured operational state (bundle upload counts, comment velocity, and license entitlement errors) alongside text into the multimodal text-plus-telemetry 4B model. Under 5-fold out-of-fold calibration, MCC rose to 0.485 (0.464 Saerens, 0.567 CV-optimal). That beat a matched tabular-plus-TF-IDF baseline at 0.332 out-of-fold MCC. False positives dropped by 90.7% (54 down to 12 under 5-fold out-of-fold calibration, and 6 under CV-optimal, an 88.9% reduction). The model caught 8 of the 10 (80.0%) silent-churn tickets.

Verification gates and methodology checks

• Offline shadow-mode replay vs human ground truth (n=793). Shadow-mode agreement with Gemini (62.2% overall, 82.7% at >=0.90 confidence) measures concordance, not accuracy. Production cutover is gated on completing the 130-ticket blind engineer annotation benchmark (issue-category-gold-v1, Clause 9.1 human calibration rule).
• Deterministic rarity vs 4B log-odds fusion (n=29 rules). We tested $1 - f_r$ alone without the 4B model. Rarity alone reached 0.816 AUROC vs 0.843 with the 4B model (+2.7 points). We report the rule-level gain as modest and credit the 4B model only where case-specific text matters (n=83 case pairs and 18 fault-injection trials).
• Pre-deployment feature-provenance gate on RCA calibration (n=787). An offline pilot script included two post-resolution manifest columns (verified_bundle_root_cause and is_ctrl) prior to any production deployment. Those columns inflated pilot Wave 2 AUROC to 0.954 and Wave 1 runoff to 81.0% (17/21). Our provenance check (commit bc5c40d) caught and removed both columns offline, establishing the verified baselines (0.725 Wave 2 AUROC, 71.4% Wave 1 and 50.0% Wave 2 runoff).
• Matched multimodal baseline on CSAT (N=533). We tested whether the multimodal text-plus-telemetry 4B jump from 0.259 to 0.485 out-of-fold MCC came only from adding operational columns. A 5-fold out-of-fold tabular-plus-TF-IDF baseline trained on the exact same features scored 0.332 MCC (+15.3 points behind the multimodal text-plus-telemetry 4B model).

Configurations rejected by our verification gates

1. Passing historical rule fire rates $f_r$ inside the prompt (AUROC dropped from 0.667 to 0.402, Spearman rho = +0.75).
2. Using the hosted binary noul head for selective abstention (noul_floor = 0.801 clamped all interior probabilities and suppressed 6 of 6 escalations).
3. Unpermuted single-pass choice over multi-paragraph RCA reports (flipped winners on 42.9% to 81.5% of incidents across $3! = 6$ slot orders).
4. 4-bit quantization (Q4_K_M and MLX 4-bit) for safety gatekeepers (flipped 5.4% to 8.7% of borderline verdicts despite <0.3 points net accuracy change).
5. Whole-turn memory gating without sentence splitting (rejected 4 of 4 compound turns at 0.088 to 0.199 noul until split on ". ").
6. Pure-text 4B classification for B2B silent churn (15.4% to 26.1% recall without operational friction state).

Pressure tests

Every finding was checked against the six standing fronts in Methods.

What this does not show

The 793-ticket category replay is an offline shadow-mode evaluation, and production cutover remains gated on our 130-ticket blind human annotation benchmark (issue-category-gold-v1, Clause 9.1). Until human annotation finishes, the replay reports concordance with Gemini's majority vote (62.2% overall, 82.7% at >=0.90 confidence) rather than ground-truth accuracy.

On the 29 ground-truth-labelled diagnostic rules, the 95% bootstrap interval for log-odds fusion (0.663 to 0.970) overlaps deterministic rarity alone (0.642 to 0.949). On Wave 2 RCA calibration (n=368), the deterministic evidence ledger (Ablation A, 0.723 AUROC) carries almost all of the gain over verbalized confidence (0.463). The 4B head adds 0.2 points on Wave 2 (0.725) and 5.1 points on Wave 3 (0.628 to 0.680 on n=87).

Finally, 3-stage cyclic pairwise runoff closed 33.3% of the gap to the 3-model oracle ceiling on Wave 1 (71.4% vs 81.0% oracle). On Wave 2 (n=92), it reached 50.0% against a 64.1% oracle ceiling (13.3% gap recovery). When two cheaper models agree on a symptom and only one finds the deeper bug, a 4B selector still misses 14.1 points of the oracle.

Reproduce

Every number printed in this report and its four figures is recorded in numbers.json, with file integrity hashes listed in MANIFEST.md.

References

• Angelopoulos, A. N., Bates, S., Candes, E. J., Jordan, M. I., & Lei, L. (2025). Learn Then Test: Calibrating Predictive Algorithms to Achieve Risk Control. The Annals of Applied Statistics. DOI. arXiv.
• Geifman, Y., & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. Advances in Neural Information Processing Systems (NeurIPS 2017). arXiv.
• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of ICML 2017. arXiv.
• Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., et al. (2022). Language Models (Mostly) Know What They Know. arXiv preprint. arXiv.
• Murphy, A. H. (1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology. DOI.
• Pezeshkpour, P., & Hruschka, E. (2024). Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. Findings of NAACL 2024. DOI. arXiv.
• Saerens, M., Latinne, P., & Decaestecker, C. (2002). Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation. DOI.
• Tversky, A., & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science. DOI.
• Vidigal, P. (2026). From TextBlob to LLM Agents: Sentiment Model Selection for B2B Technical Support with CSAT Ground Truth. Proceedings of ACL 2026 Industry Track. DOI.
• Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. Proceedings of ICLR 2024. arXiv.
• Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate Before Use: Improving Few-Shot Performance of Language Models. Proceedings of ICML 2021. arXiv.
• Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS 2023). arXiv.