Why we run local models for high-rejection engineering workloads

The claim

When does owning local GPU capacity beat metered cloud APIs for an enterprise engineering organization?

Enterprise teams usually justify local models through data privacy and regulatory compliance (report 12). Those controls matter, yet Zero Data Retention cloud agreements already cover many compliance requirements (report 11). The stronger operational case for local models is cost structure on high-rejection verification loops.

Engineering AI workloads split into two economic regimes based on candidate rejection rates.

• Low-rejection interactive workloads (<10.0% rejection). In interactive developer chat, a human engineer reads almost every response (>=90.0% acceptance). Metered cloud APIs with prompt caching fit interactive work well (report 04).
• High-rejection background verification loops (80.0% to 95.0% rejection). In background verification loops, deterministic gates reject 80.0% to 95.0% of model proposals (Snell et al., 2024; Brown et al., 2024; Lin et al., 2025). In our security loop, deterministic gates rejected 81.3% (26 of 32) of high-confidence static security claims. In our C++ storage lab, hardware gates rejected 64.3% (9 of 14) of Lean-verified fast-path candidates (report 08). In our Rust log parser, benchmark gates rejected 32.1% (9 of 28) of optimization arms (report 06).

Paying per-token API rates when a compiler, sanitizer, or hardware benchmark discards most candidates taxes verification. Across 2,580 engineering sessions and 212,218 tool executions, our workflows processed 63.04 billion tokens. That volume comprised 59.91 billion cached input, 2.94 billion uncached and cache-write input, and 189.35 million output and reasoning tokens. On metered cloud APIs, that volume represents $30,838 ($30.8k) on mid-tier cached pricing to $154,190 ($154.2k) on frontier cached pricing. On uncached multi-repository sweeps, the same volume represents $191,390 ($191.4k) to $956,950 ($957.0k). Parallel cloud bursts also encountered 3,109 rate-limit (HTTP 429) events.

For high-rejection background loops over public codebases, synthetic fault harnesses, and compiler benchmarks, we run three models on 4 AMD Radeon AI PRO R9700 GPUs. Customer production telemetry never touches this local workstation. All customer support data stays inside regional enterprise cloud boundaries under our ISO/IEC 42001 (2023) controls (report 09 to report 11). The workstation provides 128 GB total VRAM ($3.50 per day or $1,278 per year in electricity) with 0 ms model-swap latency.

1. A 4B single-pass classifier (report 07, 0 output tokens, 212 to 226 ms over local HTTP, 29.0 ms kernel p50, 4.48 GB VRAM at 8-bit).
2. A 1,024-dimension embedding model (76.8 ms per item for semantic deduplication at cosine threshold 0.85, 2.52 GB VRAM, Abbas et al., 2023).
3. A 38B code-analysis model (1,305 to 3,184 tokens/s prefill, 57.7 to 165.3 tokens/s decode, 96.0 GB VRAM with KV cache, 0 rate-limit stalls).

Because marginal token cost is near zero (Alcott, 2005; Ding et al., 2024; Narayan et al., 2025), engineers do not ration pre-release verification runs. We pair these internal verification gates with independent external ISO/IEC 42001 (2023) certification audits by an accredited third-party certification body.

Four rules for routing workloads between cloud APIs and local GPUs

1. Keep low-rejection interactive reasoning on cached cloud APIs. When an engineer reads a warm-cache case briefing, prompt caching cuts cloud generation cost by 71.1% to 74.8% (report 04).
2. Move high-rejection verifier loops to fixed-cost local GPUs. Deterministic gates reject 32.1% to 81.3% of our candidate proposals. Converting per-token OpEx into $3.50/day local power removes the financial penalty on strict verification (Snell et al., 2024; Brown et al., 2024).
3. Filter and deduplicate locally before invoking deep code analysis. Our 4B classifier (29.0 ms p50) and 1,024-dimension clusterer (cos >= 0.85) collapse 338 slices into 245 clusters (-27.5%, Abbas et al., 2023).
4. Pin all pipeline stages co-resident in VRAM (0 model swaps). Allocating 103.0 GB across three models within 128.0 GB VRAM (leaving 25.0 GB for activation buffers) eliminates weight reloading across 24/7 sweeps.

Low-rejection chat vs high-rejection verification loops

Repeated sampling from small models beats single-shot calls when paired with a verifier (Brown et al., 2024; Snell et al., 2024; Shypula et al., 2024). However, verifier-guided search changes the unit economics of inference. In background "sleep-time" pipelines (Lin et al., 2025), the system generates many candidate patches, proofs, or exploit traces. A deterministic oracle then rejects every invalid branch.

Across three high-rejection verification loops (one open-source security audit and two internal systems labs), deterministic gates rejected 44 of 74 (59.5%) post-triage proposals.

• Security auditing (81.3% rejection, 26 of 32 static claims). Standalone models suffer false-positive rates above 50.0% to 80.0% on static review because caller bounds live outside the local slice (Ullah et al., 2024). Across 68 repositories, our 4B gate and 1,024-dimension SemDeDup stage reduced 338 hazard slices to 245 clusters (93 removed, 27.5%). Out of 32 high-confidence static claims escalated by the reasoning tier, compiled sanitizers (ASan and UBSan) and socket harnesses refuted 26 claims (81.3%). Only 6 claims (18.7%) from that cohort survived execution.
• C++ storage fast-path optimization (64.3% rejection, 9 of 14 candidates). In report 08, we evaluated 14 C++ storage fast-path candidates using 170 Lean 4 theorems and 36.0M to 251.9M fuzz checks. We then ran 28-run single-VM ABBA hardware benchmarks. Even after formal equivalence proofs passed, hardware gates rejected 9 of 14 candidates (64.3%) that regressed on x86 or fell inside the 2.0% noise floor. Only 5 candidates (35.7%) were promoted.
• Rust log-parser optimization (32.1% rejection, 9 of 28 arms). In report 06, we tested 28 pre-registered optimization arms on our Rust log-to-Parquet engine (currently being prepared for open-source release) across 32 bundles. Our 1.10x peak-RSS, liveness, and 128-bit multiset-equivalence gates rejected 9 of 28 arms (32.1%) and accepted 19 (67.9%).

Telemetry across 2,580 sessions, 63.04 billion tokens, and 3,109 rate-limit stalls

We audited token and rate-limit telemetry across 2,580 engineering sessions and 212,218 tool executions (82.3 tool calls per session on average). That ledger quantifies the cost of running iterative agent workflows on metered APIs versus local hardware.

In Table 2, the base token rows sum to $29,633 on mid-tier cached pricing ($17,973 cached read, $8,820 uncached input, and $2,840 output). On frontier cached pricing, those base rows sum to $148,166 ($89,865 cached read, $44,100 uncached input, and $14,201 output). Adding the 1.25x cache-write creation surcharge (+$1,205 mid-tier and +$6,024 frontier) yields the billed totals of $30,838 and $154,190. Amortizing an $8,000 4-GPU workstation over 3 years adds $2,667/year in CapEx to $1,278/year in power ($3,945/year, or $10.81/day all-in TCO). That all-in TCO still cuts annual cost by 7.8x to 39.1x against cached cloud APIs and 48.5x to 242.6x against uncached sweeps.

Three operational findings emerged from the 63.04-billion-token ledger:

1. Prefill dominates agentic verification (99.70% input vs 0.30% output). Across 212,218 tool calls, input tokens accounted for 99.70% of volume (59.91 billion cached at 95.0% plus 2.94 billion uncached at 4.7%). Output and reasoning tokens accounted for only 189.35 million (0.30%). Verification loops read large codebases, AST slices, and sanitizer traces repeatedly while emitting compact verdicts or diffs.
2. Multi-repository sweeps break prompt-cache economics ($30.8k vs $191.4k). In a single session, a 95.0% cache hit rate holds cost to $30,838 ($30.8k) on mid-tier and $154,190 ($154.2k) on frontier APIs. However, when a background daemon sweeps across 68 distinct repositories, context prefixes rotate continuously. Without warm prefix hits, sweeping 63.04 billion tokens costs $191,390 ($191.4k) on mid-tier APIs and $956,950 ($957.0k) on frontier APIs. That uncached cloud spend is 149.8x to 748.8x above our $1,278 annual local power cost.
3. Cloud rate-limit quotas (3,109 HTTP 429 events) throttle parallel fan-out. When multi-worker audit harnesses burst across candidate slices concurrently, cloud endpoints enforce tokens-per-minute and requests-per-minute ceilings. Our telemetry recorded 3,109 HTTP 429 rate-limit stalls, whereas local GPUs executed parallel waves with 0 quota stalls.

Co-resident 128 GB VRAM pipeline: 4B triage, 1,024-d SemDeDup, and 38B code analysis

Hybrid routing studies filter large contexts with small models first (Chen et al., 2023; Ding et al., 2024; Ong et al., 2024). In MinionS, local small models cut cloud spend by 5.7x while recovering 97.9% of frontier accuracy (Narayan et al., 2025).

To run that cascade continuously without per-token billing, we provision a local workstation with 4 AMD Radeon AI PRO R9700 GPUs (32 GB per GPU, 128.0 GB total VRAM). All three models stay pinned in VRAM simultaneously with 0 ms model-swap overhead.

Each co-resident tier targets a specific bottleneck in the verification funnel:

1. Tier 1 (4B single-pass classifier, 4.48 GB VRAM at 8-bit Q8_0). Qualified in report 07 and report 12, this model emits calibrated float32 probabilities in one prefill pass (0 output tokens, Kadavath et al., 2022). Local GPU kernel latency is 29.0 ms p50. Warm local HTTP latency across 6 concurrent workers is 212 to 226 ms (219.0 ms midpoint, 11.7 requests/s burst). It scores candidate hazard slices via unclamped 2-option choice routing (escalate vs suppress at threshold 0.28) with symmetrized (A, B) and (B, A) runoff.
2. Tier 2 (1,024-dimension embedding model, 2.52 GB VRAM at BF16). Parallel static slicers and fuzzers frequently surface the same defect pattern across multiple call sites. Before waking the 38B model, we embed each normalized slice header into a 1,024-dimension L2-normalized vector in 76.8 ms per item (13.0 items/s). Applying SemDeDup clustering (Abbas et al., 2023) at a 0.85 cosine threshold collapsed 338 static slices into 245 unique clusters (eliminating 93 near-duplicates, or 27.5%). One exemplar per cluster advances with its multiplicity count attached.
3. Tier 3 (38B code-analysis model, 96.0 GB VRAM with 2-worker KV cache). Because 99.70% of our tokens are input prefill, the 38B model's 1,305 to 3,184 tokens/s prefill speed (2,244.5 midpoint) keeps cross-file analysis fast. Multi-token-prediction decode runs at 57.7 to 165.3 tokens/s (111.5 midpoint, 61% to 85% draft acceptance). It operates in two passes: Pass 1 (2,048 max tokens) checks caller invariants to filter false positives, and Pass 2 (4,096 max tokens) synthesizes minimal patches.

When marginal inference cost drops from $30.8k-$957.0k to $3.50/day in electricity, engineering behavior shifts. In resource economics, Jevons' paradox describes how lowering the unit cost of a resource expands total demand rather than merely saving money (Alcott, 2005). Under metered billing, teams ration multi-repository security sweeps and 10,000-seed fault-injection loops. On fixed-cost local GPUs, those verification loops run continuously (cutting test-time compute by 5.0x via offline pre-computation, Lin et al., 2025), and only verified proofs reach engineers.

Verification gates and methodology checks

• Token and rate-limit ledger audit (2,580 sessions, 212,218 tool calls). Verified that cached input (59.91B), uncached input (2.94B), and output (189.35M) tokens sum to 63,039.35M (63.04B) tokens, with 3,109 logged HTTP 429 cloud rate-limit stalls.
• Co-resident VRAM and zero-swap verification (128.0 GB across 4 GPUs). Verified that the 4B classifier (4.48 GB), 1,024-d embedding model (2.52 GB), and 38B model (96.0 GB) remain resident simultaneously with 0 model-eviction swaps.
• Prompt-Exclusion and unclamped choice guardrails (report 07). Verified that Tier 1 (4B) excludes raw line and commit frequencies from prompt text (fusing priors in log-odds space) and uses unclamped 2-option choice routing (0 output tokens).
• Deterministic execution gate before human review (74 proposals across 3 loops). Verified that every escalated security claim, C++ patch, and Rust arm is gated by a sanitizer (ASan/UBSan), Lean 4 proof, or ABBA benchmark.

Infrastructure antipatterns rejected by our gates

1. Justifying local models solely through data privacy. Zero Data Retention cloud contracts already cover many compliance paths. Local GPUs win primarily through near-zero marginal cost on 80% to 95% rejection loops.
2. Running uncached multi-repository background sweeps on metered frontier APIs. Rotating across 68 repositories invalidates prompt caches, turning a $30.8k cached workload into a $191.4k to $957.0k bill with 3,109 HTTP 429 stalls.
3. Sending every raw static slice directly to a reasoning model without SemDeDup. Skipping the 29.0 ms 4B classifier and 76.8 ms embedding stage wastes 27.5% of 38B calls on duplicates (cos >= 0.85).
4. Time-sharing a single GPU by swapping model weights from disk. Reloading a 38B model between triage and audit stages adds multi-second I/O stalls. Pinning all 3 models across 128 GB VRAM holds swap overhead at 0 ms.
5. Accepting unverified static model claims without an execution oracle. Rejected after sanitizers, Lean 4 proofs, and ABBA gates refuted 81.3% of security claims, 64.3% of C++ patches, and 32.1% of Rust arms.

Pressure tests

Every finding was checked against the six standing fronts in Methods.

What this does not show

First, our local 4-GPU workstation processes only public open-source repositories, synthetic fault-injection inputs, and compiler benchmarks. Customer support tickets, diagnostic bundles, and production telemetry never run on unmanaged desktop GPUs. Customer data stays inside governed regional cloud boundaries (report 09 to report 11). Frontier cloud models also handle low-rejection incident synthesis (report 02, report 03).

Second, our $3.50 per day ($1,278 per year) operating figure measures electricity alone, before hardware depreciation. Amortizing an $8,000 4-GPU AMD Radeon AI PRO R9700 (128 GB VRAM) workstation over 3 years ($2,667/year CapEx plus $1,278/year power) gives $3,945/year ($10.81/day all-in TCO). Across 2,580 sessions (63.04B tokens), that all-in TCO still cuts annual cost by 7.8x to 39.1x against cached APIs (48.5x to 242.6x uncached).

Third, near-zero marginal token cost improves engineering quality only when paired with a deterministic verifier. Without compiled AddressSanitizer harnesses, Lean 4 proofs (report 08), or single-VM ABBA benchmarks (report 06), running local models 24/7 merely generates unverified code suggestions faster.

Reproduce

All metrics in this report and its three figures are stored in numbers.json, with SHA-256 integrity digests recorded in MANIFEST.md.

References

• Abbas, A., Tirumala, K., Simig, D., Ganguli, S., & Morcos, A. S. (2023). SemDeDup: Data-Efficient Learning at Web-Scale Through Semantic Deduplication. arXiv preprint. arXiv.
• Alcott, B. (2005). Jevons' Paradox. Ecological Economics. DOI.
• Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Re, C., & Mirhoseini, A. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv preprint. arXiv.
• Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv preprint. arXiv.
• Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., & Awadallah, A. H. (2024). Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. Proceedings of ICLR 2024. arXiv.
• ISO/IEC. (2023). ISO/IEC 42001 (2023): Information Technology - Artificial Intelligence - Management System. International Organization for Standardization.
• Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., et al. (2022). Language Models (Mostly) Know What They Know. arXiv preprint. arXiv.
• Lin, K., Snell, C., Wang, Y., Packer, C., Wooders, S., Stoica, I., & Gonzalez, J. E. (2025). Sleep-Time Compute: Beyond Inference Scaling at Test-Time. arXiv preprint. arXiv.
• Narayan, A., Biderman, D., Eyuboglu, S., May, A., Linderman, S., Zou, J., & Re, C. (2025). Minions: Cost-Efficient Collaboration Between On-Device and Cloud Language Models. arXiv preprint. arXiv.
• Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv preprint. arXiv.
• Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2024). Learning Performance-Improving Code Edits. Proceedings of ICLR 2024. arXiv.
• Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arXiv preprint. arXiv.
• Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., & Stringhini, G. (2024). LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?). Proceedings of IEEE Symposium on Security and Privacy (S&P 2024). arXiv.