Running local models inside fault-injection and chaos harnesses

The claim

How can local models uncover operational failure modes that standard unit tests and golden benchmarks miss? Standard test suites verify expected inputs on healthy nodes. In production distributed systems, incidents usually stem from malformed inputs, process races, or cross-component interactions (Basiri et al., 2016; Yuan et al., 2014).

Uncovering those failure modes requires synthesizing hundreds of adversarial inputs, kill schedules, and container topologies (Kingsbury, 2019; Meng et al., 2023; Pan et al., 2024). Under metered cloud APIs, running multi-agent chaos loops across every pre-release build is costly because most generated probes pass without failing.

Our local 128 GB VRAM GPU stack (report 13) has 0 per-token surcharge. That cost structure lets us run role-specialized local agents (38B code-analysis model at 1,305 to 3,184 tokens/s prefill) inside deterministic fault-injection harnesses. The local agents read source code and synthesize malformed archives, truncated streams, concurrent writers, and container topologies (Vikram et al., 2023; Kikuta et al., 2025). Every synthesized fault executes against a compiled binary and a deterministic pass/fail oracle, complementing our sanitizer-gated security workflow (report 14).

We applied this pre-release verification workflow across two core infrastructure systems before deployment, achieving 0 customer production impact.

• Pre-release fault injection on our Rust log-to-Parquet engine (18 edge cases isolated prior to open-source release). Our Rust log-to-Parquet engine (report 06, report 08) is currently being prepared for open-source release under the Apache 2.0 license. We ran two fault-injection rounds: 5 stress clusters (7 input-boundary fixes) in Round 1 and 11 concurrency and file-lifecycle scenarios in Round 2. Together, the harness isolated and verified fixes for 18 edge cases before release. All 18 invariants are locked into 496 automated workspace regression tests (0 failures).
• Go-to-Rust node telemetry collector verification (15 rounds across 6 container and Kubernetes topologies). We ported our node telemetry collector from Go (25.05 to 52.57 MB binary) to Rust (4.20 to 6.55 MB static binary). Across 1,500 synthetic test secrets (0 false positives), Rust redacted logs 9.1x to 13.5x faster (580 to 850 MB/s vs 61 to 93 MB/s). We then ran a 15-round topology harness across 6 Docker and Kubernetes configurations (0.013 to 0.087 CPU cores per node, +8.98% p99 database latency within our 10.0% gate). That harness isolated and eliminated a 14.4x collection latency stall (24.5 s reduced to 1.7 s) before customer deployment (0 customer production impact).

Four design rules for model-driven fault injection

1. Use the local model to synthesize adversarial inputs, never to judge pass or fail. Language models are effective at reading parser or collector source code and proposing boundary-breaking inputs (Vikram et al., 2023; Claessen & Hughes, 2000). The pass/fail oracle remains a deterministic binary check: process exit codes, flock locks, DuckDB Parquet reads, or statement_timeout counters.
2. Test process lifecycle and SIGKILL recovery alongside input parsing. File parsers that pass 100% of unit tests can still corrupt output directories when two processes write concurrently. Likewise, an out-of-memory SIGKILL during shard merge can leave a 0-byte file behind (Alvaro et al., 2015).
3. Verify every fix with a paired pre-remediation vs post-remediation (PRE/POST) binary harness. In Round 2, our first filename-cap patch truncated the .parquet extension at 255 bytes and passed a naive unit test vacuously. The PRE/POST binary harness caught that 0 files matched **/*.parquet, forcing a 247-byte cap that reserves space for .parquet and temporary staging suffixes.
4. Replace single-container smoke tests with multi-version, degraded cluster topologies. Distributed collectors fail on split master and tserver roles, unreachable seed nodes, read-only /tmp mounts (0555), or disabled features like change-data-capture (CDC).

Pre-release fault injection on our Rust log-to-Parquet engine (18 edge cases)

Before releasing our Rust log-to-Parquet engine under the Apache 2.0 license, we ran role-specialized local agents (38B code-analysis model) across its workspace. Each agent role audited a distinct failure surface: stream truncation, parser recursion, timestamp arithmetic, concurrent file I/O, and filesystem path limits.

Across two fault-injection rounds, the harness isolated and verified fixes for 18 pre-release edge cases (7 in Round 1 and 11 in Round 2).

Four of the 18 pre-release fixes illustrate how model-synthesized faults catch edge cases that golden benchmarks miss.

1. Bounding parser recursion on 30,000 chained key= tokens to maximum depth 16 (Round 1, F1). On normal logs, key-value decomposition recurses 1 to 3 levels deep. The local agent synthesized a single log line containing 30,000 chained key= tokens, which overflowed the thread stack via $O(N)$ recursion. Capping token recursion at depth 16 converted stack usage to $O(1)$ and processed all 30,000 tokens in milliseconds (0 panics across all 5 clusters).
2. Enforcing a 64x gzip decompression ratio cap to block archive-expansion bombs (Round 1, F3). As shown in report 06, legitimate compressed log files expand by 5x to 41x. Reading an adversarial .gz stream without a ratio bound risks host memory exhaustion or silent stream truncation. Wrapping decompression in a budgeted reader with a 64x expansion ratio cap (GZIP_EXPANSION_RATIO_CAP = 64) blocked decompression bombs deterministically.
3. Adding advisory output-directory file locking to prevent races between concurrent writers (Round 2, S1). When two parser processes targeted the same --output-dir simultaneously, both exited 0 while interleaving data_N.parquet shards. Downstream DuckDB read_parquet('**/*.parquet') queries then failed with TProtocolException: Invalid data. Acquiring an exclusive advisory flock on the output lockfile rejected the racing writer immediately and preserved all 3,001 rows in the primary output.
4. Quarantining 0-byte truncated Parquet files left behind after SIGKILL (Round 2, S2). Killing a process mid-merge via SIGKILL or hitting a 100 KiB RLIMIT_FSIZE limit (S3) left a 0-byte data_999.parquet stub on disk. That 0-byte file permanently broke wildcard Parquet queries on subsequent runs. Writing to hidden temporary staging files before an atomic rename() and quarantining 0-byte stubs to .corrupt on startup restored queryability across all 72,145 rows.

Every fault scenario was converted into a permanent regression test before merge. The workspace suite started at 459 baseline tests and reached 470 tests after our 30-format declarative and 301-case Lean 4 FFI checks (report 08). Adding 15 tests in Round 1 and 11 differential tests in Round 2 brought the suite to 496 automated workspace tests (0 failures).

Porting our node telemetry collector from Go to Rust (15 rounds across 6 topologies)

Diagnostic engines only see what the node collector captures. To eliminate garbage-collection heap overhead and regex bottlenecks, we ported our collector from Go to safe Rust (forbid(unsafe_code), Jung et al., 2021). Compiled systems languages also avoid garbage-collection runtime overhead (Pereira et al., 2017).

Before evaluating cluster topologies, we benchmarked the Go and Rust implementations head to head on identical hardware.

In Rust, compile-time monomorphization and link-time dead-code elimination shrunk the full collector binary from 52.57 MB (25.05 MB with parquet-go) down to 6.55 MB. That 6.55 MB binary embeds Arrow, Parquet, Zstandard, and our 2.38 MiB (2.49 MB) template catalog. The standalone musl node agent is 4.20 MB (5.00 MB on linux/amd64).

For source-side redaction, a 38-literal SIMD AhoCorasick prefilter (Aho & Corasick, 1975; Wang et al., 2019) runs before 21 regex-automata lazy DFA rules (Cox, 2007). Throughput reached 580 to 850 MB/s on no-hit lines (61 to 93 MB/s in Go) and 150 to 177 MB/s on hint-heavy lines (11.1 MB/s in Go). Across 1,500 synthetic test secrets (15 secret classes) and 20,000 benign trap lines, Rust matched 100.0% of secrets with 0 false positives (9.1x to 15.9x speedup).

How the 15-round topology harness isolated the 24.5 s (14.4x) catalog stall

Unit benchmarks and single-container smoke tests did not expose how the collector behaved across diverse cluster deployments. We therefore ran 15 verification rounds (8 pre-registered benchmark gates, 2 live container, Kubernetes, and 3-region cloud fault matrices, and 5 multi-agent audit rounds). Those 15 rounds tested 6 container and Kubernetes topologies (8 live sub-cases).

The standout finding emerged in Topology 1A, a synthetic 2-TServer (tserver-0..1) container harness running below the RF=3 replication factor required to place new system tablets. Collecting SQL views with --enable ysql on that under-replicated harness stalled for 24.5 s per TServer instead of finishing in under 2.0 s.

Inspecting the tablet-server logs revealed why. Querying pg_stat_replication or pg_replication_slots invoked PgClientService.ListSlotEntries, which calls CreateCDCStateTableIfNotFound (CreateTable system.cdc_state). On standard RF=3 production clusters, system.cdc_state is created once in milliseconds. In Topology 1A, however, only 2 TServers were present (below RF=3), so tablet placement waited and polled IsCreateTableDone every 2.0 s until the 20.0 s statement_timeout fired (leaving a 120.0 s master RPC).

Switching the collector to query pg_stat_activity WHERE backend_type = 'walsender' reads WAL sender state directly from backend shared memory (0 catalog writes). That change guarantees sub-2-second collection (24.5 s down to 1.7 s, 14.4x faster, 14.41x exact) even on degraded or under-replicated clusters.

Across all 6 topologies, collector resource usage stayed well inside our pre-registered production safety budgets (H4 and H5). Per-node CPU overhead ranged from 0.013 to 0.087 effective cores (0.013 to 0.045 cores in getrusage self-telemetry and 0.032 to 0.087 cores in Linux cgroup cpu.stat accounting). The collector runs pinned to <= 1.0 core with nice 19, ionice -c 3, and a 50 MiB/s disk read cap. Resident memory peaked at 44.3 to 46.2 MiB (<= 256.0 MiB ceiling) with 0 bytes written to node disks.

We also measured live database query overhead across 3 alternating 26 s paired trials (1,600 point queries per trial on a 3-node RF=3 cluster). That cluster held 14 tablets and 42 replicas. Median p50 SQL latency moved from 0.865 ms to 0.891 ms (+3.0%). Median p99 SQL latency moved from 1.893 ms to 2.063 ms (+8.98%), passing our pre-registered <= 10.0% gate.

Verification gates and methodology checks

• Paired PRE/POST binary verification (Round 2, 11 scenarios). Every Round 2 fault scenario ran against both the pre-remediation binary and the post-remediation binary. That gate caught our initial 255-byte filename cap truncating .parquet extensions (S6) before merge.
• Downstream 28-step pipeline completeness gate (8 topology bundles). Every bundle produced across the 6 topologies (8 sub-cases) had to pass verify --check-redaction (0 leaks across 243 to 501 audited files). Each bundle also had to complete all 28 of 28 (100.0%) deterministic downstream analyzer steps.
• Multi-byte UTF-8 boundary and 43-vector redaction torture gate (Round 5 audit). Injecting multi-byte UTF-8 characters (µs and typographic quotes) at exact 320-byte and 512-byte truncation boundaries caught panics prior to is_char_boundary() checks. A 43-vector torture suite also verified JSON-escaped password and file-pointer (aws_access_key_id) redaction.
• Independent external ISO/IEC 42001 (2023) audit readiness. Under our governance controls (report 09, report 11), pre-registered fault-injection ledgers are retained for independent external ISO/IEC 42001 audits by an accredited certification body.

Testing antipatterns rejected by our gates

1. Relying on healthy-node golden benchmarks to certify parsers and collectors (missed 18 input-boundary and SIGKILL edge cases in our log engine and 7 topology bugs).
2. Allowing concurrent processes to write Parquet shards without an advisory lock (caused shard interleaving and TProtocolException read failures until advisory output locking was added).
3. Writing Parquet output directly to final filenames instead of atomic temporary staging files (left 0-byte data_N.parquet files after SIGKILL that permanently broke read_parquet('**/*.parquet') until startup .corrupt quarantine was added).
4. Querying pg_stat_replication instead of pg_stat_activity from a diagnostic collector. On healthy RF=3 production clusters system.cdc_state is created once in milliseconds, but on our 2-TServer (< RF=3) harness tablet placement stalled 24.5 s per TServer. Querying pg_stat_activity WHERE backend_type = 'walsender' finishes in 1.7 s (14.4x faster) even on degraded or under-replicated clusters.
5. Scraping /prometheus-metrics without reset_histograms=false or using kubectl get -A under namespace-scoped RBAC (both rejected by our collector safety gates).

Pressure tests

Every finding was checked against the six standing fronts in Methods.

What this does not show

First, fault injection and chaos harnesses cannot prove the absence of bugs on unexplored code paths. We verify glog parser termination (10|s| + 50 steps) and burst conservation (sum(repetition_count) = N) in Lean 4 (1,368 lines). Those proofs link into cargo test across 301 FFI checks (report 08).

Second, two Round 2 timing scenarios required deterministic proxies for repeatable A/B gating. In S2 (SIGKILL mid-merge), an external timer-based kill can miss the sub-millisecond rename window. We therefore paired live kill tests with a deterministic planted 0-byte file test (72,145 rows recovered). In S3 (ENOSPC disk full), mounting a full RAM disk requires root privileges on macOS. We therefore enforced an unprivileged 100 KiB RLIMIT_FSIZE process limit alongside unit drop-cleanup tests.

Third, the +8.98% median p99 database latency delta (1.893 ms without collector vs 2.063 ms with collector) was measured across 3 alternating 26 s runs. Those runs executed on 3 containers sharing a 6-vCPU host (~62 QPS, where Trial 1 reached 2.177 ms). On dedicated multi-core production VMs, the collector's single-thread pin (nice 19, ionice -c 3, 0.013 to 0.087 cores per node) contends with a smaller fraction of host CPU capacity.

Reproduce

Our Rust log-to-Parquet engine is currently being prepared for open-source release under the Apache 2.0 license. Every metric in this report and its three figures is stored in numbers.json and verified against MANIFEST.md.

References

• Aho, A. V., & Corasick, M. J. (1975). Efficient String Matching: An Aid to Bibliographic Search. Communications of the ACM. DOI.
• Alvaro, P., Rosen, J., & Hellerstein, J. M. (2015). Lineage-Driven Fault Injection. Proceedings of ACM SIGMOD 2015. DOI.
• Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016). Chaos Engineering. IEEE Software. DOI.
• Claessen, K., & Hughes, J. (2000). QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. Proceedings of ACM SIGPLAN ICFP 2000. DOI.
• Cox, R. (2007). Regular Expression Matching Can Be Simple And Fast. Technical Note.
• ISO/IEC. (2023). ISO/IEC 42001 (2023): Information Technology - Artificial Intelligence - Management System. International Organization for Standardization.
• Jung, R., Jourdan, J.-H., Krebbers, R., & Dreyer, D. (2021). Safe Systems Programming in Rust. Communications of the ACM. DOI.
• Kikuta, D., Ikeuchi, H., & Tajiri, K. (2025). LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost. Proceedings of IEEE/ACM ASE 2025. DOI. arXiv.
• Kingsbury, K. (2019). Jepsen: YugaByte DB 1.3.1. Jepsen Technical Report.
• Meng, R., Pîrlea, G., Roychoudhury, A., & Sergey, I. (2023). Greybox Fuzzing of Distributed Systems. Proceedings of ACM CCS 2023. DOI.
• Pan, J., Wu, H., Leesatapornwongsa, T., Nath, S., & Huang, P. (2024). Efficient Reproduction of Fault-Induced Failures in Distributed Systems with Feedback-Driven Fault Injection. Proceedings of ACM SOSP 2024. DOI.
• Pereira, R., Couto, M., Ribeiro, F., Rua, R., Cunha, J., Fernandes, J. P., & Saraiva, J. (2017). Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate? Proceedings of ACM SIGPLAN SLE 2017. DOI.
• Vikram, V., Lemieux, C., Sunshine, J., & Padhye, R. (2023). Can Large Language Models Write Good Property-Based Tests? arXiv preprint. arXiv.
• Wang, X., Hong, Y., Chang, H., Park, K., Langdale, G., Hu, J., & Zhu, H. (2019). Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs. Proceedings of USENIX NSDI 2019.
• Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Zhao, X., Zhang, Y., Jain, P. U., & Stumm, M. (2014). Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. Proceedings of USENIX OSDI 2014.