Pairing local models with sanitizers to audit 68 open-source repositories

The claim

How do you use language models for vulnerability discovery without flooding engineers and open-source maintainers with false positives?

Standalone language models produce false-positive rates above 50% to 80% on static security review (Ullah et al., 2024; Li et al., 2025). When a model inspects a single function slice, caller bounds checks, compiler flags, and sentinel conventions sit outside its context window (Guo et al., 2025). Submitting unverified static model output to open-source maintainers or internal product teams wastes engineering time and erodes trust (Amusuo et al., 2025).

We solved this problem by pairing our co-resident local GPU stack (report 13) with a mandatory execution gate (E1). Before a reasoning model inspects any code, semantic deduplication (cosine >= 0.85) and a local 4B triage gate (report 07) filter candidate hazard slices. A local 38B code tracer then audits surviving clusters across caller chains and proposes a concrete trigger input.

No static finding enters our disclosure or remediation queue on model reasoning alone. Every candidate must reproduce under AddressSanitizer (Serebryany et al., 2012), UndefinedBehaviorSanitizer, go test, or a local loopback container socket harness. Both stages combine under a conjunctive law.

$$\text{Disclosable}(x) = \text{StaticTrace}(x) \land \text{HarnessRepro}(x).$$

Across 68 repositories (28 C/C++ campaigns and 40 deep audits), this gate held.

• 81.3% of high-confidence static model claims were false positives. In our 28 automated C/C++ campaigns, the slicer extracted 338 static hazard slices and deduplicated them into 245 clusters (-27.5%). The local 4B gate (tau = 0.28) escalated 89 clusters (36.3%) to the 38B tracer, which produced 32 high-confidence static vulnerability claims. Compiled execution harnesses refuted 26 of those 32 claims (81.3%, or 81.25% exact), leaving 6 confirmed C/C++ vulnerabilities (18.8%, or 18.75% exact).
• Only reproducible E1 proofs entered coordinated disclosure. Across all 68 repositories (28 C/C++ campaigns plus 40 multi-language deep audits), our harnesses verified 85 E1 execution proofs across 32 projects. Another 137 code-level source traces (E2) without live execution proofs were held back from standalone disclosure. Across those 32 projects, we prepared 31 private coordinated vulnerability disclosure (CVD) packages (with 1 low-severity test-only finding excluded from advisory filing). All 31 packages are held under a 90-day embargo (ISO/IEC 29147).
• 11 heavily deployed open-source codebases (10 applications and frameworks plus pyyaml's LibYAML C core) passed with 0 disclosable vulnerabilities. Local adversarial probe batteries cleared vercel/next.js, axios/axios, twisted/twisted, authelia/authelia, envoy, mlflow, fastmcp, mcp-python-sdk, xerj-org/xerj, and github/github-mcp-server. These runs included 201,904 fuzz inputs on envoy, 535 route checks on mlflow, and 120 concurrent two-tenant races on github/github-mcp-server. In addition, 3,250,000 (3.25 million) AddressSanitizer runs on pyyaml's LibYAML C loader produced 0 memory-corruption errors.
• Enterprise pre-merge supply-chain checks. Engineering teams can run this sanitizer-gated workflow as a pre-merge control on vendored C dependencies, verifying NUL-termination guards and CMake build pins before release.

The 68-repository open-source auditing pipeline and local-LLM triage stack in this study are independent research built with Carlos Juzarte Rolo. Carlos is a longtime friend, fellow Instituto Superior Técnico alumnus, and local-LLM hardware and systems specialist.

Four engineering rules for model-assisted security auditing

1. Grade every finding by execution evidence (E1, E2, E3) and gate disclosure on E1. Class E3 covers unverified static slices. Class E2 covers source-traced defects verified at repository HEAD without a live exploit. Class E1 requires a reproducible sanitizer crash, unit test failure, or socket proof. Only E1 findings trigger external advisories.
2. Deduplicate and triage locally before invoking deep reasoning. Embed normalized hazard slices in 76.8 ms (1,024 dimensions) and collapse near-duplicates at cosine 0.85 (Abbas et al., 2023). Score each cluster exemplar on a local 4B single-pass classifier in 212 to 226 ms (0 output tokens) before waking the 38B model.
3. Test release-build macro semantics alongside debug assertions (debug vs release builds). Many C and C++ codebases rely on assert or DCHECK macros that vanish under -DNDEBUG in release builds. Conversely, other projects define custom macros that remain active in release binaries. Compiling both configurations under AddressSanitizer eliminates macro guessing.
4. Verify harness portability to separate virtual-machine traps from memory bugs. Compiling C/C++ targets with -march=native inside a container hypervisor can trigger CPU SIGILL instruction traps. Every E1 crash must reproduce under a portable -O1 -fsanitize=address,undefined build before being recorded as a memory defect.

The three-stage local triage and sanitizer verification pipeline

Running multi-repository security sweeps over metered cloud APIs is expensive because deterministic gates reject over 80% of candidates (report 13). Winning DARPA AIxCC systems treated models as hypothesis generators gated by sanitizers (Kim et al., 2025; Zhang et al., 2026).

We run three co-resident models across 4 local GPUs (128 GB VRAM). Those models feed containerized sanitizer build runners (Li et al., 2025; Nie et al., 2025).

Across the 28 automated C/C++ campaigns, SemDeDup collapsed 338 raw hazard slices into 245 distinct clusters, removing 93 redundant slices (27.5%). Next, 89 of the 245 clusters (36.3%) cleared the local 4B triage gate (tau = 0.28) to enter the 38B code tracer. The 38B tracer then escalated 32 high-confidence CONFIRMED static vulnerability claims.

When we compiled those 32 targets with AddressSanitizer (Serebryany et al., 2012) and UndefinedBehaviorSanitizer, 26 of the 32 triggers (81.3%) failed to reproduce. Only 6 claims (18.8%) survived as genuine C/C++ vulnerabilities. We applied the same E1 execution rule across all 68 repositories (28 automated C/C++ campaigns and 40 deep audits). That full program produced 85 E1 executed proofs across 32 projects and 31 embargoed CVD patch packages.

Why 81.3% of high-confidence static claims failed execution testing

Inspecting the 26 refuted static claims shows why static language-model auditing produces high false-positive rates (Ullah et al., 2024; Zhang et al., 2024). In every refuted case, the 38B model flagged a plausible local hazard inside the target function. However, it missed an invariant enforced elsewhere in the repository or build system.

Four named examples from the 26 refutations illustrate these failure modes.

• Upstream dimension clamping in pgvector (4 claims refuted). The static tracer flagged calloc and palloc integer-overflow hazards in sparsevec and vector_in at high confidence. Tracing the upstream entry points and running AddressSanitizer showed that sparsevec_dim() already clamps dimensions to SPARSEVEC_MAX_DIM = 16383. That bound limits max_blocks to 128 KiB and checks written + nvalues > ndims before allocation, making 32-bit multiplication wrap impossible.
• Window and table clamping in libarchive (3 claims refuted). In libarchive's RAR5 reader, the reasoning model claimed a critical out-of-bounds heap write on window_size and an INT_MIN table allocation overflow in make_table(). A callee-by-callee trace and 2,044-run sanitizer sweep proved that every caller caps R < 256 KiB (bytes_to_read <= 768 KiB inside a 1 MiB window_buf_size). Similarly, bits_read_int(16) and the existing tablesize <= 9 guard clamp tablesize into [10, 5040].
• Empty-string array sentinels in haproxy (1 claim refuted at 0.92 confidence). On haproxy, the 38B model reported a CWE-476 NULL pointer dereference (0.92 confidence) in cfg_parse_users_group when strdup(args[cur_arg + 1]) runs with users as the final configuration token. Compiling haproxy natively and passing group g1 users returned exit code 0 with 0 crashes. Throughout haproxy, configuration argument arrays terminate with empty-string sentinels ("") rather than NULL, so strdup safely copies an empty string.
• Input routing and macro semantics in clickhouse (4 claims refuted). In clickhouse, static claims predicted an out-of-bounds write in MergeTreeRewriteSource and release-build assertion crashes. Running 230 curated reproducer seeds under a sanitizer build produced 0 crashes because FixedString inputs route to FixedStringReadBinaryDataSource rather than the hypothesized sink.
• Virtual machine -march=native SIGILL traps (2 claims refuted). In 2 early containerized fuzzing runs, our harness reported crash exit codes that looked like memory corruption. Inspecting the signal frames revealed SIGILL (illegal instruction) caused by -march=native emitting host CPU instructions that the virtual machine hypervisor masked out. Enforcing portable -O1 compilation eliminated both false confirmations.

Eleven heavily deployed codebases cleared by adversarial probe batteries

A reliable security auditing pipeline should also measure and publish negative results. When an open-source project hardens its trust boundaries, verifying that adversarial probes fail closed gives engineering leaders concrete supply-chain assurance.

Across our 68-repository program, 11 heavily deployed open-source codebases (10 applications and frameworks plus pyyaml LibYAML C core) passed local E1 batteries with 0 disclosable vulnerabilities. In particular, pyyaml's underlying LibYAML C parser passed 3,250,000 (3.25 million) AddressSanitizer and UndefinedBehaviorSanitizer executions with 0 memory-safety crashes.

One self-refutation on vercel/next.js highlights why local probe execution is mandatory even on web frameworks. During source review, our auditing agent drafted a candidate Server-Side Request Forgery (SSRF) claim using a scoped IPv6 loopback zone literal (%eth0). When the agent ran its own verbatim probe against a local container instance of the vercel/next.js image-optimization pipeline, the attack failed closed. Both the WHATWG URL parser and ipaddr.js unwrapped the IPv4-mapped address and rejected the private range before any socket dialed. Without the E1 execution rule, that false positive would have produced a spurious report.

Applying sanitizer-gated auditing to enterprise supply-chain dependencies

Enterprise engineering teams can apply this same sanitizer verification loop as a proactive pre-merge control over third-party and vendored C dependencies.

When a security team audits an open-source dependency class externally, it can run three sibling checks in continuous integration before shipping any release artifact.

1. Verifying NUL-termination guards on authentication packets. In C connection poolers, reading a client PasswordMessage via malloc(len) and memcpy(len) without a trailing NUL check risks heap over-reads. Downstream strlen or LDAP bind calls can then read past the buffer. Pre-merge sanitizer gates verify explicit NUL-termination bounds checks and safe buffer handling before merge (0 memory over-reads).
2. Auditing CMake build pins and compiler flags. Static review of source files can miss how CMakeLists.txt or container build scripts compile vendored C targets. Automated build checks verify that release flags preserve mandatory bounds guards and link against patched dependency revisions before any artifact ships.
3. Pairing automated internal gates with independent external certification audits. Under ISO/IEC 42001 (2023) governance (report 09, report 11), internal automated verification is necessary but never sufficient on its own. Enterprise teams should pair automated pre-merge sanitizer gates with independent external ISO/IEC 42001 certification audits conducted by an accredited third-party certification body.

Coordinated vulnerability disclosure and embargo governance

Our harnesses produced 85 verified E1 execution proofs across 32 open-source projects, for which we prepared 31 private CVD packages (with 1 low-severity test-only finding excluded from advisory filing). For every confirmed finding, we follow Coordinated Vulnerability Disclosure (CVD) rules under ISO/IEC 29147 and ISO/IEC 30111.

• Zero public naming of unpatched targets. None of the 31 open-source repositories with active private disclosure reports are named in this study. All file paths, proof-of-concept inputs, and reproduction scripts remain strictly confidential until upstream maintainers publish fixes or the standard 90-day disclosure window closes.
• Minimal unified-diff patches included with every report. Maintainers of open-source projects are often volunteer-constrained. Each of our 31 private disclosure packages includes a minimal unified diff (1 to 15 lines of code). We compiled and verified every patch against its E1 harness (0 post-patch sanitizer errors and 100% test pass rate).
• Channel verification before submission. Before sending any report, our workflow checks each repository's SECURITY.md policy. It routes findings through GitHub Private Vulnerability Reporting (PVR), security mailboxes, or maintainer-specified fix pull requests.

Security auditing antipatterns rejected by our gates

1. Submitting unverified static LLM output (E3) to maintainers (rejected after execution harnesses refuted 26 of 32 [81.3%] high-confidence static C/C++ claims).
2. Treating model verbalized confidence (0.92) as proof of exploitability (rejected after a 0.92-confidence CWE-476 claim on haproxy returned rc = 0 due to empty-string array sentinels).
3. Trusting container crash exit codes without signal and flag inspection (rejected after 2 apparent crashes turned out to be -march=native SIGILL traps inside the container hypervisor).
4. Filing duplicate advisories for documented threat-model exclusions or known issues (prevented by checking repository SECURITY.md, THREAT_MODEL.md, and historical GHSA lineages before drafting any report).
5. Publishing vulnerability details before upstream patches ship (blocked by our 90-day CVD embargo across all 31 private disclosure packages).

Pressure tests

Every finding was checked against the six standing fronts in Methods.

What this does not show

First, passing an adversarial probe battery (0 disclosable vulnerabilities across the 10 cleared targets) does not prove the absence of all bugs. Dynamic sanitizers and targeted protocol probes verify only the execution paths, build configurations, and threat models exercised by the harness.

Second, holding 137 code-level E2 source traces out of standalone CVD queues does not mean those code paths are risk-free. Many E2 findings represent genuine defense-in-depth gaps whose reachability depends on non-default deployment topologies or unexercised protocol states.

Third, local models do not replace human security engineers on threat-model judgment. Deciding whether a behavior violates a project's documented security boundary or represents intended operator functionality requires reading SECURITY.md and verifying live configuration defaults.

Reproduce

All aggregate metrics in this report and its three figures are stored in numbers.json, with SHA-256 integrity digests recorded in MANIFEST.md. In accordance with ISO/IEC 29147 coordinated vulnerability disclosure rules, target identities and reproduction harnesses for the 31 embargoed advisories are withheld until upstream remediation completes.

References

• Abbas, A., Tirumala, K., Simig, D., Ganguli, S., & Morcos, A. S. (2023). SemDeDup: Data-Efficient Learning at Web-Scale Through Semantic Deduplication. arXiv preprint. arXiv.
• Amusuo, P. C., Liu, D., Mendez, R. A. C., Metzman, J., Chang, O., & Davis, J. C. (2025). FalseCrashReducer: Mitigating False Positive Crashes in OSS-Fuzz-Gen Using Agentic AI. arXiv preprint. arXiv.
• Guo, J., Wang, C., Xu, X., Su, Z., & Zhang, X. (2025). RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing. Proceedings of ICML 2025. arXiv.
• ISO/IEC. (2018). ISO/IEC 29147: Information Technology - Security Techniques - Vulnerability Disclosure. International Organization for Standardization.
• ISO/IEC. (2019). ISO/IEC 30111: Information Technology - Security Techniques - Vulnerability Handling Processes. International Organization for Standardization.
• ISO/IEC. (2023). ISO/IEC 42001 (2023): Information Technology - Artificial Intelligence - Management System. International Organization for Standardization.
• Kim, T., Han, H., Park, S., Jeong, D. R., Kim, D., Kim, D., et al. (2025). ATLANTIS: AI-Driven Threat Localization, Analysis, and Triage Intelligence System. arXiv preprint. arXiv.
• Li, Z., Dutta, S., & Naik, M. (2025). IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. Proceedings of ICLR 2025. arXiv.
• Li, Z.-Z., Zhang, D., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y., et al. (2025). From System 1 to System 2: A Survey of Reasoning Large Language Models. arXiv preprint. arXiv.
• Nie, Y., Li, H., Guo, C., Jiang, R., Wang, Z., Li, B., et al. (2025). VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection. arXiv preprint. arXiv.
• Serebryany, K., Bruening, D., Potapenko, A., & Vyukov, D. (2012). AddressSanitizer: A Fast Address Sanity Checker. Proceedings of USENIX ATC 2012.
• Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., & Stringhini, G. (2024). LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?). Proceedings of IEEE Symposium on Security and Privacy (S&P 2024). arXiv.
• Zhang, C., Park, Y., Fleischer, F., Fu, Y.-F., Kim, J., Kim, D., et al. (2026). SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned. Proceedings of USENIX Security 2026. arXiv.
• Zhang, Y., Wang, J., Berzin, D., Mirchev, M., Liu, D., Arya, A., et al. (2024). Fixing Security Vulnerabilities with AI in OSS-Fuzz. arXiv preprint. arXiv.