Research repository documenting LLM generalization ceilings in code security vulnerability detection, with cross-evaluation across synthetic and real-world datasets (Juliet, OWASP, CVEfixes, VUDENC) for SQL injection, path traversal, and access control flaws.
Organization: Bytepro AI
Status: Active research — private repository
Target: arXiv Q3 2026 + ARC Prize Paper Track Nov 2026
Empirically document that the single-prompt ceiling observed in mathematical reasoning (SAIR, arXiv:2504.18897) replicates in code security vulnerability detection, confirming the router hypothesis is domain-independent.
gpt-oss-20b + structured cheatsheet matches frontier models on synthetic distributions (Juliet, OWASP Benchmark) but collapses under distribution shift to real-world data (CVEfixes, VUDENC), documenting a generalization ceiling in LLMs for code security tasks.
| Category | CWE | Data Source |
|---|---|---|
| SQL Injection | CWE-89 | VUDENC + Juliet + synthetic |
| Path Traversal | CWE-22 | VUDENC + Juliet + synthetic |
| Hardcoded Credentials | CWE-798 | Juliet + synthetic |
| N+1 Query Pattern | — | Synthetic only |
| Broken Access Control | CWE-284 | Synthetic only |
Cross-evaluation matrix:
