DeepSeek 4.1 Flash mogs OpenAI on our PR Security Review benchmark!
Summary
We ran our PR security-review benchmark across 16 repos, each with a planted vulnerability over the latest models. DeepSeek 4.1 Flash is our pick for day-to-day review: 64% of the planted bugs found, about 6 minutes a PR. We're moving our workloads over to it. Although OpenAI GPT6 Astra finds 5% more vulnerabilities, it is pricey and tries to break out of its sandbox.
NOTE: this PR Benchmark is specifically scanning differential code-commits (PRs) and looking for recently introduced vulnerabilities. This is not an exhaustive "full scan" across a code base.
Highlights
- DeepSeek v4.1 Flash delivers the best bang-for-buck. It performs as well as the US models for finding vulnerabilities (64%). It is inexpensive (via Vercel) at a fraction of the price: $0.42 per PR. However, it is the noisiest and generates a lot of findings that might distract developers and create unwanted goose-chases.
- GPT-6 Astra performs the best overall finding 68% (Recall) of all vulns. However, it is costly at $8.80/PR. Solid. Not a clean sweep.
- Gemini 3.8 Flash is a lemon on this task. 35% recall, tied for last on findings. Despite being a public-bench specialist it fails our private bench.
- Anthropic is still out. Anthropic tanked last 2 times we ran this report. This time it is flanked by OpenAI and Deepseek. DeepSeek bests all Anthropic models on all dimensions.
How we choose:
- There is no "best" model. You need to choose between: highest recall, a review a team would actually wait for in PRs (surely that's <10m?), and cost.
- Cost is a key factor. A team that produces 1,000 PRs per month can easily consume $150k USD in tokens over a year. So we need to be careful with which model we choose.
- Time to review must be <10 minutes. Developers get annoyed by a PR security review that blocks their workflow. Our internal goal is 3-4 minutes and we believe most teams will not accept reviews above 7 minutes per review.
This benchmark is a significant upgrade from the last time we ran it, and includes all recent model drops: Gemini 3.8 Flash, GPT6 Astra, Deepseek v4.1 Flash and Fable 5.1.
All models are set to "High" thinking mode. Typically above High, the models become too slow for a PR review.
Results
| Model | Recall | Cost / PR | Time / PR | Findings / TP | Output tokens |
|---|---|---|---|---|---|
| GPT-6 Astra | 68% | $8.80 | 5m 13s | 4.5 | 1,174,942 |
| DeepSeek V4.1 Flash (via Vercel) | 64% | $0.42 | 6m 32s | 8.8 | 28,771,452 |
| Claude Opus 5 | 64% | $11.08 | 7m 10s | 5.0 | 9,440,542 |
| Fable 5.1 | 63% | $7.31 | 3m 44s | 3.7 | 3,433,907 |
| GPT-5.6 Sol | 61% | $5.82 | 3m 33s | 5.2 | 2,881,032 |
| GLM 5.3 (via OpenRouter) | 58% | $2.09 | 6m 43s | 4.9 | 8,326,740 |
| Kimi K3 (via OpenRouter) | 56% | $4.01 | 8m 28s | 5.1 | 5,658,253 |
| GLM 5.3 Flash (via OpenRouter) | 49% | $0.11 | 8m 6s | 5.0 | 7,978,225 |
| Grok 4.6 (via OpenRouter) | 48% | $4.66 | 4m 42s | 2.4 | 8,470,460 |
| GPT-5.6 Luna | 39% | $0.28 | 2m 52s | 4.1 | 3,386,337 |
| Claude Sonnet 5 | 35% | $3.18 | 5m 23s | 4.7 | 7,115,275 |
| Gemini 3.8 Flash | 35% | $3.45 | 4m 32s | 1.8 | 18,892,349 |
Definitions:
- TP: True Positive
- Recall: same as True Positive. The number of planted vulnerabilities found.
- Time / PR: The time it takes to complete the security scan of the PR.
- Findings: the overall number of vulnerabilities the model reported
- Findings / TP: A measure of noisiness. Higher is worse.
Findings at any cost?
Mean review time per PR vs recall. Blue points define the time/quality frontier.
If you strictly want to find the most vulnerabilities, and you do not care how long the review takes or how much it costs, OpenAI GPT6 Astra is the pick.
Developer time is costly: optimizing for fewer false positives
Findings per true positive vs recall. Lower on this axis means less junk for developers to triage. Blue points define the noise/quality frontier.
If developer productivity is the goal then you want fewer findings overall, even if that means potentially missing some real vulns. Some models just have to say something: ask "is there is a vulnerability?", and they will hunt until they can say yes, whether or not the bug is real.
OpenAI GPT6 Astra is the cleanest of the models that catches most of the planted bugs and only has 4.5 findings per true positive.
DeepSeek V4.1 Flash is the other extreme: it finds the most true positives, and it also emits the most extra findings (5.4 per true positive). Gemini 3.8 Flash and Grok 4.6 look quiet on this chart, but they miss more than half of the planted bugs.
Methodology
This benchmark measures one workload: finding a planted vulnerability in a pull request. It is not a full-repository hunt. Rankings here should not be read as a general "best security model" list.
We scored 16 repos, with a mix of vulnerability types like IDOR, SQL injection, and missing authentication.
The repos span TypeScript, JavaScript, Java, Ruby, Rust and Python.
How we run the models
Every model reviews the same 16 pull requests through the same Dam Secure Vulnerability Scanner configuration. The agent sees the PR diff and can inspect surrounding code (read files, search the changed tree). Reasoning effort is pinned to high for every model; what varies is only the underlying model.
Each configuration is run five times. That is 80 scored reviews (16 repos × 5). We score whether the planted bug was found, missed, or crowded out by extra findings.
Securing Sandboxes
AI agents can pursue goals beyond their assigned tasks. The METR investigation into the Hugging Face incident documented agents bypassing intended isolation and coordinating attempts to tamper with a benchmark’s scoring system.
At Dam Secure, we’ve invested heavily in isolating agents, including those used for this benchmark. That means specialised execution environments, hardening tools such as Git, and building custom agent tools that restrict what our agents can do.
Before giving an LLM access to code to search for vulnerabilities, build strong sandboxes.
Keeping the benchmark clean
We reverse-replay real vulnerabilities into open source codebases: take a known bug (typically an already-fixed CVE), restore the vulnerable code, and present it as an ordinary pull request. The PR reads like day-to-day work, and we try hard not to give away that the bug was planted.
We will not publish the source repositories. That would invite training-set contamination and give away how the fixtures were built. The pull requests themselves stay unpublished for the same reason.
The hard part of a security benchmark is contamination through two means:
- Regular model training — Public vulnerability write-ups get absorbed into training data, so a model can recall the fix instead of finding it. Juice Shop is the obvious example: frontier models call the bugs on sight.
- Bench-maxxing — Training on the benchmark itself. That destroys the ranking.
This run is reverse-replay only, not the private synthetic apps from earlier versions. Ground truth is the reverted CVE; the surrounding code is real. Please email us if you want to review the repos.