Simon Harloff

Summary​

We ran our PR security-review benchmark across 16 repos, each with a planted vulnerability over the latest models. DeepSeek 4.1 Flash is our pick for day-to-day review: 64% of the planted bugs found, about 6 minutes a PR. We're moving our workloads over to it. Although OpenAI GPT6 Astra finds 5% more vulnerabilities, it is pricey and tries to break out of its sandbox.

NOTE: this PR Benchmark is specifically scanning differential code-commits (PRs) and looking for recently introduced vulnerabilities. This is not an exhaustive "full scan" across a code base.

Summary​

We re-ran the same PR security-review benchmark (not a full-code scan) over a new wave of models: Kimi K3, Gemini 3.6, Opus 5, GLM5.2 and more OpenAI models. Overall, GPT-5.6 Sol is still on top. Kimi K3 is the strongest open-weights performer, GLM 5.2 is cheap and precise but lags hard on recall, and Opus 5 performed well but was even more expensive than Fable. More below!

We wanted to take the guesswork out of which model to use for PR security reviews. To our surprise no Anthropic model reaches the frontier at all and the latest GPT5.6, released only 3 days ago, is the king. Although Fable is known to perform well on a full-code scan, PR reviews are a common workload for our users.

In the rush to integrate generative AI into the software development lifecycle, engineering teams frequently confuse two distinct operational concepts: guardrails and guidelines. To put it simply, a guideline advises: it's soft natural-language context, like an AGENTS.md or Skills file, that the model may or may not act on. A guardrail gates: it's a firm, automated constraint that forces an explicit, binary pass/fail and blocks on failure. Relying on soft guidelines for critical validation is why automated code reviews are risky. They're non-deterministic and quietly ignored.

We have some exciting news to share! Dam Secure has officially achieved SOC 2 Type 2.

You can request our SOC 2 Type 2 report through our Trust Center.