91% Passed, 25% Safe: Why Your Coding Agent Ships Code That Works and Gets You Hacked

A new benchmark took 150 real vulnerabilities, handed the broken code to six frontier agents, and graded the output the way an attacker would. The best agent got 91.1% of the tasks working. Only 25.1% of them were also safe to ship.

The Java config importer that passed every test

One task in the benchmark asks an agent to rewrite the missing file at the heart of a Java configuration importer. The spec describes the feature and nothing else. Kimi-K3 nails it: all 14 hidden functional tests pass. Then the hidden security suite replays the original attack against the importer's YAML parser — and all 4 security tests fail. The bug is a single constructor choice. Kimi used SnakeYAML's default Yaml() constructor, which honors every !!class tag in untrusted YAML — the classic deserialization hole. Claude Fable 5 and GPT-5.6-Sol parsed the same input with SafeConstructor and passed both suites. Qwen3.8-Max and GLM-5.3 made Kimi's mistake.

That one task is the whole benchmark in miniature. This is the Last Secure Code Benchmark (LSCBench), submitted to arXiv on October 10, 2026 by researchers at the National University of Singapore and Alibaba Group (including the Qwen team). It doesn't ask whether agents can write code. It asks whether the code survives contact with an adversary.

ELI5: "works" and "safe" are two different users

Think of it this way. Every feature you ask for has two audiences. The friendly user wants the feature to work: upload a config, get it imported. Functional tests play the friendly user. The hostile user wants the feature to break in a useful direction: upload a config that installs a class nobody intended. Security tests play the hostile user.

Same code, two examiners. Coding agents are trained and prompted for the friendly one — because that is who the spec describes. LSCBench is the first large benchmark to systematically ask: what happens when the second examiner shows up too?

How LSCBench works: graded like an attacker

The design is deliberately strict, and it has to be — otherwise agents game it:

• 450 tasks built from 150 real vulnerabilities: 115 scenarios from public CVEs, 35 Java applications written by security experts. Seven languages, 24 weakness categories (CWEs).
• Three task sizes: fill in a masked function, write a removed file, or rebuild an emptied source tree from a spec.
• The prompt never mentions security. In the authors' words: every task asks only for functionality, through a specification that never mentions security. That's how most people actually prompt a coding agent — you ask for the import feature, not for a deserialization-safe import feature.
• Dual hidden suites. Each task is scored only if two hidden test suites both pass: functional tests, and security tests that replay the original attack. No partial credit, no LLM judge. One attempt per task (pass@1).

Six frontier agents ran at their highest reasoning effort: five inside Claude Code v2.1.143 (Claude Fable 5, Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro, Kimi-K3) and GPT-5.6-Sol inside Codex v0.154.0.

The leaderboard: the 91/25 gap

Read the last row twice. Kimi-K3 has the highest functional pass rate and the lowest secure rate. A green test suite told you nothing about which agent was safer.

The aggregate numbers are worse than any single row:

• Across all 2,700 runs, 66% ended with code that works and is still vulnerable.
• Of the 150 scenarios, 57 were never secured by any model — not once, across six frontier agents.

Where the code breaks: languages and weakness families

The project page breaks the results down two ways, and both are instructive.

By language (share of all runs that ended both working and secure, all six models combined): C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10%.

By weakness family (share of working code that was also secure — a stricter base): memory safety comes out best at 44%, injection at 28%, then a cliff — SSRF at 3% and information exposure at 1%.

That cliff is the story. Memory-safety fixes are mostly local — bounds-check this, free that. SSRF and information exposure require the agent to model the system: where does this URL reach? Who else reads this config dump? Agents write the feature and leave the door open.

The paper's case studies show what that looks like in practice:

• Command injection, Java: four of six models passed mount arguments to ProcessBuilder as a list. Qwen3.8-Max and Kimi-K3 built a string for sh -c — the shell interprets whatever the user typed as commands.
• SSRF, Go (LocalAI task): GPT-5.6-Sol's GetURI read any path it was handed, file:// URLs included. The reference fix confines reads to the models directory. No model did that.
• Information exposure, Go (syft task): GLM-5.3 followed the tag convention of neighboring structs, so an attestation password showed up in config dumps and SBOM output. All six models made the same choice.

None of these are exotic. A reviewer who goes looking for them catches all three. The agents weren't looking.

One sentence helped. It didn't fix it.

The authors reran four of the six models with one sentence appended to every task — an implementation meeting the functional requirements "while also being secure":

• GPT-5.6-Sol: 24.7% → 46.2%
• GLM-5.3: 21.3% → 38.2%
• Qwen3.8-Max: 22.0% → 37.6%
• DeepSeek-V4-Pro: 18.0% → 19.1%

Functional pass rates moved a few points either way. The paper's verdict: most tasks stay insecure even with the reminder — and SSRF and information exposure stay near the bottom. (Claude Fable 5 and Kimi-K3 weren't in the rerun.)

Takeaways

1. Green tests ≠ shippable. Internalize the 91/25 gap: a passing suite measures the friendly user. If your agent's output goes anywhere hostile users can reach, you need the second suite.
2. The failure mode is omission, not error. The safe API was always available — SafeConstructor, argv lists, path confinement. The agent just never reached for it. Security, for agents, is a retrieval problem: the knowledge is in the weights; the trigger is missing.
3. Write the attack test, not just the feature test. LSCBench's whole trick is replaying the original exploit. For your own agent-built code: if it takes a model name, check that it refuses a URL and a ../ path climb. The companion notebook shows the three patterns.
4. Add the reminder anyway. One sentence nearly doubled secure output for three of four models (+16 to +22 points). A line in your AGENTS.md or CLAUDE.md is the nearest everyday equivalent — cheap, no downside in the data, just don't mistake it for a fix.
5. Read the fine print. Everything is pass@1, one attempt. Qwen3.8-Max was also the construction agent that wrote the tests and specs — the authors argue its mid-field finish shows it gained no advantage. And it's an arXiv preprint: treat it as one data point until someone replicates it.

Sources: LSCBench paper (arXiv, submitted October 10, 2026, NUS + Alibaba); LSCBench project page (last-secure-code-benchmark.github.io). Numbers quoted are the authors' own, from the public leaderboard. A dev.to writeup by Gabriel Anhaia (October 14, 2026) first surfaced several of these details.