PatchLoop: Vulnerability Detection and Patching Harness

Verify with the machinery that raised the alert. Propose a patch, apply it, verify it with your own detectors, feed the verbatim failure back to the model, and repair on top of the patched state. Bounded attempts, typed errors, fail-fast verification.

Disclaimer. These results are from the original PatchEval benchmark. On July 24, 2026, after these runs, ByteDance released PatchEval-Verified, with revised Docker environments and a new leaderboard. The numbers on this page predate that release and are not comparable to PatchEval-Verified. We plan to re-run against the updated version and will follow up.

PatchLoop on PatchEval (original)

+25.2pp from PatchLoop with feedback loop

57.4%

one verify-feedback round (132/230)

33.5%

PatchEval strict, no feedback (77/230)

~$0.17

per CVE

≤3

bounded attempts

PatchEval (ByteDance) is a benchmark of 230 real-world CVEs from 2015 to 2025, across Go, JavaScript, and Python. Each case ships as a Docker image containing the vulnerable repo and a validation entrypoint that checks whether the vulnerability is still exploitable after a patch.

vs. original PatchEval leaderboard

System	Model	Type	Overall	Go	Python	Node.js
PatchLoop	gpt-5-codex	verify-feedback loop	57.4%	50.6%	61.4%	61.0%
OpenHands	GPT-5	agent	36.1%	36.1%	26.8%	44.7%
ClaudeCode	GPT-5	agent	34.4%	38.5%	22.5%	40.8%
PatchLoop	gpt-5-codex	base LLM, no loop	33.5%	-	-	-
SWE-agent	GPT-5	agent	33.0%	32.5%	22.5%	43.4%

Board rows are from the original PatchEval leaderboard (values as of 2026-07-27). PatchLoop's verify-feedback row uses validation output, so it was not eligible for that leaderboard and is shown for context only. See the disclaimer above regarding PatchEval-Verified.

attempt - / 3

Finding

Normalized, schema-validated at the edge (decodeFinding).

CVE-2099-0001 path traversal

Router

kind → harness

cve → CVE harness pentest → PoC harness ... → yours

RemediationHarness. the loop

Propose

One structured direct-edit request. No agent round-trips.

DirectEditPatcher

Apply

Exact search/replace onto the git workspace.

GitWorkspace

Verify: cheapest first, fail fast

Rescan: finding id gone? cheap

Regression: tests still pass? moderate

Reproduction: PoC dead? expensive

↩ Feedback path: verbatim failure output + cumulative diff → repair ON TOP of patched state

awaiting verified diff...

Harness log

// press Play to run a remediation

1Empty patch ≠ successThe largest silent-failure bucket in early runs. Counted as a failed attempt with explicit feedback.

2Repair on patched stateNo reset between attempts. Prompt carries current files, cumulative diff, verbatim failure.

3Reset only on bad applyWorkspace no longer matches what the model believes. Start clean and say so.

4Crashing verifier = failurecatchAllCause: any defect becomes a failed verdict. Enforced by types, not a remembered try/catch.

5Bounded attemptsThe first feedback round buys the most; returns fall off fast after 2 to 3.

A general coding agent (Claude Code style) is built for open-ended work: a ~20k-token system prompt, a catalog of built-in tools + MCPs, and a forced <think> → tool → <think> loop until it finally emits text. For a scoped job (patch this CVE), we already know the shape of the answer. So we skip all of that overhead and spend the intelligence budget on the verification loop instead.

same finding · CVE-2099-0001

General tool-calling agent

Claude Code style

Context every turn

system prompt ≈ ~20,000 tokens · re-sent every round

You are Claude Code, Anthropic's official CLI for Claude. You are an interactive CLI tool that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user. IMPORTANT: Assist with defensive security tasks only... Always follow these principles when writing code... Do what has been asked; nothing more, nothing less... NEVER create files unless they're absolutely necessary... Prefer editing existing files... When writing commit messages or PR descriptions... Be careful not to introduce security vulnerabilities... [...continues for ~20k tokens of tool schemas, MCP wiring, style rules, examples...]

Tool catalog (schemas + descriptions, every turn)

Think Read Glob Grep Edit Write Bash MCPxN

Forced turn loop

<think>plan approach...+1.2k

Glob**/*download*+0.4k

<think>narrow candidates...+0.9k

Readsrc/api/download.ts+2.1k

<think>locate traversal...+1.4k

Greppath.join|../+0.6k

<think>draft fix...+1.1k

Editsanitize path+0.8k

<think>re-check neighbors...+1.0k

Bashnpm test+1.5k

<think>fix one test...+1.3k

Editrestore behavior+0.7k

text"patched the CVE..."+0.4k

cumulative tokens this CVE 0

est. cost $0.00

PatchLoop DirectEdit

specialized harness

Context: once per attempt

finding + file contents ≈ ~2 to 4k tokens · no agent persona

Patch this finding with exact search/replace edits. Return only the structured EditPatch JSON.

What we skip

x 20k-token "you are a coding assistant..." system prompt

x tool schemas, descriptions, MCP catalogs

x forced <think> / Read / Glob / Bash round-trips

x the model deciding how to explore. We already know the job.

One structured request per attempt

proposeEditPatch JSON (attempt 1)~3k

applyworkspace.apply (free)0

verifyrescan / tests / PoC (your detectors)0

proposerepair on patched state (attempt 2)~3.5k

verifypassed → verified diff0

cumulative tokens this CVE 0

est. cost $0.00

~45k tok

General agent: system prompt + tool schemas re-paid on every turn, then ~12 tool/think cycles before a text answer.

~7k tok

PatchLoop: one structured propose per attempt. Verification is your scanners, not another model round-trip.

~10x

cheaper for the same scoped job. Intelligence budget spent on the verify-feedback loop, the part that actually moved PatchEval +25pp.