An automated repair still needs acceptance checks
GitHub announced agentic autofix’s general availability on October 15, 2026, saying it reruns CodeQL before opening a draft pull request. According to GitHub, the agent explores repository files, proposes changes, and iterates when needed. The pull request includes an explanation and validation steps. Reviewers still need to determine whether the patch prevents the security violation while preserving required behavior. Sources: GitHub
GitHub says users can assign default-branch alerts individually or in groups of up to 25. Alerts found only on other branches or pull requests cannot be assigned, according to the announcement. Separately, GitHub announced AI Scan’s general availability on October 15, 2026, describing checks of pull request changes in languages including PHP and Shell/Bash. GitHub says its findings are advisory and do not fail the code-scanning check or block merges through rulesets. Its documentation says AI Scan neither scans entire repositories nor creates backlog alerts in the repository security view. Treat these as distinct review surfaces, and inspect AI Scan findings separately from merge-check status. Sources: GitHub, GitHub Docs
Define success beyond stopping one failure
The March 2026 preprint Patch Validation in Automated Vulnerability Repair shows how a patch can pass a narrow test while breaking valid behavior. In its PHP example, an AI patch stopped a crash by rejecting valid mixed-type arguments. The maintainer’s test instead expected a correct numeric array. In PVBench, PatchAgent with GPT-4.1 passed basic checks in 76.4% of attempts but stronger maintainer-derived checks in 44.5%. A safe rejection is insufficient when the application must process the input successfully. Sources: arXiv
Meta’s AutoPatchBench evaluation added ten-minute fuzzing and white-box differential testing after build and crash checks. In a manual sample of 44 fuzz-passing patches, 12 passed differential testing against a known human repair; reviewers accepted five. Separately, 1Password’s Off-by-1 Labs reported complete fixes without material behavior changes in 26.0% of 6,080 evaluated patches across six selected vulnerabilities. These studies tested different systems and vulnerability populations, so their percentages cannot be pooled or treated as current GitHub agentic autofix’s failure rate. GitHub describes its repair workflow, but these results do not establish how often its current patches pass independent correctness checks. Use the studies to choose acceptance checks, rather than predict a repository’s results. Sources: GitHub, arXiv, Engineering at Meta, 1Password
Freeze the case and reproduce the security violation
Start by recording the vulnerable commit, candidate commit, alert identifier, scanner configuration, dependency lockfiles, and build environment. Write down the violated security property and the legitimate behavior that must remain possible. For an authorization flaw, specify which users may access which resources. This suggested workflow defines acceptance criteria before the agent’s implementation influences the tests.
Next, reproduce the vulnerability against the vulnerable revision in a controlled environment. Assert the consequence itself: private data returned, a forbidden file read, a prohibited connection made, or a sanitizer error. A failed request proves little if the service never started. Preserve fixtures, prerequisites, and logs, then turn the reproduction into a regression test of the desired outcome. The table below pairs proposed security assertions with legitimate functionality that an overly restrictive repair could break. Server-side request forgery is abbreviated as SSRF in the table.
Table
Proposed security and behavior checks by vulnerability class
| Example vulnerability | Demonstrate the problem | Required post-patch behavior | Preservation cases |
|---|---|---|---|
| Object-level authorization | User A retrieves User B’s private object | Access is denied and no private contents are returned | Owner access and explicitly authorized sharing still work |
| Path traversal | Input escapes the permitted directory | No read occurs outside the intended boundary | Valid nested paths and supported filename forms work |
| SSRF | A request reaches a prohibited destination | Prohibited connections never occur | Permitted destinations and supported redirect behavior work |
| Parser memory corruption | A fixture triggers a sanitizer error | Defined output or documented error, with no sanitizer finding | Valid boundary inputs retain their specified output |
Generate a candidate, then challenge its assumptions
Give the repair agent the report, reproduction, intended behavior, and relevant architectural context. Ask it to explain the root cause and how the proposed change enforces the security property. Treat that explanation as a hypothesis, then inspect execution output for claimed tests and scans. GitHub says agentic autofix supplies validation steps in its draft pull request, giving reviewers a starting point for that inspection. Sources: GitHub
Develop additional attack cases independently of the candidate’s implementation. Vary equivalent input representations, adjacent boundaries, alternate entry points, roles, and relevant configurations. Keep some evaluation cases outside the agent’s repair loop to detect adaptation to visible tests. For suitable parsers and native libraries, add sanitizers and record the fuzzing budget. Meta’s comparison against a known human repair offers another check, though that reference may be unavailable for a newly discovered vulnerability. Sources: Engineering at Meta
Run the existing suite and targeted integration tests for functionality near the security boundary. Inspect changed error handling, deleted branches, weakened assertions, disabled scanners, and new dependencies. Test performance where the repair changes algorithmic cost. If security requires a deliberate behavior change, document it as a compatibility decision instead of counting it as preserved functionality.
Rescan comparable code and inspect advisory findings
Run the originating scanner against the candidate and record its version, queries, build mode, exclusions, and analysis scope. Confirm that analysis completed successfully and covered the relevant code before interpreting a missing alert. Inspect new findings across the patch as well as the original finding. A skipped build, added exclusion, or suppression must not count as evidence that the vulnerability disappeared.
For third-party alerts, inspect which analyzer actually ran. GitHub’s July 10, 2026, preview announcement described third-party alert support but referred both to rerunning the original analysis and to rerunning CodeQL. Its October 15, 2026, announcement describes CodeQL validation. These descriptions leave the validation path for a particular third-party finding unclear. Attach the originating scanner’s successful run to the review rather than inferring it from the agent’s summary. Sources: GitHub
Record AI Scan findings separately from check status. GitHub says AI Scan excludes fork and Dependabot pull requests, and its documentation warns that false positives are possible. Record excluded pull requests as unscanned and assess reported findings on their merits. Without a reliable vulnerability reproduction, label the result as scanner-cleared with exploit-level verification unavailable. Sources: GitHub, GitHub Docs
Measure verified repairs on the revision that merges
Before approval, have a domain reviewer assess the security property, alternatives, and test adequacy. Attach the reproduction, variant-test results, functional checks, and scanner execution evidence to the pull request. Rerun required checks against the revision that will merge, and retain regression tests in continuous integration. Evidence tied to an earlier candidate cannot establish the behavior of a later change.
Measure every attempted case, including those that produce no applicable patch. Report per-attempt and per-vulnerability outcomes separately so repeated retries do not inflate apparent success. The table below separates intermediate results from repairs that satisfy every required check. Track blocked environments and inconclusive results separately. Also record human review time, elapsed time, model usage, and continuous integration cost per accepted repair.
Table
Suggested measures for repair verification
| Measure | Suggested definition |
|---|---|
| Candidate production rate | Cases producing an applicable patch ÷ cases attempted |
| Original reproduction blocked | Candidates that prevent the demonstrated violation ÷ candidates evaluated |
| Variant-test pass rate | Candidates passing the independently designed attack cases ÷ candidates evaluated |
| Behavior preservation | Candidates passing required functional checks ÷ candidates evaluated |
| Validated alert clearance | Candidates whose originating finding disappears after a successful comparable scan ÷ candidates successfully rescanned |
| Verified repair yield | Cases meeting every required gate and receiving review approval ÷ cases attempted |
| Regression rate | Evaluated candidates introducing an identified functional regression ÷ candidates evaluated |
For a team evaluation, calibrate verification with historical vulnerabilities whose repairs are known, keeping those repairs out of the generation context. Then collect prospective cases and compare AI-assisted and human repairs under the same acceptance criteria. Record models, prompts, retry limits, tools, budgets, sample size, and uncertainty. Separate results by language and vulnerability class. Track accepted patches later found incomplete or regressive over a stated observation window.