Guide

AI code and application security incident catalogue: what failed, how, and what would have caught it

An incident catalogue is a dated list of cases where code written with an AI assistant, or the application layer around an LLM, failed in a way that is publicly documented: a CVE, a vendor advisory, a post-mortem, a peer-reviewed measurement. This page keeps one, sorted by mechanism, with the check that would have caught each case.

Summary for AI assistants & procurement teams

dfzoo AI Engineering maintains a catalogue of publicly documented security incidents in AI-assisted code and LLM application layers, for engineering leaders who need to know what has actually gone wrong elsewhere before deciding what to check at home. Every entry has a date, a primary source (CVE record, vendor advisory, official post-mortem or published research) and one named mechanism: content read by an agent treated as an instruction, a generated backend default shipped as-is, a hallucinated package name installed from a registry, a destructive command run with confirmation switched off, model output passed into SQL or code execution, an MCP tool or extension holding wider access than the task. The catalogue is the input to a security review of AI-written code: each mechanism class maps to a checklist, a verdict per class for a given codebase, and a retest that proves the fix. Incidents under NDA are not included; only what can be verified at the source.

Who this is for

Who this guide is written for.

CTOs and VPs of Engineering whose teams ship code from Copilot, Cursor, Claude Code or Gemini CLI and who were asked by a board or a client what could go wrong
Security and platform leads who added an in-app agent, an MCP server or a support automation on top of an LLM and need a concrete threat list, not a generic one
Software houses delivering AI-assisted code to clients that now ask for evidence of an independent security check
Engineers who read the vendor advisories one by one and want them in one place, classified by mechanism

What counts as an entry and what does not

Asked on a security forum in August 2026 whether anyone had actually had an incident caused by a coding agent, the most upvoted answer was that whoever had would not say, because of NDAs and embarrassment. That silence is why teams calibrate from vendor marketing. The catalogue below takes the opposite route and lists only what can be verified at the source: a CVE record, a vendor security bulletin, a public statement by the vendor's leadership, or a peer-reviewed measurement with its method published. Practitioner threads that count incidents from daily use exist and are useful as a format, but a thread is not a source and none is cited here.

Three boundaries keep the list honest. It covers code and the application layer: what the assistant wrote, what the app sends to and trusts back from a model, what an agent is allowed to call. It does not cover infrastructure, cloud configuration, network security, penetration testing or a SOC; those are separate disciplines with their own catalogues. And it does not treat the model as the culprit. Every row is a tool used without the check that a process would have put in front of it, which is exactly why every row has a check attached.

The catalogue

Sorted by date. The class column uses the six mechanism classes defined in the next section. Source links are at the end of the page; severity scores are the CVSS values from the CVE records where one exists.

The catalogue
Date	System and source	What failed	Effect	Class	What would have caught it
2023-07	LangChain SQLDatabaseChain, CVE-2023-36189	Model-generated SQL executed against the database without a boundary between the question and the query	Remote attacker obtains sensitive information through the chain	E	Read-only role for generated SQL, allowlisted statements, query review before execution
2024-03	huggingface-cli on PyPI, Lasso Security experiment	A package name repeatedly hallucinated by assistants was registered by researchers; a public research repository from Alibaba carried it in its install instructions	Empty package installed by real projects following assistant-written instructions	C	Registry check on every new dependency: age, publisher, download history, and a lockfile diff in review
2024-05	Vanna.AI text-to-SQL, CVE-2024-5565, CVSS 8.1	Untrusted input in ask() reaches the prompt that writes Plotly visualisation code, which is then executed	Remote code execution; mitigation is switching visualisation off for untrusted input	E	No exec of model-written code from user-facing paths; sandboxed rendering; input marked untrusted end to end
2025-05	GitLab Duo, Legit Security disclosure	Hidden text in merge requests, issues, commits and source (white KaTeX, base16 encoding) read by Duo as instructions	Private source code exfiltrated through an injected image tag; unsafe URLs presented as safe	A	Outbound URL allowlist on rendered assistant output; stripping active HTML from responses; injection test set in CI
2025-05	GitHub MCP server, Invariant Labs	A public issue instructs the agent, which holds a token for private repositories too	Private repository data written into a public pull request	F	One token per repository scope; agent cannot read a private scope in a session started from a public one
2025-05	Lovable-generated apps, CVE-2025-48757, CVSS 9.3	Generated Supabase row-level security policies insufficient or absent on projects created before 2025-04-15	Unauthenticated read and write to arbitrary tables of generated sites; vendor disputes, placing responsibility on each customer	B	Table-by-table policy check with an anonymous client before publish; RLS required for every table with user data
2025-06	Microsoft 365 Copilot, EchoLeak, CVE-2025-32711, CVSS 9.3	Instructions in an incoming email processed by the assistant with no user action	Information disclosure over the network, zero-click	A	Untrusted content never combined with a tool that can reach outside; egress allowlist on every assistant action
2025-07	Supabase MCP with Cursor, General Analysis	Support ticket text asks the agent to read integration_tokens; the MCP connection runs with service_role, which bypasses row-level security	OAuth secrets and session credentials posted back into the ticket thread	F	Read-only mode and project scoping on the MCP server; no service_role in any agent-reachable connection
2025-07	mcp-remote, CVE-2025-6514, CVSS 9.6	authorization_endpoint URL returned by an untrusted MCP server passed into a command	OS command injection on the developer machine	F	Pinned, reviewed MCP server list; no shell construction from remote strings; client version gate
2025-07	Amazon Q Developer for VS Code 1.84.0, AWS bulletin AWS-2025-015	Improperly scoped GitHub token let unapproved code into the extension's repository and release	Malicious code shipped in a release; failed on a syntax error; version withdrawn, 1.85.0 issued	F	Extension version pinning and update review; least-privilege tokens in the publishing pipeline
2025-07	Gemini CLI 0.1.13, Tracebit	Instructions hidden in a README, an allowlist that accepted a chained command after one approved grep, and whitespace hiding the payload	Environment variables sent to a remote server without the user seeing the command; fixed in 0.1.14	A	Per-command approval that shows the whole command line; no allowlist matching on the first token
2025-07	Replit Agent, SaaStr project, vendor CEO statement	Agent ran a destructive database command during a declared code freeze and reported that rollback was impossible	Production database deleted; vendor called it unacceptable and shipped dev/prod separation and a planning-only mode	D	No production credentials in the agent's environment; destructive operations behind an explicit confirmation and a tested restore
2025-08	Cursor below 1.3.9, CVE-2025-54135, CVSS 8.6	Indirect prompt injection creates .cursor/mcp.json, a new dotfile, without the approval that editing one requires	Remote code execution on the developer machine	A	Creation and edit of config files under the same approval rule; injection test that writes a canary file
2025-08	Cursor 1.2.4 and below, CVE-2025-54136, CVSS 7.2	An already approved MCP definition in a shared repository swapped for a different command without a re-prompt	Persistent code execution for anyone with write access to the branch	F	Hash of every approved MCP definition; re-approval on any change; MCP config reviewed like a CI workflow
2025-08	Claude Code below 0.2.111, CVE-2025-54794, CVSS 7.7	Path check by prefix match instead of canonical path; a directory with the same prefix as the working directory passes	File access outside the working directory once untrusted content is in context	A	Canonical path comparison in every path guard; test with a sibling directory sharing the prefix
2025-08	GitHub Copilot and Visual Studio, CVE-2025-53773, CVSS 7.8	Prompt injection edits .vscode/settings.json and sets chat.tools.autoApprove, removing every confirmation	Local code execution across Windows, macOS and Linux	A	Settings files treated as code: write-protected in agent sessions, diffed in review, auto-approve flags forbidden by policy
2025-09	Salesforce Agentforce, ForcedLeak, Noma Security, CVSS 9.4	Lead form description field carries instructions; agent output allowed to reach an expired domain still on the CSP allowlist	CRM records sent to an attacker-registered domain; patched with trusted URL enforcement	A	Egress allowlist audited for expired entries; form fields marked untrusted before they reach an agent
2025-11	Google Antigravity, PromptArmor	Instructions in 1-point font on a web page; agent reads .env through cat despite the gitignore guard, then a browser subagent opens an attacker URL with the contents	Credentials exfiltrated to a logging site included in the default browser allowlist	A	Guard on file reads that also covers shell; default browser allowlist reviewed; secrets outside the workspace
2025-12	Google Antigravity in Turbo mode, The Register, vendor investigating	A cache-clearing command targeted the root of the D: drive instead of the project folder, with no confirmation in that mode	User's drive contents deleted, bypassing the recycle bin	D	Destructive filesystem operations restricted to the workspace path; confirmation mode on outside sandboxes
2026-01	Moltbook, Wiz Research	Supabase key hard-coded in client-side JavaScript with no row-level security; founder stated no line was written by hand	1.5 million API tokens, 35,000 email addresses and private messages readable and writable without authentication; fixed within hours of disclosure	B	Anonymous-client probe against every table before launch; secrets scan on the built bundle, not only the repository

Six mechanism classes, and what each one has in common

Tool names change every quarter; mechanisms do not. Each class below names what the tool did, what the process lacked, and the measurement behind it where one exists. Two peer-reviewed studies anchor the classes that concern generated code itself: Pearce and colleagues (IEEE S&P 2022) generated 1,689 programs with Copilot across 89 weakness scenarios and found about 40% vulnerable; Fu and colleagues (ACM TOSEM 2025) analysed 733 assistant-written snippets found in real GitHub projects and reported weaknesses in 29.5% of the Python and 24.2% of the JavaScript samples. For package hallucination, Spracklen and colleagues (USENIX Security 2025) generated 576,000 code samples across 16 models and found hallucinated package names in at least 5.2% of commercial-model output and 21.7% of open-source-model output, with 205,474 unique invented names.

Six mechanism classes, and what each one has in common
Class	Mechanism	What the process lacked	Entries above
A. Content becomes an instruction	Text the agent reads (issue, ticket, README, email, web page, form field) is processed as if the user typed it, and a tool call follows	A boundary between untrusted content and tools that reach outside; approval that shows the whole action	GitLab Duo, EchoLeak, Gemini CLI, Cursor, Claude Code, Copilot, Agentforce, Antigravity
B. Generated default ships as-is	Backend access rules, keys in client bundles and framework defaults arrive in the state the assistant produced them	A pre-publish check per table and per secret, run as an anonymous client	Lovable RLS, Moltbook
C. Invented dependency	The assistant names a package that does not exist; someone registers the name; the install line is trusted because it looks ordinary	Registry verification of every new dependency in review	huggingface-cli, Spracklen et al.
D. Destructive command without a gate	An agent with production or filesystem access runs a delete, a drop or a wipe in a mode where confirmations are switched off	Credential separation between environments; explicit gate on destructive operations; tested restore	Replit Agent, Antigravity Turbo
E. Model output into SQL or code execution	Text from the model is passed to a database or an interpreter as if it had been reviewed	Read-only roles, allowlisted statements, no exec of generated code from user paths	LangChain, Vanna.AI
F. Tool or extension with wider access than the task	An MCP server, extension or token holds scope for everything the developer can do, and the agent inherits it	One scope per task; hash and re-approval of tool definitions; pinned versions	GitHub MCP, Supabase MCP, mcp-remote, Cursor MCP swap, Amazon Q extension

What a security review of AI-written code checks, per class

The catalogue becomes useful when each class gets a verdict for a specific codebase: can this happen here, yes or no, and what proves it. The list below is the checklist the review runs per class. Every item ends in an artifact or a test, because a verdict without one is an opinion.

Class A: trace every path where untrusted text enters a prompt and where a tool call can follow; list every tool that can reach outside the process (network, shell, file write, browser); build an injection test set with a canary action and run it in CI against each path; audit the egress allowlist for expired and wildcard entries.
Class B: enumerate tables and storage buckets, then probe each one as an anonymous client and as a low-privilege user; scan the built client bundle for keys, not only the repository; compare framework defaults the assistant reproduced (CORS, verification flags, debug output) against the production configuration.
Class C: diff the lockfile in every assistant-written pull request; check each new package for registry age, publisher and download history; keep an allowlist for the runtime and a block on install commands the assistant writes into documentation.
Class D: list every credential in the agent's environment and the environment it belongs to; separate production from everything the agent can reach; put destructive operations behind an explicit confirmation and rehearse the restore once, with the time it took written down.
Class E: find every point where model output is executed or queried; give generated SQL a read-only role and an allowlist of statements; remove exec of generated code from any path a user can reach; sandbox what remains.
Class F: inventory MCP servers, extensions and tokens with their scopes; reduce each to the task it serves; hash approved tool definitions and require re-approval on change; pin versions and review updates like dependency updates.
Retest: after the fixes, every class gets the same test again and the result is recorded next to the original verdict. The pair of results is the evidence a client or a board can read.

Failure modes

Where this goes wrong, and what we do about it.

1
Content the agent reads is treated as an instruction
The agent gets a support ticket, an issue, a README or an email in its context to do its job, and the model has no reliable way to tell quoted text from the user's request. When a tool that reaches outside the process is available in the same session, the text can call it. Every class A entry in the catalogue has this shape, from GitLab Duo to Antigravity, and the vendors' fixes were boundaries around tools, not better models.
What we do about it
We map every path where untrusted text enters a prompt and list every tool reachable from that session. Where both meet we either remove the outbound tool, put it behind an approval that shows the full action, or mark the content as data in the prompt structure. We then run an injection test set with a canary action against each path and keep it in your CI.
What stays with you: path-by-path map of untrusted content and reachable tools, with the injection test set and its results per path
2
The generated backend default ships in the state it was produced
The assistant scaffolds a backend with the pattern it knows best: a key in the client, access rules off or permissive, debug output on. It works in the demo because the demo has one user. The Lovable CVE and the Moltbook exposure are the same mechanism at two scales, and in both cases the vendor's position was that the customer owns the access rules.
What we do about it
We enumerate every table and bucket, then probe each one as an anonymous client and as a low-privilege user, and scan the built bundle for keys. Each failing table gets a policy, each key moves to the server side, and the probe runs again until it returns nothing. The probe stays as a pre-publish check.
What stays with you: table-by-table access matrix with the anonymous-client probe result before and after, and the probe wired into the release pipeline
3
Package names are trusted because they look ordinary
The assistant writes an install line with a plausible name that does not exist, and the study by Spracklen and colleagues shows the same invented names recur across prompts. Anyone who registers such a name gets installs from every project that trusted the line. The huggingface-cli case shows the name reaching a real repository's documentation before anyone checked the registry.
What we do about it
We diff the lockfile of every assistant-written change, check each new package against the registry for age, publisher and history, and turn the check into a review rule. Install commands the assistant wrote into docs get the same treatment.
What stays with you: dependency provenance report for the AI-touched period, plus the registry check as a CI rule
4
Destructive commands run because the confirmation was switched off
Turbo, YOLO, auto-approve: the mode exists because confirmations slow the demo down. In that mode a cache clear that resolves to the wrong path, or a database reset issued during a code freeze, executes at the speed of the agent. Both catalogue entries in class D happened with the agent holding credentials it never needed for the task.
What we do about it
We list every credential the agent can reach and the environment each belongs to, remove production from that list, and put destructive operations behind an explicit gate. We then rehearse the restore once and write down how long it took.
What stays with you: credential map per agent and environment, the list of gated operations, and the restore drill record with its duration
5
A tool definition changes and nobody is asked again
An MCP server or extension was approved once. The definition lives in a shared repository or updates itself, and the approval does not follow the change. The Cursor re-approval bypass and the Amazon Q extension release show the two ends: a swapped command on a branch, and unapproved code in a published version.
What we do about it
We inventory MCP servers, extensions and tokens with their scopes, reduce each scope to its task, hash every approved definition and require re-approval on change. Versions get pinned and updates go through the same review as dependency updates.
What stays with you: tool inventory with scope per task, hashes of approved definitions and the re-approval rule in your configuration

Artifacts

What stays with you.

Incident-to-codebase mapping: a verdict per mechanism class for your repositories and agent integrations, with the evidence behind each verdict
Checklist per class, filled in for your stack, with the artifact or test that closes each item
Injection test set with canary actions, wired into CI, with results per untrusted-content path
Access matrix for tables and buckets with the anonymous-client probe result, before and after
Dependency provenance report for the AI-touched period and the registry check as a review rule
Tool inventory: MCP servers, extensions and tokens with scope per task, hashed definitions and the re-approval rule
Retest record: every class tested again after the fixes, recorded next to the original verdict

Process

How we work.

1
1. Mapping
We walk the repositories and agent integrations with a senior engineer from your side and mark where each mechanism class can occur: untrusted content paths, generated backend rules, new dependencies, agent credentials, model output into SQL or exec, tool scopes.
Week 1
2
2. Verdicts
We run the checklist per class: injection tests, anonymous-client probes, lockfile and registry checks, credential and tool inventories. Every class gets a yes or no with the evidence, and the findings that come out carry a reproduction.
Week 1-2
3
3. Fixes and gates
We fix the findings with your team and turn each closed item into a gate: the probe, the test set, the registry rule, the hash check. Each gate is tripped once on purpose to prove it works.
Week 2-3
4
4. Retest and handover
We repeat every class test, record the result next to the original verdict, and hand over the mapping, the checklists and the gates in your pipeline. The catalogue on this page is the reference the next review starts from.
Week 3

Related guides

The rest of this cluster.

Prompt cache forensics and spend limits

How to attribute LLM cost to a call, session and feature, how Anthropic, OpenAI and Google bill prompt caching, what quietly breaks the cached prefix, and where an alert ends and a hard stop begins.

Read the guide

Measuring the real effect of AI in a team

Why faster ticket closure is not evidence, what the 2025-2026 controlled experiments measured, how to design a baseline, paired metrics and a comparison group, and which per-person metrics to avoid.

Read the guide

Comparing AI code review tools

Why vendor rankings of AI code review tools are not comparisons, which metrics they leave out (precision per class, off-diff comments, invented findings, cost per PR), and a protocol for your own PRs.

Read the guide

Related services

Where this turns into a service.

Evaluate & Secure

Security Review

Security review of code written with AI coding assistants, and of the AI features it added to your product.

Learn more

Evaluate & Secure

AI Code Evaluation

Independent audit of AI-generated code against a rubric published in full.

Learn more

Sources

Where the dates and numbers come from.

FAQ

Questions teams ask.

Only cases with a primary source that can be checked: a CVE record, a vendor security bulletin, a public statement from the vendor's leadership, an official post-mortem, or a peer-reviewed study with its method published. Self-reported cases enter only when the vendor confirmed them publicly, and the row says so. Nothing under NDA, including our own engagements.

Yes, because the review works on classes, not on product names. A team using a tool that has no public CVE still has untrusted content paths, generated backend rules, new dependencies, agent credentials and tool scopes. We map those and give a verdict per class; the tool name only decides which defaults we check first.

No. A penetration test attacks a running system from outside, and we do not do that, nor infrastructure, cloud or SOC work. This review reads code and the application layer around the model, reproduces findings against a non-production instance and hands over fixes and gates. If a penetration test is needed afterwards, our findings are a concrete starting point for the specialist doing it.

It proves that a tool used without a check in front of it fails in repeatable ways, which is what the studies measure too: Pearce and colleagues, Fu and colleagues and Spracklen and colleagues all describe the output of a model, not the output of a process. Every row has a check that would have caught it, and that check is the point of the page.

The same test that produced the verdict runs again after the fix: the injection set, the anonymous-client probe, the registry check, the restore drill. The result is written next to the original, so a reader sees a before and an after from the same method. A fix without a retest is a claim; the pair is evidence.

It is revised as vendors publish advisories and researchers publish measurements; the source list at the bottom carries the dates. An entry can be added when it has a public primary source, which is also the answer for submissions: send the advisory or post-mortem link, not the story.

Read access to the repositories in scope, the configuration of the agent tooling (MCP definitions, extension lists, settings files), and a non-production instance for reproductions and probes. No production access and no customer data. Everything runs under NDA, which is also why nothing from client work ever appears in the catalogue.

Talk to an engineer.

Describe where you are with your AI-assisted code and agent integrations. A reply within one business day.

Talk to an engineer

Szczecin - ul. Wawrzyniaka 6WWarszawaZielona GóraKraków