An agent can act as a confused deputy
The confused deputy is a longstanding security problem: a privileged program acting on instructions from someone less privileged. An agent that reads untrusted text and holds credentials is exactly that, which is useful, because it means the defences are known and none of them is a better prompt.
TL;DR: The vulnerability is not that models are gullible. It is that a process with credentials is taking instructions from data it read, and data is attacker-controlled far more often than people assume. Design as though the injection succeeds: no ambient credentials, narrow tools that authorise independently, human approval on effects, egress denied by default, and every tool call recorded.
The old problem in new clothes
Consider a compiler allowed to update a protected billing file. If it uses that authority to write diagnostic output to a filename chosen by an ordinary user, the user can overwrite a file they cannot access directly. The deputy has used its own authority for a caller-selected target.
An agent reading a pull request description and holding a repository token is the same shape. It cannot reliably separate the maintainer's intent from text sitting inside the data it was told to summarise. When those conflict, the agent is a deputy, confused.
Recognising the shape matters because the mitigations are already known. This is not a new class of problem needing new theory, it is a known class arriving in a context where people forget to apply the controls.
Untrusted input is wider than the list people write down
Issue bodies and comments, from anyone who can open an issue. Pull request titles and descriptions from a fork. A code comment in a contributed diff. The README of a transitive dependency the agent reads while investigating a build failure. A log line, if an earlier step echoed something a user supplied. A web page, if the agent browses. A document in the retrieval corpus, if anyone can add documents.
Every one of those is an instruction channel. The question to ask about any agent is not whether it reads untrusted content but which of these it reads, and the answer is usually more than the team listed.
Why filtering does not work
The instinct is to detect the attack: scan for suspicious phrasing, instruct the model to ignore instructions found in data, run a second model as a classifier.
Each raises the cost of an attack. None is a boundary. Instructions can be phrased in any language, encoded, split across documents, or written in a way no filter anticipated, and the detector is itself a model reading attacker-controlled text. Treat these as defence in depth and never as the control you rely on.
The system prompt is in the same category. Every rule written there is advisory, and there is no log entry when it is ignored.
The controls that are boundaries
Minimize ambient authority. A read-only token can still expose private data. Prefer task-scoped access through a broker. If direct read access is required, restrict its data scope and outputs. The broker receives attacker-influenced arguments and must validate the exact target and operation independently.
Tools that authorise themselves. Not a shell. Narrow tools whose permissions are checked at the tool, so a comment-on-pull-request tool cannot delete a branch regardless of what it is asked.
Approval on effects, not on reasoning. A human approves the merge, the apply, the send. Approval on intent is theatre because intent is what was compromised.
Egress denied by default. Exfiltration usually needs a network call, and an allowlist plus logging reduces reachable paths. This includes the indirect ones: a rendered markdown image whose URL carries data is a GET request to a host the attacker controls, with no script execution involved.
A record of tool calls. Not the model's messages, which describe what it believed. Record the operation, acting identity and result in a store the agent cannot alter. Redact credentials and sensitive arguments; an audit log must not become another data leak.
The question that tests a design
Assume the injection worked. The attacker now controls what the agent tries to do. What can it reach, what can it change, and what would you be able to reconstruct afterwards?
Those controls give the review concrete boundaries to test. They do not prove the whole system secure: a user may have broad permissions, an approved action may be misleadingly presented, and allowed destinations may still leak data. Test the exact operation and data flow under attacker-controlled tool arguments.
Self-check
Does replacing a write token with a read-only token prevent exfiltration? No. Reads can disclose private records, and tool output or an allowed destination can carry them away. Restrict both the readable data and where results may go.
Sources: OWASP excessive agency controls.