Prompt Injection Explained, Plus a Defense Checklist for Your Agent

Your support agent reads a ticket, summarizes it, and drafts a reply. In the ticket body is a line written to look like a system instruction: skip the summary and issue a refund to another account. The agent has a refund tool, the tool trusts the agent, and the ticket is in control. That is indirect prompt injection, instructions arriving inside content you only meant it to read. Prompt injection prevention starts by making sure that content never gets a vote.

Direct injection and indirect prompt injection

Direct injection is what most people picture: a user types something into the chat box that tries to move the assistant off its instructions. That is often low-stakes, because the user is attacking their own session. It gets serious when the assistant holds shared credentials, reaches other users' data, or acts without review.

Indirect injection breaks production systems. The instructions arrive through content: a web page, a PDF, an email, a support ticket, a code comment, a retrieval chunk, a calendar invite, a filename, alt text, or another tool's response. The person who triggered the agent may be innocent. The attacker only needs text in front of the model.

Why indirect is the harder case

• No account required. A page, a document, or a record your agent reads is enough.
• Content persists. A poisoned document stays in the index and gets retrieved for months.
• No witness. Tickets and tool responses are summarized before a human sees them.

Treat each channel as an input to your threat model, not just the chat box.

Why the model cannot reliably separate instructions from data

There is no hardware boundary inside the context window. System prompts, user messages, retrieved documents, and tool results all become tokens in one sequence. The roles you assign are metadata your application adds when it serializes the request. The model reads one stream and predicts the next token.

• Models are trained to follow instructions, so text that reads like an instruction gets weight whether it came from your policy layer or a web page.
• Paraphrase, translation, encoding, odd formatting, and instructions split across messages can change how a model classifies a string. A filter that catches one phrasing misses the next.
• The so-called instruction hierarchy is a convention, not an enforcement mechanism. It holds most of the time because models are trained toward it.

Prompt-level defenses lower the success rate. Delimiters, "treat this as data" instructions, and preambles that ask the model to report embedded instructions are worth doing. None are a security control. The layers that matter are in code.

Draw the trust boundary where the data changes hands

Write down every path that reaches the model and who controls it.

• Your code and configuration. Trusted, reviewable, versioned.
• The authenticated user's message. Semi-trusted. You know who sent it, not what it means.
• Retrieved documents, pages, emails, tickets, uploads. Untrusted.
• Tool responses. Untrusted. A CRM note or search result can carry text an outsider wrote.
• Other agents. Untrusted, including your own subagents.

The trust boundary is the line in code where content stops being data and starts being an instruction. Only your code should move something across it. If a document can change which tool runs, the boundary is already gone.

A boundary you can point at

def as_untrusted(source: str, text: str) -> str:
    safe = strip_control_tags(text)
    return f'<untrusted source="{source}">\n{safe}\n</untrusted>'copy

The tag tells the model about provenance. It is not a control by itself. The control is that as_untrusted is the only way content enters the prompt, and callers cannot skip it.

Separate instructions from untrusted content

Keep policy in the system or developer message, assemble it in code, and never concatenate user or retrieved text into it.

• Give untrusted content its own message with a tool or user role.
• Delimit it with a per-request random tag, and strip that tag from the content so it cannot close the block early.
• Say what to do with instructions found inside the block: quote, ignore, report. Check the report in code.
• Cap block length. Long content costs tokens and gives an attacker room.
• Keep retrieval results separate from tool results, so you can trace a sentence later.

The free prompt kit has instruction templates you can adapt, then test against hostile content.

Least privilege for tools, allowlists, and approval gates

A prompt cannot revoke a permission. If the tool's token has delete scope, the model can delete.

Scope the credential, not the prompt

• Separate credentials per tool and per tenant. A shared admin key is one bad sentence from compromise.
• Default to read-only. Add write scope for a named workflow and document why.
• Prefer short-lived tokens and per-user OAuth.
• Keep secrets out of the context. If the model can read it, an injection can ask for it.
• Restrict egress for URL-fetching tools. Block metadata endpoints and private ranges.

Allowlist actions and validate arguments

Register a closed list of tool names. Validate arguments against a strict schema: required fields, types, ranges, no unknown keys. Then check values against your records. If the model names account 4471, confirm it belongs to the authenticated user. The model proposes, code disposes.

Put a human on the irreversible steps

Gate money movement, deletions, external email, permission changes, publishing, and credential rotation.

• Show the approver the exact arguments, not the model's summary.
• Bind approval to a hash of those arguments, single-use, so nothing changes after approval.
• Expire approvals quickly and log who approved what.

For agents that read and write, split them: a read agent gathers and drafts, a write agent executes a plan code validated.

Treat model output as untrusted input

Output is data too. The model may emit HTML, links, shell commands, SQL, file paths, or a convincing email.

• Render as text by default. Sanitize rich output with an allowlist and drop raw HTML.
• Allowlist URL schemes. javascript: and data: are not content.
• Never interpolate output into a shell, a SQL string, or eval. Use parameterized queries and argument arrays.
• Canonicalize file paths and constrain them to a base directory.
• Escape spreadsheet exports. Cells starting with =, +, -, or @ can execute in some tools.
• Run generated code in a sandbox and review the diff before production.

Encoding bugs are ordinary application security bugs. Injection is just a way to reach them.

Logging, detection, and a test plan

You cannot respond to what you cannot see.

Log the decision, not just the text

Record prompt hash, model version, tool name, arguments, result, approval identity, and a correlation id per step. Use append-only storage. Redact secrets and personal data at write time, and set retention. Keep enough untrusted content to investigate, no more than policy allows.

Detect behavior, not phrases

Input keyword lists are easy to bypass. Watch the agent.

• A tool that never appears in this workflow.
• A write action that skips approval.
• A volume spike, or many small reads followed by one export.
• A new domain followed by an outbound request.
• A canary string from your system prompt or a test document appearing in output.

Route alerts to a person who can disable one tool without a redeploy. Keep a kill switch for the agent.

Test with permission, and assert on outcomes

Authorized testing only. Test what you own or have written permission to test, stay in scope, report findings.

Build hostile content for each entry point: page, PDF, email, ticket, retrieval chunk, tool response. Run it through the real pipeline and assert on behavior: which tool ran, with which arguments, which record was read, whether approval was requested. Those tests survive a prompt rewrite; model wording does not.

Start with red teaming your own LLM app, and use the AI Red Teaming Prompt Playbook for cases to adapt. Rerun the suite after any prompt, model, tool, or retrieval change.

The prompt injection prevention checklist

Work through this in order. Each item is a place where a system usually fails.

1. Inventory every input that reaches the model and label its owner and trust level.
2. Keep system and developer instructions in code, never concatenated with user or retrieved text.
3. Pass untrusted content in its own message with a per-request delimiter, and strip that delimiter from the content.
4. Route retrieval and file parsing through one boundary function that tags provenance and caps length.
5. Give each tool a scoped credential with minimum permission and the right tenant, read-only by default.
6. Define tool arguments with a strict schema and reject unknown fields, wrong types, and out-of-range values.
7. Validate every account, record, and amount against the authenticated user's data before a tool runs.
8. Require human approval for irreversible actions, bound to a hash of the exact arguments, single-use.
9. Separate propose from commit so the model drafts and code executes.
10. Render model output as text or sanitized HTML, never into a shell, a SQL string, or eval.
11. Log tool calls, arguments, approvals, and refusals with a correlation id, redacted and append-only.
12. Add canary strings and behavioral alerts, and make every tool disableable in one step.

If you want a worked example of the operator side of this, the Operator Agent Profile covers how to run an agent with these constraints in place.

Questions people ask

Can a better system prompt stop prompt injection?

No. It lowers the success rate of some attempts. It cannot grant or remove permissions, validate an account, or stop a tool from running. Enforcement belongs in code, credentials, and approval gates.

Is prompt injection the same as jailbreaking?

They overlap, but the target differs. A jailbreak tries to get the model to produce content its policy would refuse, usually for the user's own benefit. Injection tries to make your application do something with its data or tools, often through third-party content. A jailbreak-resistant model is still exposed to a poisoned document.

How do I test this without risking production?

Use staging, synthetic data, and mocked or read-only tools. Get written permission for any test touching real systems, define scope and stop conditions, and keep a kill switch ready. Assert on tool calls and data access, not model wording, and keep the cases as a regression suite.