How to Red Team Your Own LLM App: A Scoped, Authorized Test Plan

Most LLM app assessments start in the wrong place. Someone opens the chat window, pastes a jailbreak found on social media, gets a rude answer, and writes it up as a finding. Nobody can say what was in scope, what the impact was, or whether the fix held. The team learns nothing it can act on.

The scoping problem is harder than the prompts. Serious LLM red teaming starts with deciding how to test an LLM application you have written permission to touch: which endpoints, which accounts, which data, which hours, and who gets called when something breaks. Once that is settled, the testing is mostly disciplined repetition.

This plan covers one application over one short engagement. It assumes you own the system or hold written permission from the owner. If neither is true, stop and get the permission.

The scoping problem comes before the prompts

An LLM app has no clean perimeter. The model is a dependency you do not control, the prompt is code spread across three services, and the retrieval index is a copy of your data with its own access rules. Testers who skip scoping spend the engagement arguing about whether a behavior is a bug or the model's personality.

Answer three questions before you open a terminal:

• What can this app do on its own? Read, summarize, send, book, pay, delete, call other systems.
• What can it reach? Retrieval sources, user records, internal APIs, credentials in the runtime.
• Who can reach it? Anonymous visitors, free accounts, paying customers, staff.

Write them down. They set the boundary for what counts as a finding.

Rules of engagement for LLM red teaming

Put the agreement in writing and have the system owner sign it, not your manager. Verbal approval from a colleague is not authorization. The Authorized Use Policy is a reasonable starting point for the language.

The agreement should name:

• The parties, the approver, and the date approval was given.
• The environment, the URLs, the accounts, and the test tenant.
• The data you may use and the data you may not touch.
• Actions that are out of bounds: load testing, destructive tool calls, exfiltration beyond what proves the point, any third-party system.
• A spend cap and a request rate cap.
• Stop conditions that end the test immediately, and the person to call.
• The disclosure window and where evidence is stored.

If the app writes to production data, get a test tenant with synthetic records. If that is impossible, use only accounts you created and data the owner approved.

The one-page test plan

Every engagement gets the same sheet, filled in before testing starts. Keep it in the evidence folder and attach it to the report.

ENGAGEMENT:  <app name, version, commit>
APPROVER:    <name, role, written approval date>
WINDOW:      <start and end, hours, timezone>
ENVIRONMENT: <staging URL, production allowed or not>
ACCOUNTS:    <test users and their roles>
DATA:        <test tenant, synthetic records, excluded data>
IN SCOPE:    <endpoints, UI surfaces, tools, retrieval sources>
OUT SCOPE:   <model provider, infra, third parties, other tenants>
CAPS:        <max spend, max requests per hour>
STOP:        <data leak, outage, cap reached, P1 contact>
EVIDENCE:    <storage location, retention, access list>
DISCLOSURE:  <who, within how many hours>
RETEST:      <date, pass criteria>copy

Threat model the app you actually shipped

A generic LLM threat list is useless. Model the app in front of you.

Start with assets: the system prompt and model config, the retrieval corpus, other users' data, tool credentials, and money. Then list actors: anonymous visitor, low-privilege account, staff account, a poisoned document in the index, a compromised upstream API. Then mark the trust boundaries where text stops being data and becomes instruction: user input into the prompt, retrieved chunks into context, model output into the UI, model output into tool arguments.

Build six to ten abuse cases from that. "A customer makes the assistant reveal another customer's order" is an abuse case. "Jailbreak the model" is not.

How to test an LLM application: six categories

Each category maps to a class of failure, not a payload. Work through them in order and note the ones that do not apply.

Instruction hierarchy and prompt injection

Check whether the app separates system, developer, user, and retrieved content, and whether it enforces that separation. Try to make text from a lower trust level change behavior set at a higher one. Put instructions inside a retrieved document, an uploaded file, or a field the app echoes back. Watch whether untrusted content is labeled as data in the prompt. The prompt injection defense checklist covers the controls to compare against.

Sensitive data exposure

Ask for the system prompt, the tool list, and any secrets the app has seen. Check error messages, debug routes, and logs for stack traces or configuration. Then test isolation: with two accounts you control, try to reach the other account's records through retrieval, memory, or cached context. Cross-tenant leakage matters most here.

Tool and function misuse

List every tool the model can call and the arguments it controls. For each one, ask where authorization lives. If the only check is a sentence in the prompt, the tool is unprotected. Try to reach a tool with arguments outside your permissions, with another user's identifier, or with a value the UI never offers. Check whether destructive tools require confirmation and whether that confirmation is enforced server-side.

Insecure output handling

Model output is untrusted input to whatever renders it. Check markdown and HTML rendering, link and image embedding, and any place output is placed into a template, a query, a shell command, or another agent's context. A poisoned retrieved document can produce the same output as a hostile user, so test both paths.

Excessive agency

Map what the agent can do without a human in the loop: spend money, send email, change records, call internal APIs, write to memory. Check the scope of the credentials the agent runs with. Try to chain a harmless request into an action the user could not take directly. Long-lived memory is worth testing too: can one session plant context that changes a later one?

Cost and rate abuse

Measure token amplification: input that causes a much larger internal loop, repeated tool calls, or long retries. Check per-user quotas, context limits, and whether a failed request still costs money. Stay inside the agreed cap and stop when you hit it. A cost finding is a number and a reproduction, not a stress test.

Capture evidence while you test

A transcript pasted into a doc is not evidence. For every case, record the case ID, timestamp, account, endpoint, exact input, full output, request ID, model name and version, and any sampling settings you can see. Keep the raw response body, not a screenshot of it.

Run each case three times. Model behavior varies, and one sample tells you almost nothing about whether a control holds. Record how many runs reproduced the issue. If it fires once in five, say so.

Redact real user data as you capture it, not at the end. Store everything in one folder per engagement with a manifest that maps case IDs to files.

Score severity without inflating it

Score on impact and reachability, not on how clever the prompt was. Two questions do most of the work: who can trigger this, and what can the app do once it is triggered? An anonymous user reaching another tenant's data is critical. A logged-in user getting the assistant to swear is informational.

Separate model behavior from system defect. If the only thing between the model and a destructive action is a polite sentence in the system prompt, that is a missing control, not a model quirk. Write it up as the control failure.

Mark confidence. "Reproduced 5 of 5 runs against commit abc123" is a different claim from "observed once on staging."

Write the report the engineer can act on

Lead with five lines for the owner: what is broken, what it exposes, what to fix first. Then scope and method, including what you could not test.

Each finding needs an ID, a title, severity, affected component, preconditions, reproduction steps, observed behavior, expected behavior, impact, evidence references, and a suggested fix. Keep the suggested fix at the layer that failed. If authorization is missing in a tool, adding a sentence to the prompt is not a fix.

End with coverage: what you tested and cleared, and what you did not test. That section saves the next tester a week. The report template in the AI Red Teaming Prompt Playbook is built for this shape if you want a starting point.

Fix, retest, and close

Fix at the layer that failed: authorization in the tool, encoding in the renderer, filtering at retrieval, limits at the gateway. A prompt edit is tuning, not a control.

Agree pass criteria before remediation starts. "The model refuses" is not a criterion. "The tool rejects the call when the authenticated user does not own the record, verified across five runs and two accounts" is.

Retest the original cases plus close variants, on the fixed build, with the same accounts and data. Record the build hash. If a fix lands in the prompt only, expect the issue to return at the next model upgrade, and say so.

Turn the cases into a small regression suite. A dozen scripted conversations in CI catch the regressions that matter for almost nothing.

Questions people ask

How long does an LLM red team take?

For one application with a handful of tools, plan three to five days plus a day for the report. Agents with write access or several user roles take longer. Scoping and approval often take as long as the testing, so start early.

Can I test the production app?

Yes, with written permission, dedicated test accounts, a spend cap, and an agreed stop condition. Test tenants are better. If you must use production, avoid real customer records and never trigger actions that send, pay, or delete. The free prompt kit includes a short scope sheet for the approval conversation.

Do I need to test the model itself?

No. You are testing the application: its prompts, its retrieval, its tools, its rendering, and its permissions. Model benchmarks measure a different thing. What matters here is what your system does with model output, including when the output is wrong or hostile.