AI red teaming for LLM applications and agents

What AI red teaming means — and what it doesn't

AI red teaming is adversarial testing of AI systems to discover how they can be made to behave in unsafe, insecure or unintended ways. The term covers two related but distinct activities, and conflating them leads to buying the wrong thing.

• Model-level red teaming probes a foundation model itself for harmful capabilities and safety-policy failures: will it produce dangerous content, can its refusals be bypassed, does it leak training data? Model labs do this extensively before release.
• Application- and agent-level red teaming attacks the system you built around a model: the system prompt, retrieval pipeline, tools, permissions, memory and integrations. This is where most enterprise risk lives, because this is where the model touches your data and can take actions.

For most security leaders, the second is the urgent one. A perfectly aligned model wired to a refund API with no authorization check is still a refund API with no authorization check — now with a natural-language interface anyone can talk to.

A threat model for LLM applications and agents

The core insight: an LLM treats every piece of text in its context window as potential instructions. Anything the model reads — a user's message, a web page, a PDF, a support ticket, an email, a tool's output — is an input channel an attacker can use. Combine that with tools, and you have a privileged user who takes orders from strangers.

We map every engagement against four questions:

1. What can the model read? User input, retrieved documents, browsing results, emails, tool outputs, other users' content. Each is an injection channel.
2. What can the model do? Every tool, function, plugin and MCP server — with which credentials and which scope.
3. What does the model know? System prompts, retrieved data, secrets in context, conversation memory across users.
4. Who sees the output? Rendered markdown and links can exfiltrate data; output piped into other systems can become code execution.

The dangerous combinations are where those overlap: untrusted input + sensitive data + an outbound channel is a data-exfiltration path; untrusted input + a powerful tool is an unauthorized-action path.

The attack classes we test every time

We also map techniques to MITRE ATLAS so findings slot into the same threat-intelligence language your SOC uses for everything else.

How to run an AI red team engagement

Scope the system, not the model

Inventory every input channel, tool, data source and output sink. Draw the trust boundaries. The most valuable hour of an engagement is often spent discovering that the "read-only" assistant has a write-scoped token.

Define objectives an executive understands

"Bypass the safety filter" is not an objective. "Make the agent send customer PII to an external address" or "issue a refund without authorization" is. Objectives keep testing grounded in business impact.

Automate the breadth, hand-craft the depth

Prompt injection is probabilistic: an attack that fails nine times may succeed on the tenth with a different phrasing, encoding or turn order. Automated adversarial models — like Rogue/AI — generate and mutate thousands of attempts per objective. Human operators then take the near-misses and craft the multi-step, context-dependent attacks that automation hasn't seen.

Measure attack success rate, not pass/fail

Report the percentage of attempts that succeed per objective and per channel, before and after mitigations. A guardrail that reduces success from 40% to 3% is meaningful progress — but 3% across a million daily conversations is still 30,000 incidents.

Retest every release

Changing the model version, the system prompt or a single tool description can reopen a closed hole. AI red teaming that runs once a year is theatre. Wire the automated suite into CI so every release is attacked before it ships.

What actually reduces AI agent risk

Red teaming is only useful if it changes the design. The mitigations that hold up under our testing are architectural, not prompt-based:

• Least privilege for tools. Give agents the narrowest credentials possible, scoped to the current user's permissions — never a shared service account.
• Human confirmation for consequential actions. Payments, deletions, external communication and permission changes should require an explicit user confirmation that the model cannot fake.
• Separate untrusted content. Treat retrieved and tool-returned text as data. Use structured outputs and allowlisted actions instead of letting free text drive tool calls.
• Close exfiltration channels. Block rendering of model-generated links and images to arbitrary domains; restrict outbound requests from agent tools.
• Log and monitor agent actions as you would any privileged user, and send them to your SOC. Nightwatch treats agent tool calls as first-class security telemetry.
• Layered guardrails for input and output classification help — but they are a speed bump, not a wall. Assume they will be bypassed and design so that a bypass isn't catastrophic.

Building an AI red teaming program

One engagement finds holes. A program keeps them closed while your AI features keep changing. The programs we see working share a few traits.

Own it in security, build it with product

AI red teaming fails when it lives only in a responsible-AI committee or only in an ML team. Security should own the threat model, the findings process and the risk acceptance; the product and ML teams own the fixes and the release gates. Treat an AI feature like any other internet-facing system that handles sensitive data, because that is what it is.

Maintain an AI asset inventory

Most organisations cannot list every place a model is deployed, which tools each agent can call and which credentials those tools use. Start there. Shadow AI features — a team that wired an assistant to a production database during a hackathon — are a common source of critical findings.

Write an attack library you own

Keep a versioned library of the attacks that worked against your systems, including the exact phrasing, documents and multi-turn sequences. These become regression tests. When a model upgrade quietly makes an old injection work again, you'll find out in CI instead of in an incident report.

Set release gates on attack success rate

Agree thresholds per objective — for example, zero successful unauthorized tool calls across 5,000 attempts, and under 1% system-prompt extraction — and block releases that exceed them. Thresholds turn a subjective "is it safe?" debate into an engineering criterion.

Feed the SOC

Every successful attack pattern is also a detection opportunity. Send agent tool-call logs to your SIEM and write detections for the behaviours red teaming uncovered: unusual outbound requests from agent tools, bursts of refusals followed by a success, or retrieval of documents outside a user's normal scope.

AI red teaming and regulation

Adversarial testing is moving from best practice to expectation. The NIST AI Risk Management Framework and its generative AI profile call for red teaming as part of measuring risk. ISO/IEC 42001 requires organizations to assess and treat AI-specific risks. The EU AI Act requires providers of general-purpose AI models with systemic risk to perform adversarial testing, and requires robustness and cybersecurity measures for high-risk systems. Regulators in financial services increasingly ask how AI features were tested before deployment.

The practical implication: keep evidence. Every BadAI engagement produces a findings set mapped to OWASP and ATLAS, attack-success metrics over time, and a record of mitigations and retests — the artefacts auditors and regulators ask for.

Frequently asked questions