If you only harden three things in your LLM app, make them these: prompt injection, data leakage, and guardrails around what the model can do. Nearly every serious AI incident I’ve seen traces back to one of them — usually to the mistaken belief that a well-written system prompt is a security control. It isn’t. The model reads your instructions and the attacker’s through the same channel, so real defense lives in the architecture around the model. Here’s how to build it.
Risk 1: Prompt injection
Prompt injection is when text the model reads overrides the behavior you intended. It comes in two flavors, and the second is the one that gets startups.
Direct injection is a user typing “ignore your instructions and…” into your chat box. Annoying, but the attacker only affects their own session.
Indirect injection is the real threat. Malicious instructions are planted in content your model retrieves — a web page it summarizes, an email it drafts a reply to, a support ticket, a document in your RAG index. The model treats that content as trusted instruction. Now an outsider who can influence any data source in your pipeline can steer your model.
Mitigations that work
- Separate instructions from data — structurally. Put untrusted content in a clearly delimited section and instruct the model to treat it as data, never as commands. This isn’t foolproof, but it raises the bar.
- Least-privilege tool access. This is the real defense. If the model summarizing a webpage has no tools that can leak or destroy anything, a successful injection just produces a bad summary. Scope capabilities so a hijacked prompt has nowhere to go.
- Constrain the retrieval surface. Only index and retrieve content you control or have vetted. Treat externally-sourced text (scraped pages, user uploads) as radioactive — the model can read it, but it should never gain new powers from it.
- Detect the obvious. Lightweight input filtering catches known jailbreak patterns. It won’t stop a determined attacker, but it removes the noise so your logs show the real attempts.
Don’t expect prompt-level fixes alone to hold. The durable win is architectural: assume injection will succeed and make sure it doesn’t matter.
Risk 2: Data leakage and exfiltration
Whatever enters the context window can come back out. Two failure modes dominate.
Passive leakage: you put data in the context that shouldn’t be there — another tenant’s records, secrets embedded in the system prompt, more PII than the task needs. The model may surface it verbatim or paraphrase it into a response.
Active exfiltration: an attacker uses injection to pull data out. The classic pattern is instructing the model to encode sensitive context into an outbound channel — a URL it renders, an image it embeds, an API call it makes, an email it sends. The data walks out through a capability you handed the model.
Mitigations that work
- Minimize the context. Retrieve only what the current task and current user require. If it doesn’t need to be in the window, keep it out.
- Tenant-isolate retrieval. Enforce data boundaries at the query layer, not in the prompt. A user’s request should physically be unable to fetch another tenant’s rows. Filter by identity in the retrieval call — never rely on the model to “remember” whose data it’s allowed to see.
- Keep secrets out of prompts entirely. API keys, credentials, and internal tokens never belong in a system prompt or context window. Fetch them server-side, use them server-side, and never let them cross the model boundary.
- Control the outbound channels. If the model can produce links, make outbound requests, or trigger sends, those are exfiltration paths. Allowlist destinations, strip or sanitize model-generated URLs, and don’t auto-fetch links the model emits.
Risk 3: Weak guardrails and output handling
The most under-appreciated risk: teams sanitize input and then trust the output completely. LLM output is untrusted input to whatever consumes it.
- Rendered as HTML? You have stored XSS.
- Passed to a shell or
eval? Remote code execution. - Interpolated into a SQL query? Injection.
- Handed to another agent? You’ve chained the vulnerability.
Guardrail patterns
- Validate output at the boundary. Before model output is rendered, executed, or forwarded, treat it as hostile. Escape HTML, parameterize queries, and never pass raw output to an interpreter.
- Structured output over free text. When the model drives an action, have it return a constrained schema (a specific JSON shape, an enum of allowed operations) rather than free-form text you parse loosely. Reject anything that doesn’t validate.
- Allowlist actions, don’t blocklist them. Define the exact set of operations the model may trigger and refuse everything else. Blocklists always miss a case.
- Rate-limit and budget-cap. Bound tokens, tool calls, and loop iterations. This stops runaway agents and denial-of-wallet attacks where an adversary drives your API bill through the roof.
- Human-in-the-loop for high-risk actions. Anything irreversible or costly — deleting data, moving money, sending external communications, changing permissions — gets an explicit human approval step. You can relax this later once telemetry proves the flow is safe; you can’t un-send a wire.
How do you test an LLM app for these vulnerabilities?
Hardening is only half the job — you have to prove the guardrails hold. LLM security testing looks different from traditional QA because the system is non-deterministic, so you test for properties rather than fixed outputs.
- Red-team the injection surface. For every source of retrieved text, plant a benign “injection” — an instruction telling the model to reveal its system prompt, call a tool it shouldn’t, or emit a marker string. If the marker ever shows up in output or a tool call, you have a live indirect-injection path.
- Probe tenant isolation. Authenticate as tenant A and ask the model, in a dozen phrasings, for tenant B’s data. A correct system makes this physically impossible at the query layer, so it should fail every time regardless of how clever the prompt is.
- Fuzz the output boundary. Feed the model inputs designed to make it emit HTML, shell metacharacters, SQL fragments, and oversized payloads, then confirm your consumers escape, parameterize, or reject them.
- Test the exfiltration channels. Try to make the model encode context into a URL, an image tag, or an outbound API call. If your allowlist and URL sanitization work, the data never leaves.
- Automate what you can into CI. Bundle your highest-value adversarial prompts into a regression suite. You won’t catch every jailbreak, but you’ll catch the ones you already fixed from silently regressing after a prompt or model change.
The goal isn’t a green checkmark that says “secure.” It’s confidence that when injection succeeds — and it eventually will — your architecture contains it.
LLM security guardrails checklist
If you want a fast self-audit, this is the short list I run against any LLM feature before it ships:
- Retrieval is scoped and tenant-isolated at the query layer, not the prompt.
- Tools follow least privilege — read-only where possible, per-user and per-resource where not.
- No secrets in the context window — keys and tokens stay server-side.
- Output is validated as hostile — escaped, parameterized, and schema-constrained before it drives an action.
- Actions are allowlisted, not blocklisted.
- Outbound channels are controlled — destinations allowlisted, model-generated URLs sanitized, no auto-fetch.
- Rate limits and budget caps bound tokens, tool calls, and loop iterations.
- High-risk actions require human approval until telemetry earns the right to remove it.
- Everything is logged — prompts, tool calls, and outputs, immutable and queryable.
Anything you can’t check off is a gap worth closing before your next release.
Putting it together
A defensible LLM app follows one principle end to end: constrain the model’s inputs, capabilities, and outputs so that a compromised prompt can’t cause real harm. Concretely, that means scoped retrieval, least-privilege tools, no secrets in context, validated structured output, and human approval on the actions that would actually hurt. None of it requires ML expertise — it’s disciplined engineering applied to a component that happens to blur data and code.
Build these three defenses and you’ve covered the overwhelming majority of real-world AI risk. Skip them, and no system prompt will save you.
Frequently asked questions
What is the difference between direct and indirect prompt injection? Direct injection is a user typing adversarial instructions into your chat box — the damage is limited to their own session. Indirect injection hides malicious instructions in content the model retrieves, like a web page, email, or document in your RAG index. Indirect injection is the more dangerous of the two because any outsider who can influence a data source in your pipeline can steer your model.
How do you prevent data leakage in LLM applications? Minimize what enters the context window, enforce tenant isolation at the query layer so a request physically cannot fetch another tenant’s data, keep secrets out of prompts entirely, and control outbound channels by allowlisting destinations and sanitizing model-generated URLs. Leakage happens when sensitive data is either in the context needlessly or can be pushed out through a capability you handed the model.
Are LLM guardrails enough to secure an AI app? Guardrails around output and actions are necessary but not sufficient on their own. Real security comes from constraining inputs, capabilities, and outputs together — scoped retrieval, least-privilege tools, no secrets in context, validated structured output, and human approval on high-risk actions — so that a compromised prompt can’t cause real harm.
Should LLM output be treated as trusted? Never. LLM output is untrusted input to whatever consumes it. Rendered as HTML it’s stored XSS, passed to a shell it’s remote code execution, interpolated into SQL it’s injection. Validate every token the model produces at the boundary before it’s rendered, executed, or forwarded.
Do these mitigations require machine learning expertise? No. Scoped retrieval, least-privilege tools, output validation, and human-in-the-loop approval are disciplined engineering, not ML research. They’re the same security fundamentals you’d apply anywhere, adapted to a component that blurs the line between data and code.
If you’d like a second set of eyes on how your LLM features handle injection, data boundaries, and output, I run a fixed-scope AI Security Review that turns these principles into a prioritized fix list for your codebase.