Guardrails: what to check going in and coming out

A guardrail is code on the request path, not a sentence in the prompt. Going in, you check what reaches the provider and whether untrusted content is being treated as instruction. Coming out, you check that the answer is supported, in scope, and free of anything that should not leave. Every one of these is a classifier with an error rate, and the false positive rate is the number teams never measure.

“We told it not to do that” describes an intention. A guardrail is what happens when the intention is not enough.

Key takeaways

• Anything you can enforce in code should not be enforced in the prompt.
• The input path and the output path need different checks, and most systems build only one.
• Content is data. The moment retrieved or forwarded text can issue instructions, the boundary is gone.
• Every guardrail blocks legitimate traffic sometimes. If you have not measured how often, you are running an unmeasured refusal rate.
• Firing is not the same as refusing. Redact, escalate or caveat are usually better responses than a block.

Who this applies to

You are building or reviewing a system that handles content you do not control - customer messages, forwarded email, retrieved documents, web pages - or that produces content going somewhere it cannot be retracted from.

Two adjacent questions are covered elsewhere. What an agent is permitted to do is tool-calling and agent permissions. Redacting sensitive data before it reaches your logs is one of the week-one decisions in responsible AI as an engineering decision. This article is about the content on the live request path in both directions.

The prompt is not a control

Instructions in a prompt are followed most of the time. That is a useful property and it is not a security property.

The distinction that matters when scoping: a rule in the prompt is a request to the model, and a rule in code is a constraint on the system. If the consequence of the rule being ignored is a refund, a complaint or a disclosure, it belongs in code. If the consequence is a slightly worse answer, the prompt is fine.

This is the same argument as enforcing tool permissions in the tool rather than the instructions, and it generalises: the model is a component that can be influenced, so the controls have to sit outside it.

Going in: four checks before the model sees it

Sensitive data. Decide what must not reach the provider at all - card numbers, national identifiers, health details - and detect and mask it before the call. Doing this at the boundary rather than after the fact is what keeps it out of the provider’s request logs as well as yours.

Untrusted content, marked as untrusted. Anything you did not author is data: a retrieved document, a forwarded email, a scraped page, a support attachment. It goes into the request as content to be read, never as instruction to be followed, and the system prompt should state that explicitly while the architecture assumes the statement will not always hold.

Scope. A support assistant asked for medical or legal advice should route rather than answer. Topic rules are absolute and belong in code, because a probabilistic gate is the wrong instrument for a category that is never acceptable.

Size and rate. A hundred-page attachment pasted into a chat box is a cost and latency incident, and repeated identical requests are usually either a bug or abuse. Both are cheap to bound and expensive to discover later on an invoice.

Prompt injection, in the terms that help

The attack is that untrusted content carries instructions and the model follows them. The defence is not a filter that recognises malicious text, because that is a classifier and classifiers have gaps. The defence is that following the instruction should not be able to achieve anything.

Three properties do the work:

Content never grants authority. No text arriving from outside can widen what the system is allowed to do. Permissions come from the session, not from the request body.

Every tool argument is validated against a source of truth. If the model asks to refund order 4471, the system checks that this customer owns that order and that its amount matches, before anything happens.

Reading and deciding are separated. The step that summarises an untrusted document is not the step that holds the credentials.

With those three, an injected instruction produces a wrong sentence rather than an unauthorised action. Without them, no amount of filtering closes the gap. Detection is still worth adding as a second layer, and it is a second layer rather than the answer.

Coming out: four checks before the user sees it

Is it supported? For anything answered from your documents, check that the claims appear in what was retrieved. This is the single highest-value output check, and it is the same measurement as an acceptable hallucination rate applied per request instead of per month.

Is anything leaving that should not? Another customer’s details, an internal document title, a system prompt, an API key that appeared in a retrieved log file. Retrieval permissions are the primary control here - see RAG permissions - and an output check is the backstop for when the index is wrong.

Is it in scope and in policy? Prices it invented, commitments it has no authority to make, competitor comparisons, guarantees. These are business rules, and they are cheap to check for and expensive to publish.

Is it the right shape? If the output feeds a system rather than a person, validate against a schema and fail loudly - covered in structured output.

Every guardrail is a classifier

This is the part vendors leave out. A guardrail makes a judgement, and judgements have two error types.

False negatives are what everyone worries about: the thing you built it to catch, getting through.

False positives are what everyone experiences: a legitimate customer blocked, a real question refused, a valid answer suppressed. They are invisible to you and highly visible to the person they happen to. A support assistant that refuses on the word “chest” because a health rule was written broadly is not a safe system, it is a broken one with a good excuse.

So a guardrail needs its own evaluation set, with both halves in it: cases it must catch, and ordinary traffic it must let through. Run both when the rule changes. The set is built the same way as any other, described in an eval set from real tickets, and the second half is the one that gets skipped.

The other cost is latency. A check that calls a model adds a model call, and if it runs before anything streams, the user waits for it - which is a line item in a latency budget, not a rounding error. Cheap deterministic checks first, model-based checks only where they earn it, and run in parallel with generation where the design allows.

What to do when one fires

Blocking is one option out of four, and it is the one most systems use for everything.

Choosing between these is a business decision rather than an engineering one, and it is worth making explicitly per category. A system that only knows how to refuse will refuse things it should have escalated, and the customer experiences both as a dead end. What a good handover looks like is in confidence thresholds and escalation design.

Whatever fires should be logged with the rule that fired and the input that triggered it, so that the false positive rate is measurable at all rather than being a thing people mention in meetings.

What we build in by default

Four things go into every build without a line item, because each is close to free at the start and none is separable later: masking sensitive fields before the provider call, treating all retrieved and forwarded content as data, validating every tool argument against the source of truth, and logging which rule fired on which request.

The one we argue about is the support check on the way out. It costs a second call on answers grounded in documents, and clients reasonably ask whether it is worth the latency and the spend. The answer we give is that it converts the failure mode from a confident wrong answer into a caveat or a handover, and that a wrong answer about a policy generates a complaint, a correction and a support contact - so the check is usually cheaper than the incidents it prevents. Where volume makes that arithmetic fail, sample it rather than dropping it.

Where we have been wrong: we have written topic rules too broadly and blocked ordinary questions, more than once. The pattern was always the same - the rule was written from the risk, tested against the risk, and never tested against a week of normal traffic. Now nothing ships as a rule until it has been run over real historical messages and the false positive count is a number somebody has seen.

When this is not worth it

Internal tools with trusted users and no external content. A drafting assistant used by six people over your own documents does not need an injection boundary, because there is no untrusted input path.

When nothing irreversible is downstream. If the output is a suggestion a person reads and edits, the person is the guardrail. Adding a filtering layer in front of a human reviewer usually costs more than it returns.

Before you know what you are protecting against. A guardrail with no named failure it prevents is a latency cost and a false positive rate, bought speculatively. Name the incident first, then decide whether code, a prompt or a process is the cheapest place to prevent it.

Frequently asked questions

Can a model check its own output?

Reasonably well for support and policy checks, and less well for correctness, because the same reasoning that produced the answer produces the assessment. Use a separate call with a narrow question rather than asking the answering model to review itself in the same breath, and hold the check to the same evidence standard as any other measurement.

Do we need a specialist guardrails product?

For redaction and common policy categories a library or service saves real time. The parts that matter most in practice - the trust boundary, argument validation, and what happens when a rule fires - are system design and are not something a product can supply.

Where should redaction happen, exactly?

Before the provider call for anything that must not leave your systems, and again at logging, since the two boundaries are different. If you only do one, do it at the earlier point, because logs can be cleaned up afterwards and a request that has already been sent cannot.

Will guardrails make the assistant feel restrictive?

Only if refusal is the only response you implemented. Redaction, escalation and caveated answers cover most of what would otherwise be a block, and the difference in how the system feels is almost entirely about which of the four it reaches for.

How do we test the input side without writing attacks?

Use your own historical traffic plus published test collections for injection and safety categories. The measurement that matters most is the one on ordinary traffic, and you already have twelve months of it.

Next step

If your system reads content you did not author, or writes anywhere a mistake cannot be withdrawn, the controls belong in the build rather than in a later hardening pass. Custom AI development builds these in from the first version, and they are close to free there.

Related: Tool-calling agents: what to let them touch · RAG permissions · Structured output: reliable JSON · Responsible AI as an engineering decision · Custom AI development