AI Agents in Production: The Guardrails That Keep Them Safe
How to run AI agents in production: scoped tools, confidence thresholds, human approval, evals, audit logs and cost control, with lessons from live systems.
To put AI agents into production safely, limit what they can do and measure what they actually do. Give each agent a short list of typed, permission-checked tools, route low-confidence outputs to people, require human approval for actions that move money or change records, test every change against an evaluation set, log every step and cap cost per task.
Demos of AI agents are easy. Agents that run every day against real systems, without surprising anyone, take more engineering. This article covers the guardrails we use, in the order we would add them, with short examples. It applies equally to tool-using agents and to simpler LLM workflows where a model handles one or two steps.
What is the difference between an AI agent and an LLM workflow?
An LLM workflow is a fixed sequence of steps where a model does part of the work, such as extracting fields from an invoice or drafting an email. The code decides what happens next.
An AI agent is a loop in which the model decides which tool to call next, looks at the result and continues until the task is done or it needs help. The model decides the path.
| LLM workflow | AI agent | |
|---|---|---|
| Who decides the next step | Your code | The model |
| Predictability | High | Lower |
| Cost per run | Easier to bound | Varies with the path taken |
| Testing | Step-level tests | Outcome and trajectory evaluation |
| Good for | Repeated processes with known steps | Tasks that need judgment across several systems |
Our default advice: if the steps are known, build a workflow. Use an agent when the task genuinely needs the model to choose what to do. Most of the guardrails below apply to both.
Why do AI agents fail in production?
The failures we see most are not exotic:
- Too many tools and too much access. An agent with a general database tool or broad API key can do far more than the task needs.
- No idea when it is wrong. The model returns a confident answer either way, and nothing checks it.
- Untrusted input treated as instructions. Emails, documents and web pages can contain text that tries to redirect the agent (prompt injection).
- No evaluation. A prompt or model change makes things worse and nobody notices for weeks.
- No trace. When something goes wrong, nobody can reconstruct what the agent saw and did.
- Runaway cost. Retries, long contexts and loops multiply tokens.
Each guardrail below addresses one of these.
1. Scope the tools, not just the prompt
The prompt is not a security boundary. The tools are. Give the agent the narrowest tools that do the job, with typed inputs and server-side checks:
{
"name": "create_refund_request",
"description": "Create a refund request for a single order. Does not issue the refund.",
"input_schema": {
"type": "object",
"properties": {
"order_id": { "type": "string", "pattern": "^ORD-[0-9]{8}$" },
"amount": { "type": "number", "exclusiveMinimum": 0 },
"reason": { "type": "string", "maxLength": 500 }
},
"required": ["order_id", "amount", "reason"]
}
}
Notice what this tool does not do: it requests a refund rather than issuing one, and it works on one order at a time. On the server side:
- Authenticate the tool call as the agent’s service identity, with its own least-privilege credentials.
- Check business limits in code (maximum amount, order belongs to this customer, not already refunded).
- Separate read tools from write tools, and start pilots with read-only or approval-gated writes.
- Use allow-lists for anything that reaches outside: domains, email recipients, file paths.
The same applies if you expose tools through the Model Context Protocol (MCP): every server needs authentication and permission checks of its own.
2. Route on confidence, not on hope
Wherever a model extracts, classifies or scores, get a confidence signal and act on it. That might be a calibrated probability from a classifier, the confidence score from a document AI service, agreement between two methods, or validation rules that check the output against known facts.
def route(result):
if not result.passes_validation:
return "reject"
if result.confidence >= AUTO_APPROVE_THRESHOLD:
return "auto_approve"
if result.confidence >= REVIEW_THRESHOLD:
return "human_review"
return "reject"
Set thresholds from evaluation data, not intuition, and review them as you collect more cases. Validation rules matter as much as the score: a tax ID that fails its checksum should never be auto-approved, however confident the model is.
3. Put people in front of consequential actions
Decide in advance which actions an agent may never take alone. Typical examples: payments and refunds above a limit, changes to contracts or customer records, external emails to new recipients, and anything that deletes data.
Good human-in-the-loop design is more than an “approve” button:
- Show the reviewer the evidence: the source document, extracted values, the tool calls planned and why.
- Make approve, edit and reject equally easy, and record which one was chosen.
- Feed corrections back into your evaluation set.
- Measure the review rate. If people approve everything without looking, the step is not working.
4. Treat all external content as data
Anything the agent reads from outside, including emails, PDFs, web pages, tickets and tool results, may contain instructions written by someone else. Defenses that work in practice:
- Keep untrusted content clearly separated from instructions in the prompt, and tell the model it is data.
- Never let content alone trigger a write tool. Writes depend on the task definition and on checks in code.
- Limit what can leave the system: allow-listed recipients, no arbitrary URLs, no raw data in outbound messages.
- Review your design against the OWASP Top 10 for LLM Applications.
5. Build evaluations before you scale
An evaluation set is a collection of real cases with known good outcomes. Run it on every change to prompts, tools, models or thresholds.
- Start small. A few dozen real, labeled cases are a useful first set; grow it from production runs. Include the awkward ones.
- Score outcomes, such as correct fields extracted or correct record updated, and trajectories, such as whether the agent called the right tools in a sensible order without unnecessary steps.
- Use deterministic checks first and an LLM grader only where judgment is needed, and spot-check the grader.
- Gate releases on evaluation results in CI, the same way unit tests gate code.
- Compare models on your data before switching. A provider abstraction makes this cheap.
6. Log everything you would need in an incident
For every run, record the input, the model and prompt version, every tool call with its arguments and results, confidence scores, the routing decision, any human approval and the final action. Store it with a run ID you can search.
These logs serve three audiences: engineers debugging failures, auditors asking why a decision was made, and your evaluation process, which draws new test cases from real runs. Mask personal data in logs where the full value is not needed.
7. Control cost per task
Agents spend money in proportion to the tokens they use and the steps they take. Practical controls:
- Measure cost per completed task, not just per call. Count tokens on every request.
- Set hard limits: maximum steps per run, maximum tokens per run and daily caps per workflow.
- Use the smallest model that passes your evaluation for each step, and a larger one only where needed.
- Trim context. Send the fields the model needs, not whole documents or full histories.
- Cache stable results such as classifications of identical inputs.
- Fail closed. When a limit is hit, stop and hand the task to a person rather than retrying indefinitely.
A production readiness checklist
- Each tool is typed, scoped and permission-checked on the server
- Read and write tools are separated; writes start approval-gated
- Confidence thresholds and validation rules are set from evaluation data
- Consequential actions require human approval, with evidence shown
- External content is treated as data, with outbound allow-lists
- An evaluation set runs on every change and gates releases
- Every run is traced and logged with a searchable ID
- Step, token and daily limits are enforced, with cost per task tracked
- Feature flags let you switch behavior off without a redeploy
- An owner reviews failures and review rates every week
From our projects
These guardrails come from systems we run in production:
- FinAIQ extracts data from invoices and receipts with Google Document AI. Confidence scores drive auto-approve, review and reject thresholds, GSTIN, PAN and HSN rules validate the output, SHA-256 fingerprints catch duplicates, and audit history covers 79 data entities before anything posts to the ledger.
- Exelynt starts an AI code review from a signature-verified GitHub webhook and returns weighted scores with specific feedback. Its provider abstraction supports DeepSeek, OpenAI GPT-4 and Anthropic Claude, so models can be compared and switched.
- Sayso drafts survey questions from source documents with an LLM, tracks model confidence per question and keeps a human approval step before anything reaches panelists.
- Our own sales automation runs as n8n workflows with deduplication, email verification, daily send caps and a reply check before every follow-up, and humans review anything the AI is unsure about.
- For a hotel-distribution business, an LLM ticket assistant and AI pull-request reviewer run behind feature toggles, CloudWatch metrics and token counting. See the AI ticket triage use case.
For more on how we design agents, see our AI agent development service.
How much does a production AI agent cost?
We do not quote prices without understanding the task. The main cost drivers are:
- The number of tools and systems the agent must use, and their API quality.
- How much autonomy it needs, and so how much approval and evaluation work.
- Compliance and data-residency requirements, which affect model and hosting choices.
- Task volume, which drives model usage costs.
- Whether a deterministic workflow could do the job instead, which is usually cheaper.
A scoped pilot on one real task gives you performance data and a reliable estimate for the rest.
If you are planning an agent or LLM workflow and want a second pair of eyes on the guardrails, we offer a free 30-minute review with one of our engineers. Tell us about the process you want to automate.
FAQ
What are guardrails for AI agents?
Guardrails are the controls around a model that limit what it can do and check what it produces: scoped tools, input and output validation, confidence thresholds, human approval, spending limits, logging and evaluation.
How do you test an AI agent before production?
Build an evaluation set from real cases with known good outcomes, score both outcomes and tool use, run it on every change, and pilot the agent with read-only or approval-gated tools alongside your team.
When should an AI agent require human approval?
For any action that is hard to reverse or has real consequences: moving money, changing contracts or customer records, sending external messages to new recipients, or deleting data. Start strict and relax limits only when evaluation results support it.
How do you stop prompt injection in AI agents?
You cannot fully prevent it in the model, so limit its impact. Treat external content as data, never let content alone trigger write actions, enforce permissions in tool code and restrict what can be sent outside.
How do you control LLM costs in production?
Track cost per completed task, set step and token limits per run and daily caps per workflow, use the smallest model that passes evaluation for each step, trim context and cache repeated results.