AI Agent Security: What Developers Need to Know

This is the description of a SQL Server view, written to keep salaries away from an AI agent:

Staff directory (hr.Employees without confidential pay data)

The view existed so the agent could never see hr.Employees or its Salary column. The permissions were right: the agent's database user was denied the table and granted the view. But the agent's prompt is built from the schema, descriptions included — so every prompt the model received named the hidden table and announced that it held confidential pay data.

No attacker found this. An integration test did, in the AI Database Agent's tool-calling version. The fix was one line in a database script.

That is AI agent security in miniature. The model wasn't jailbroken and nothing was exploited. A system that looked locked down was handing the model information it was built to withhold, through a channel nobody thought of as a channel. This guide covers the channels: what the model reads, what it can call, where its output goes, and who it acts for — with the real failures and fixes from three .NET projects.

The rule everything else follows from

The model is not a security boundary. The code around it is.

An agent is a planner that reads untrusted text and decides what to do with your credentials. You can ask it to behave in the system prompt, and you should. But a language model can be talked into almost anything, and it can also just get things wrong without anyone talking it into it. In the AI Database Agent's first version, "Mark all pending orders as shipped" became an UPDATE and "Drop the payments table" became a DROP TABLE. Both were sent to SQL Server. Nobody was attacking; the model complied.

So every control in this article answers the same question: if the model does the worst thing it possibly could here, what stops it? If the answer is "the prompt tells it not to", there is no control.

The OWASP Top 10 for LLM Applications names most of these risks, and the sections below use its vocabulary — prompt injection, excessive agency, improper output handling, sensitive information disclosure, unbounded consumption — but start from what actually broke.

1. Prompt injection: limit what a successful one can do

Everything the model reads can carry instructions:

The first row is direct prompt injection. The others are indirect: the attacker never talks to your agent, they just put text where your agent will read it. The AI Knowledge Base Chat is a clean example of the exposure — users upload PDFs, Word files and text, and the chunks retrieved from those files go straight into the prompt. Every uploaded document is model input.

You cannot filter your way out of this. Injected instructions are just text, in any language, any encoding, split across chunks. What you can do is make a successful injection boring:

• Give the agent only the permissions the task needs (section 2). An injected "delete everything" is harmless against a database user that can only SELECT.
• Check the model's actions, not its intentions (section 3). Validate the SQL it wrote, the tool arguments it chose, the HTML it produced — after the model, where injection can't reach.
• Ground answers in retrieved content, and say so. The knowledge base chat tells llama3.2 to answer only from the retrieved context, and an empty index returns a fixed message without calling the model at all. That limits what the model can be steered into saying; it doesn't stop it, which is why it's the last bullet, not the first.

The AI Database Agent's benchmark includes a prompt-injection question. Its second version declined it; its first version declined none of the six questions it should have. The difference was a better prompt, which is why the agent doesn't rely on the prompt: the logins query would target a system view that isn't in the validator's allow-list, so it's rejected before SQL Server sees it.

2. Excessive agency: least privilege for every tool

Tool calling is where agents stop being chatbots. In Part 5 of the AI Database Agent series, the model got five tools. The table that matters is the third column:

Three habits make that column possible.

One identity per job, each with the least it needs. The agent queries the business database as agent_reader, a WITHOUT LOGIN user with SELECT on what may be read and nothing else. Its own audit records go to a separate AgentOps database through a different identity that can only INSERT. The agent can't write to the data it reads, and can't read or rewrite its own audit trail.

CREATE USER agent_reader WITHOUT LOGIN WITH DEFAULT_SCHEMA = sales;
ALTER ROLE agent_readers ADD MEMBER agent_reader;   -- SELECT on allowed schemas, SHOWPLAN, nothing else

Tools never trust their arguments. A tool is ordinary code taking input from an untrusted caller. A sample_values tool has to put a table and column name into SQL, and identifiers can't be parameters — so it looks the names up in the schema the agent can see and builds the query from that metadata, never from the model's string. Unknown tables and columns come back as guidance, not as raw database errors.

The final action goes through the same gate as everything else. The agent's answer passes the same validation as before tools existed (parse, allow-list, cost estimate, least-privilege execution). So adding tools couldn't make the agent less safe than the version without them. When you add autonomy, check that it routes through the existing controls rather than around them.

Ask of every tool: if the model calls this with the worst possible arguments, as often as it likes, what happens? If the answer involves the word "hopefully", the tool has too much agency.

3. Improper output handling: model output is untrusted input to the next system

Whatever the model produces flows into something else — a database, a browser, a file system, a shell. Each of those has its own injection class, and model output deserves the same treatment as a form field from an anonymous user.

Into a database. The AI Database Agent parses every generated query with ScriptDom — Microsoft's own T-SQL parser — and walks the syntax tree with an allow-list: exactly one SELECT, no SELECT ... INTO, no OPENROWSET or linked servers, only tables the agent user can read. A regex can't do this: SELECT 1 -- ; DROP TABLE x is one harmless statement with a comment, and SELECT 1; DROP TABLE x is two. The full validator is in Preventing SQL Injection in AI-Generated SQL.

Into a browser. The Real Estate AI Chatbot renders the model's answers as Markdown. Markdown can contain raw HTML, and the model can be asked — or tricked — into emitting a script tag or an onerror handler. So the rendered HTML goes through DOMPurify before it touches the page, and the front end sends a strict Content-Security-Policy as a second line:

// The pattern: render, then sanitise, then insert. Never insert model HTML directly.
bubble.innerHTML = DOMPurify.sanitize(marked.parse(answerText));

Into files. The knowledge base chat stores uploaded documents on disk. Filenames are user input, so each one is sanitised and prefixed with a UUID before it's written.

The general rule: find every place model output leaves your process, and put the check for that destination's injection class in front of it.

4. Sensitive information disclosure: what the model sees, it can repeat

Assume the model will eventually say anything in its context window to whoever is talking to it. That makes the context window your disclosure boundary:

• Don't put secrets in the system prompt. Treat it as public. It will leak — quoted back, summarised, or extracted by a user who asks cleverly.
• Filter context by the agent's permissions, not yours. The AI Database Agent reads the schema it sends to the model while impersonating agent_reader, and drops anything that user can't select. The model is never told a hidden table exists — which is exactly the rule the view description broke.
• Hide sensitive columns structurally. A view without the Salary column keeps salaries out of both the permissions and the prompt. A column-level DENY would keep it out of the permissions only.
• Keep user content out of telemetry. In the AI Database Agent, question text is never a span attribute — it's free user input — while the generated SQL is, because operators need it to debug. The chatbot's logs record IDs, token counts and error types, never message content, passwords, tokens or API keys.
• Scan the metadata. Descriptions, comments, glossary entries and example queries all reach the prompt. The leak in the introduction came from a description, and only a test found it.

5. Authorization: the agent acts for one user, not for everyone

A multi-user agent holds many people's data. The model has no idea whose request it is serving, so authorization has to live in the data layer, where the model can't argue with it.

In the chatbot, the repositories only load conversations that belong to the signed-in user. In EF Core the pattern looks like this:

// Ownership is part of the query. Another user's conversation is indistinguishable from one that
// doesn't exist, so IDs can't be probed.
var conversation = await db.Conversations
    .SingleOrDefaultAsync(c => c.Id == conversationId && c.UserId == currentUserId, ct);
if (conversation is null)
    throw new NotFoundException("Conversation", conversationId);

Other users' data returns a 404, not a 403 — a 403 would confirm the ID exists. The rest of the stack backs this up: JWT bearer auth with validated issuer, audience and lifetime; a fallback authorization policy so every endpoint requires a user unless explicitly marked anonymous; and the same rules on the SignalR hub as on the REST endpoints, with the token accepted from the query string only on /hubs paths, where browsers can't send headers.

The same thinking applies to retrieval. A RAG agent serving several teams must filter retrieved chunks by the caller's permissions before they reach the prompt — the model can't be relied on to withhold a passage it was given.

6. Unbounded consumption: cost is an attack surface

Every agent request spends GPU time, tokens or money, and one user — or one loop — can spend a lot of it.

Two details are easy to get wrong:

• Share the budget across transports. ASP.NET Core's rate-limiting middleware doesn't cover SignalR hub invocations. The chatbot uses one limiter instance for both the REST endpoint and the hub, so switching from HTTP to WebSockets doesn't give a user a second allowance.
• Know your client's real address. Behind Nginx, every visitor arrives from the proxy's IP and shares one rate-limit bucket. The AI Database Agent trusts forwarded headers only from configured proxies, so limits apply per visitor without letting anyone spoof their address.

7. Untrusted files: validate before the model ever sees them

Agents that ingest documents inherit every classic upload vulnerability before the AI part even starts. The knowledge base chat's upload path:

• an extension allow-list (PDF, DOCX, TXT);
• a 25 MB cap enforced while the file streams to disk, not after;
• a %PDF- magic-byte check for PDFs, so a renamed file doesn't get parsed as one;
• a sanitised, UUID-prefixed filename;
• anything rejected mid-stream is deleted, not left half-written;
• every limit enforced again on the server, which doesn't trust the browser's check.

None of that is AI-specific, which is the point: an agent is a web application first.

8. Fail closed — on actions

When something goes wrong, decide in advance which way each path fails.

• Refusals. When the chatbot's model declines a request (stop_reason: "refusal"), the service retries on a fallback model, and if that declines too it returns a short notice — never the partial answer the model had started.
• Security rejections are final. The AI Database Agent feeds ordinary errors back to the model for up to two repairs. A rejected write attempt is not one of them: sending "that wasn't accepted, try again" would coach the model, and anyone steering it, around the validator one attempt at a time.
• Monitoring fails open; actions fail closed. The audit log writes through a bounded queue, so a slow or missing audit database costs records, never answers. That's acceptable for logging. It would not be for an authorization check.

9. Test the attacks, and measure them

Security that isn't tested drifts. Both projects treat attacks as test cases:

• The AI Database Agent's 32-question benchmark includes six questions that should be declined — writes, confidential data, prompt injection, data that doesn't exist — and reports unsafe SQL executed as a number for every version. The naïve version scored 2. Every version since has scored 0.
• Its integration tests run the real agent loop against a real database, which is how the view description leak was found.
• The chatbot's integration tests host the real API with a fake AI provider and cover isolation between users, request size limits, rate limiting, CORS and security headers. No test calls the real model, so the security tests are deterministic.
• Validator findings are exported as metrics, flagged when security-relevant — so a burst of rejected write attempts shows up on a dashboard.

The middle row is the trap. "Unsafe SQL executed: 0" looks like the problem is solved, and it was solved by a prompt. The last row is the same number backed by a parser and a read-only database user, which hold when the prompt doesn't.

AI agent security checklist

1. Treat the model as untrusted — its output, and everything it reads.
2. Ask "what if it does the worst thing?" for every capability. "The prompt says not to" is not an answer.
3. Least privilege per identity: a read-only user for what the agent reads, a separate insert-only identity for its audit trail.
4. Tools validate their arguments and build queries from your metadata, never from model strings.
5. Route every action through the same gate, including actions taken through tools.
6. Validate output for its destination: parse SQL, sanitise HTML, sanitise filenames.
7. Build the context from the agent's permissions, keep secrets out of the system prompt, and check metadata for leaks.
8. Enforce authorization in the data layer, returning 404 for other users' data.
9. Bound the cost: size limits, shared rate limits, concurrency caps, timeouts, row and token caps.
10. Fail closed on actions, never repair a security rejection, and test the attacks as part of the benchmark.

FAQ

What is AI agent security?

It's the practice of controlling what an AI agent can read, do and output, on the assumption that the model itself can be manipulated or simply wrong. In practice that means least-privilege tools, validation of every action and output after the model, authorization in the data layer, and limits on cost.

How do you prevent prompt injection in AI agents?

You can't reliably prevent it — injected instructions are just text, and they can arrive through user messages, retrieved documents or tool results. You limit its impact instead: give the agent only the permissions the task needs, and check its actions (the SQL, the tool calls, the HTML) after the model, where injected text can't influence the check.

What is excessive agency?

It's an agent having more capability than the task requires: tools that can write when reading is enough, credentials broader than needed, or actions taken without a check. It's what turns a successful prompt injection into real damage.

Is a good system prompt enough to secure an AI agent?

No. A system prompt improves behaviour — in the AI Database Agent it took correct refusals from 0 of 6 to 4 of 6 — but a different phrasing or a different model can undo that. Use it for a better user experience, and put the security in code and permissions.

The code behind these examples is on GitHub: AI Database Agent, Real Estate AI Chatbot and AI Knowledge Base Chat. For the SQL-specific layers in depth, read Preventing SQL Injection in AI-Generated SQL.