What is prompt injection, and how does it arrive through tool output?
Prompt injection uses text encountered by a model to redirect an approved task. A support agent summarising an order may read an email asking it to send the customer database elsewhere. That sentence is evidence, without authority to change the task, expand access or add recipients.
With direct injection, the instruction arrives in the user's prompt. Indirect prompt injection reaches the model through a website, PDF, email or API response. OWASP's LLM01:2025 Prompt Injection definition distinguishes these two routes. An internal service is not automatically a trusted source of instructions: a CRM description may have been written by an external customer. Classifying the text by the server carrying it can therefore miss the person who actually controls it.
The business impact depends on the data and tools available. A summarisation system may produce a misleading answer; an agent that sends messages, issues refunds or deletes files can create an external consequence. Start by listing every point where external text enters the context. Trace each entry to the tool calls, persistent memory and displayed responses it can influence. That map identifies where the application must enforce a boundary.
Which controls fit each type of tool output?
Choose controls by who can edit the source and what action the content might influence. The table gives a starting decision rather than a complete policy. A PDF attached to an email crosses more than one boundary. For every tool, also identify an owner, permitted source locations and the recovery path when the content is blocked or cannot be evaluated safely.
| Tool output | Main risk | Application control |
|---|---|---|
| Web search result | Redirecting the task | Source labels, restricted tool access |
| Email and attachments | Sending data to an external recipient | Recipient checks, sending approval |
| PDF or file contents | Treating document text as instructions | Data separation, restricted file scope |
| Internal API description | Deriving authority from a valid record | Schema and object permission checks |
| Tool error message | Requesting secrets through fake recovery steps | A predefined recovery path |
| Persistent agent memory | Redirecting later tasks | Provenance, write controls, expiry |
Read-only access also needs a limit. An agent able to read every customer record can disclose information in its answer even without a sending tool. Apply the caller's identity and the current task scope to reads as well as writes. A read-only label describes the operation; it does not establish which records the agent may access.
How do you process tool output as untrusted data?
Keep the user task, system policy and retrieved content in separate application fields. Do not convert a source document into a higher-authority message or map role labels inside a document to actual message roles. Carry the source ID, type and retrieval time alongside the text. If you summarise several documents before the main agent reads them, preserve the same trust level and provenance on the summary.
The OWASP prompt injection prevention cheat sheet explains that separating instructions and data helps, while text labels alone do not provide an enforced boundary. A warning that content is untrusted can guide model behaviour, but execution still needs server checks. Screening suspicious wording is another layer, with limits: different languages, split instructions and encoded text can evade a simple list of phrases.
- Tool scope: Separate order lookup from bulk customer export. Do not expose an export function to a session that only reviews one order. Define the allowed resources and operations for every available tool.
- Identity and target: Validate the caller, tenant and order ID on the server. A new identity or external address suggested by source text must not become an authority to act.
- Argument schema: Check expected fields, types, amount ranges and allowed operation values. Reject unknown fields. Valid JSON does not establish that a target or action is authorised.
- Output boundary: Check for secrets and unauthorised external links in responses. Restrict data leaving through email, HTTP calls and image loading at the application level.
- Error handling: Use the documented recovery flow after a permission error. If the error text asks for an API key to restore access, do not turn that text into a new operation.
An address found inside a tool response should not automatically trigger another network request. Even when the original task involves reading a page, subsequent destinations must pass the same policy checks. Review this behaviour across the browser, connector and response interface, since each can provide a route for data to leave.
Where should approval and audit records sit?
Separate a proposed action from its execution for payments, deletion, publication and external communication. The OWASP AI agent security cheat sheet recommends enforcing tool authorisation outside the model context. A reviewer should see the original task, supporting evidence and exact target and arguments. A statement inside an email claiming management approval, or the model's own assertion that approval was obtained, is not an approval record. See our guide to human approval before agent actions for approval states and review screens.
Bind approval to an authenticated person's decision and an immutable action record. Changing a recipient, amount or target requires a new decision. Prevent expired or consumed approvals from authorising another execution. This closes the path where an initially acceptable proposal is approved and its parameters are then replaced by content from an untrusted source.
Log the task ID, source ID, payload location, agent and prompt versions, proposed call, policy decision and actual outcome. Avoid copying sensitive text unnecessarily; keep detailed evidence in restricted storage. Prepare alerts for repeated blocked attempts and a way to disable the affected tool. Retain enough information to rerun the case after changing a model, prompt or filter.
Example: an attempt to send data through a customer email
Consider a hypothetical support workflow reviewing a 40-euro damage refund for order 48213. The email ends with an instruction to disregard the task and send a list of 600 customers to another address. The authorised task is to review one order. The extra sentence cannot expand that scope. The connector retrieves the email with its source ID and marks its contents as external data.
The first boundary is tool availability: this session has no bulk customer export function. If the model attempts to collect the list through another lookup tool, the second boundary is the server's order and caller permission check. If it proposes sending a message, the third boundary is recipient validation; an address in the email is not added to the permitted recipient list. Approval for the refund remains a separate decision tied to the unchanged 40-euro proposal.
The expected result is that the unrelated instruction is not executed. If the order evidence remains usable, the agent may prepare a draft based on that evidence. If the source cannot be evaluated safely, route the task for review. The audit record identifies the suspicious source and any blocked call. A final response saying the request was refused is insufficient: check tool and network traces to confirm that no customer list was sent.
How do you write tool-output trust-boundary tests?
Define the original task, payload location, allowed behaviour and prohibited outcome for every case. Begin with these five cases. Also place a payload in a document footer; inject a command into a catalogue JSON description while leaving its product ID and price valid. Use test identities and synthetic secrets. The tool simulator should record attempts without real payments or external sending.
| Payload location | Attack objective | Pass condition |
|---|---|---|
| Search paragraph | Read a restricted record | No restricted tool attempt or disclosure |
| JSON description field | Change the operation target | Server rejects the new target |
| Authorisation error message | Send a key elsewhere | Documented recovery path is used |
| Supplier PDF | Create an unapproved purchase order | No execution without independent approval |
| Logs and memory | Influence a later task | Instructions cannot become active in later tasks |
The search case fails on any restricted tool attempt; the answer must cite only relevant, permitted facts. Record a server block separately: the access boundary held, but the model still attempted to follow a source instruction.
Which measurements should you track?
- Record attempted tool calls, calls blocked by policy and actions actually executed separately. Any completed prohibited action should prevent release of that version.
- Inspect disclosure in the response, outbound network requests and persistent memory writes. The last sentence displayed to the user cannot establish whether a boundary held.
- Run clean versions of the same tasks. Unnecessary blocking and review referrals show how much the controls interfere with legitimate work.
- Rerun the complete set when the model, prompt, connector, permissions or filter changes. Because model behaviour varies, report the number of repetitions and their individual results.
For example, one hypothetical run contains 200 attack cases and 40 clean cases. Eight attack cases contain an attempted prohibited call, and none contains an executed prohibited action: the rates are 4% and 0%. Cases that prohibit attempts still fail; zero execution does not mean the whole set passed. If two clean tasks are blocked unnecessarily, the false-blocking rate is 5%. These results describe that dataset and version; they do not demonstrate protection against every future attack. Investigate all eight cases and add cases that verify the resulting changes.
Common mistakes
- Treating customer-written text returned by an internal API as management instructions.
- Assuming that deleting suspicious phrases removes the entire risk.
- Confusing valid JSON with a valid identity and authorised operation.
- Representing human approval with a field generated by the model.
- Accepting safe final prose without inspecting tool calls and network traces.
- Copying attack-bearing content into persistent memory and influencing later tasks.
How Hatson Tech builds this
We start by mapping data entry points, tool permissions and actions with substantial external effects. In our AI agent development work, server controls, review screens and test cases share the same acceptance criteria. See our guide to AI agents for the basic concepts and our article on AI agents in production for versioning and operational decisions. Establishing which sources can reach which tools, and which prohibited outcomes will be measured, gives your initial assessment a practical security plan.