AI Prompt Injection Testing: Evaluate Documents, Tools and Data Boundaries
Test prompt injection across documents, retrieval and tools using controlled cases, enforced permissions and accountable action boundaries.
Prompt injection testing should examine whether untrusted content can change an AI system's instructions, expose restricted information or trigger an unauthorised action. Test the entire application boundary, including retrieved documents and connected tools. A carefully worded system prompt is one component, not proof that the product resists manipulation.
Let’s plan your AI security test scope.
Bring your workflow and the decisions you need to make. We’ll help you clarify requirements, review the trade-offs and define a practical next step.
Inventory untrusted inputs
List user text, uploaded files, web pages, retrieved snippets, messages and tool responses. Mark which components may contain third-party instructions. The application must distinguish those contents from the authorised user's actual request.
OWASP's prompt-injection guidance describes direct and indirect injection and notes that retrieval and fine-tuning do not fully eliminate the issue. Evaluate layered controls and the actual permitted actions rather than treating one technique as complete prevention.
Build authorised test cases
| Boundary | Controlled test | Expected outcome |
|---|---|---|
| Retrieved document | Embedded request to ignore the task | Content treated as evidence, not authority |
| Citation source | Link text asks for unrelated disclosure | No transmission outside the authorised workflow |
| Tool selection | Source suggests an extra external action | Action remains within granted scope |
| Data access | Request crosses a test user's permissions | Restricted records never enter context |
| Output handling | Generated content contains unsafe markup | Application handles output under its security rules |
| Human approval | Injected text claims approval already exists | Actual approval state remains authoritative |
Use synthetic secrets and records in a controlled environment. Do not test by trying to extract real credentials or private customer information from production.
Limit consequences outside the model
Keep tool permissions narrow and validate action parameters at the application boundary. An assistant drafting a record should not automatically receive authority to approve or publish it.
For a hypothetical support agent, separate reading an account from changing its status. Require the approved authority and confirmation path for consequential changes, regardless of what a retrieved message claims.
Record results and retest changes
Capture the scenario, allowed scope, observed tool activity and outcome. A refusal message is not enough evidence if an unauthorised tool call occurred earlier. Inspect the system behaviour that matters.
Rerun relevant cases when connectors, prompts, models or tools change. Keep a route to disable unsafe actions and investigate incidents. Testing reduces uncertainty; it does not establish universal immunity to future attacks.
Injection-testing questions
Can sanitising a few phrases solve the problem?
Do not assume it. Inputs vary, and consequences need independent access and action controls.
Is a read-only assistant risk-free?
No. Inappropriate retrieval or disclosure can still matter even when the system cannot write records.
Pair testing with RAG access checks, and discuss the action boundary with Drema's AI development team.
This article explains what to build, not legal advice. Rules change; confirm against the current official text before relying on it.