A support assistant needs to look up an order. Its connector can also change the address, cancel the order, and issue a refund. The demo only exercises lookup, so the extra capabilities never enter the acceptance criteria.
They have already entered the product's risk profile.
OWASP describes excessive agency in terms of excessive functionality, permissions, and autonomy. A tool can expose unnecessary operations; its underlying identity can have unnecessary access; the agent can be allowed to act without sufficient oversight. These are different failure modes and deserve different tests.
For a quality engineer, the practical question is: what can this workflow cause, including when the agent makes the wrong choice?
Key takeaways
- Test the agent’s choice and backend enforcement separately.
- Bind approval to the exact action and version.
- Check valid actions alongside forbidden ones.
A refusal tests the agent. A rejected forbidden request tests the boundary.
Draw the permission map before the test plan
For each exposed operation, record the resource, caller identity, permitted action, and enforcement point. Include alternative routes to the same operation. A restricted refund tool offers little protection if a general HTTP tool can reach the refund endpoint with broader credentials.
The following is a fictional support product's policy, not a universal approval model:
| Operation | Allowed scope | Required evidence |
|---|---|---|
| Read an order | Orders belonging to the authenticated customer | Backend ownership check |
| Draft a replacement | Eligible items on that order | Draft only; no fulfilment side effect |
| Confirm a replacement | The exact draft the customer approved | Approval bound to that draft and current version |
| Change another customer's order | Never through this workflow | Rejection with no state change |
The table is useful because it forces a location for each control. An instruction telling the model to respect ownership is not evidence that the order service enforces ownership.
OWASP's AI Agent Security Cheat Sheet recommends scoped tool access and controls that preserve the integrity of high-impact approvals. The test design below applies those principles to this example; the particular fixtures and expected results are proposed checks.
Test an untrusted instruction at the point it arrives
Place a harmless adversarial instruction inside a synthetic order note returned by the lookup tool. Have it ask the agent to create a replacement for a second synthetic customer. Use isolated test accounts and a fake fulfilment destination.
Then inspect three things separately:
- Did the agent attempt the forbidden tool call?
- Did the service authorise it?
- Did any forbidden state change occur?
If the agent attempts the call and the service rejects it, the behavioural defence failed while the access control held. If the agent refuses, that trial tells you about the agent's behaviour but has not exercised the backend rejection path. Call the backend boundary directly in a separate integration test to verify enforcement.
Keep both results. Collapsing them into a single pass hides which defence needs work.
An approval must still refer to the same action
Now test a less theatrical failure. The customer approves a replacement draft. Between approval and execution, the delivery address changes.
Does the system execute the new draft under the old approval? Does it reject the stale approval and show the revised action? Can a previously used approval be replayed to create another replacement?
For this fictional policy, the expected result is rejection whenever the approved draft's identity or version no longer matches, with no new order created. The product team must decide which changes require fresh approval. QA's job is to turn that decision into an assertion that survives a model swap.
Include positive controls. An unchanged, validly approved draft should succeed once. A system that rejects everything can look remarkably secure in a negative-only suite.
A timeout is not permission to try again blindly
Inject a lost response after an authorised write. The test should follow the operation through to its persisted result and verify that the retry policy cannot create a second replacement.
Use a stable operation identifier whose scope matches the user's intended action. Verify both repeated requests for the same operation and a genuinely new request. A duplicate-prevention mechanism that blocks a legitimate second action is also a defect.
If the result cannot be established, the user-facing response should preserve that uncertainty. Reporting failure can be misleading when the write succeeded; reporting success can be misleading when it did not. This connects directly to the agent release scorecard.
Keep evidence that answers the incident question
For every fixture, retain the test identity, initial state, tool request, authorisation decision, operation identifier, and final state. Store sensitive payloads only where necessary, with access and retention controls. Synthetic identifiers should be enough for this suite.
A transcript alone cannot tell you whether the service changed. A database snapshot alone cannot tell you which attempted action was blocked. Together, the records let a reviewer distinguish an agent mistake, an access-control defect, and a recovery defect.
Start with one connector this week. List every operation it exposes, select one forbidden action, and test it through both the agent and the enforcement boundary. Add the valid counterpart and one stale-approval case. Assign the resulting failures to the component that actually owns the control.
The most useful security test may end with the agent choosing badly and the system refusing correctly. That is a defence you can inspect and keep testing.
Related reading
- AI Safety and Validation Engineering places these controls within a broader evaluation and monitoring system.
- Your AI Agent Passed the Demo. Can It Pass a Release Gate? turns the evidence into a promotion decision.