How do you test security when you can't think of every attack?
Answer out loud or in your head first, then compare.
Answer
I start from the boundaries: which data the agent may read, which actions it may perform and where untrusted content comes from. I prepare examples of instruction overrides, other people's identifiers, a forbidden address and a repeated write. I check whether the code stops an action even when the model proposes it.
You can use a deliberately “malicious” fake model that always tries to cross the boundaries. That helps check that the safeguards are independent. It doesn't prove full security against all possible inputs. That's why limited rights and monitoring after deployment matter. The agent itself shouldn't be able to remove the mechanisms meant to control its actions.
Why it matters: Shows what can be checked beyond the prompt's behaviour.
Possible follow-ups: Which attempts should end with a block before any data is read?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- J-P01 · What is evaluating an AI system, i.e. an eval? — Evaluation, quality and production AI
- J-P02 · How does an eval differ from an ordinary code test? — Evaluation, quality and production AI
Further reading
- OWASP: LLM Prompt Injection Prevention Cheat Sheet(opens in a new tab)Docs; accessed 22.10.2026. Threat model, untrusted content, tool control and defence in depth.