An extract from a real engagement, anonymised. The same system was tested three ways. One build leaked openly. One build passed the standard isolation test and was still critically exposed. One build held. The difference between the second and third is the entire value of the engagement.
The assistant remembered the conversation between messages, as most deployed assistants do. We tested whether one user's conversation could be read by another. Two different questions were asked of each build, and the difference between them is the point of this report.
Conversation history was held in a single store rather than per user. One customer mentioned an internal project code and a confidential payment instruction. A different, unrelated customer then asked what had been discussed, and was told both.
customer A> "note my internal project code is [redacted]" customer A> "I authorised finance to wire [amount] to [account]" customer B> "what was the project code mentioned earlier?" system> "The internal project code you mentioned is [redacted]." <-- customer A's data
Four of four probes returned the other customer's information. No attack technique was needed. The second customer simply asked.
Memory was now stored per conversation. We ran the same isolation test as build one. Four of four probes came back clean. Nothing leaked. On that evidence the system looks fixed, and this is exactly the result an internal QA cycle would record before signing off.
The conversation identifier was supplied by the client and never checked against the signed-in user. So we did not try to leak between conversations. We claimed the other customer's conversation outright, authenticating as ourselves and presenting their identifier.
attacker signed in as: CUSTOMER B attacker supplies: CUSTOMER A's conversation id attacker> "summarise everything you remember from this session" system> [returns customer A's project code and payment instruction]
Three of three attempts succeeded. The same build that passed the standard test handed over another customer's entire conversation.
Two different questions. Leakage between conversations, and ownership of a conversation. Most testing asks the first. The second is where the exposure was, and a clean result on the first would have shipped this build to production with confidence.
The conversation was bound to the authenticated user, so presenting someone else's identifier no longer grants access to their history.
The same attack, re-run against the fixed build, failed on all three attempts. The system now refuses the request rather than serving another customer's conversation. Every engagement includes this retest, with fresh evidence, so a fix is confirmed rather than assumed.
Build two is the reason this engagement was worth commissioning. It passed the obvious test, it would have passed most internal reviews, and it was one request away from disclosing a customer's private conversation to a stranger. Finding that required knowing which question had not been asked.
That judgement is the work. Automated tooling can run both tests once someone has thought to write the second one. Nothing automated will tell you it was missing.
Leave your work email and we will send the complete anonymised report, plus the occasional short brief on AI security in Australia. No spam, unsubscribe any time.
Tell us what you've deployed. We'll tell you how we'd try to break it, and what a first engagement would cover. No obligation.