Insecure Output Handling in LLM Applications
Teams check what goes into the model. They forget what comes out. But the output of a language model is not an answer. It is untrusted text, and treating it as trusted is how an attack rides out the other side.
This is risk LLM02 in the OWASP Top 10 for LLM Applications, and it is the one developers trip over most, because it feels wrong. You validated the user’s input. The model is your own component. So whatever it returns must be safe, right?
No. The model’s output is shaped by everything it read, including data an attacker controls. If that output flows, unchecked, into a place that interprets text as instructions, a web page, a database query, a shell command, a file path, it carries whatever an attacker managed to steer it into saying.
A real case
Through 2023, researchers showed a clean version of this. Get a model to answer with a Markdown image, where the image’s web address carries the user’s private data inside it:
The moment the app renders that Markdown, the browser fetches the “image,” and the data is sent to the attacker’s server. The user sees a broken image icon, nothing more. The attack was not in the input. It rode out on ordinary-looking output that the app trusted enough to render.
Why it is so easy to miss
The danger is the mental model. You think of the model as part of your system, like a function you wrote. But a function you wrote does what you coded. A model does what the text it read steers it toward, and some of that text came from outside. So the output deserves exactly the suspicion you give a raw form field: it might be fine, and it might be an attack wearing the shape of an answer.
It gets worse the more you automate. If the model’s output becomes a SQL query, you have model-driven SQL injection. If it becomes a shell command in an agent, you have remote code execution. If it becomes HTML on a page, you have cross-site scripting. The model is simply the new place the bad string comes from.
The fix
Treat model output as untrusted input, everywhere it lands.
- Encode for the destination. Show output as plain text, not raw HTML. If it must render as HTML, escape it the same way you escape any user content. This is the same discipline as output encoding in input validation. The source changed, the rule did not.
- Never build commands by pasting model text. No model output concatenated into SQL, shell, file paths, or an eval. Use parameterized queries and fixed, typed operations, exactly as you would with user input.
- Constrain the shape. If you expect a number, a date, or one of five options, parse and validate the output against that before using it. A model told to return JSON can still return a surprise.
- Strip or block active content. Links and images in rendered output are an exfiltration channel. Disallow them, or allowlist where they may point.
The point
None of this is new. Output encoding and “never trust input” are decades old. The language model did not change the rule. It added a new, convincing source of untrusted text, one that feels like it is on your side.
So the question for your own app is the plain one: everywhere the model’s answer goes next, into a page, a query, a command, a file, would you let a stranger type that string into the same place? If not, you have an output handling problem, and the model is the stranger.