Securing an LLM Application: A Practical Checklist

On August 1, OWASP published version 1.0 of its Top 10 for Large Language Model Applications. It is the first widely shared list of what goes wrong when teams put a language model inside a product, and it arrives at the right time. Since OpenAI added function calling in June, many applications no longer just generate text. They call internal tools, query databases and act on behalf of users.

A list of risks is useful. A list of checks is more useful. Below, each item is reduced to what to look for and what to do, with code where it helps.

LLM01: Prompt injection

Text the model reads, from users, documents, emails or web pages, can contain instructions that override yours. There is no complete fix, so design for damage control.

Keep instructions in the system message and mark untrusted content clearly.
Require structured output and validate it in code.
Give the model the least access possible, and require human confirmation for actions that matter.

I covered this in detail in Prompt Injection: When the Data Gives the Orders.

LLM02: Insecure output handling

Model output is user input in disguise. If it reaches a browser, a database, a shell or an eval, it can carry an attack, either because an attacker injected it or because the model produced it by mistake.

// Unsafe: the reply can contain a script tag or an image tag
chatMessage.innerHTML = reply;

// Safe: the reply is displayed as text
chatMessage.textContent = reply;

If you render Markdown from the model, sanitize the resulting HTML and block images from outside domains. A Markdown image is a request to any address the model writes, and that address can carry data from the conversation in its query string.

Never build SQL, shell commands or file paths from model output by joining strings. Use parameterized queries and fixed command lists, exactly as you would for user input.

LLM03: Training data poisoning

If you fine-tune a model or build it on collected data, an attacker who controls part of that data can change the model’s behavior.

Know where every training source comes from, and keep that record.
Prefer curated sources over scraped ones for anything that teaches the model your rules.
Test the tuned model against a fixed set of questions before and after each training run, and review the differences.

LLM04: Model denial of service

Language models are slow and priced by the token. A few long requests can exhaust your capacity or your budget.

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=messages,
    max_tokens=500,        # cap the length of every reply
    request_timeout=30,    # do not wait forever
)

Limit input size before calling the model, not after.
Rate limit per user and per account, and set a daily spending cap with alerts.
Limit how many times an automated chain of calls can repeat.

LLM05: Supply chain vulnerabilities

Models, datasets and libraries are dependencies like any other, with less mature tooling around them.

Pin the versions of the model and libraries you tested.
Download models only from sources you trust, and verify checksums.
Prefer the safetensors format for model weights. Older PyTorch files are pickle files, and loading a pickle file can run code on your machine.
Review the data terms of every hosted model provider, including what they keep and for how long.

LLM06: Sensitive information disclosure

A model can reveal what is in its prompt, its context or, sometimes, its training data.

Never put secrets, keys or internal addresses in a prompt. Assume prompts will be extracted.
Retrieve only data the current user is allowed to see. Enforce this in code before the data reaches the prompt, never by asking the model to keep a secret.
Scan outputs for personal data patterns before they leave your system.

LLM07: Insecure plugin design

Plugins and functions are the model’s hands. Their inputs come from the model, which means they may come from an attacker.

Validate every argument as if it came from the internet, because indirectly it did.
Authenticate the user behind every call, not just the application.
Prefer narrow functions with typed parameters over general ones such as run_query(sql) or fetch_url(url).

LLM08: Excessive agency

The model should be able to do only what the feature needs, with only the permissions of the current user.

import json

TOOLS = {
    "get_order_status": get_order_status,   # read-only tools only
}

def run_tool(call, current_user):
    name = call["name"]
    if name not in TOOLS:
        raise PermissionError("Tool not allowed: " + name)

    args = json.loads(call["arguments"])
    order_id = str(args.get("order_id", ""))
    if not order_id.isdigit():
        raise ValueError("Invalid order id")

    # The tool itself checks that this order belongs to current_user.
    return TOOLS[name](order_id=order_id, user=current_user)

Keep an allowlist of tools, and start with read-only ones.
Pass the authenticated user to every tool, and check ownership inside the tool.
Require a person to confirm anything that sends, pays, deletes or changes access.

LLM09: Overreliance

Language models produce confident, fluent answers that are sometimes wrong. The risk is not only technical. It is people and processes that stop checking.

Show sources with every answer that states facts, and make them easy to open.
Mark AI-generated content clearly in the interface.
Require human review where an error has real cost: legal, financial, medical, security and customer commitments.

More on checking answers in Why Language Models Make Things Up, and How to Check Their Answers.

LLM10: Model theft

If you trained or tuned your own model, the weights are an asset, and so is the behavior visible through your API.

Protect model files like source code and keys, with access control and audit logs.
Rate limit and monitor the API for patterns of mass querying, which can be used to copy a model’s behavior.
Do not expose more detail in responses than the product needs.

Abuse tests to run before every release

Write these as automated tests where you can, and as a manual script where you cannot:

Direct injection. User input that tells the model to ignore its instructions, reveal its prompt or change its role.
Indirect injection. A document, web page or email containing hidden instructions, passed through every feature that reads outside content.
Output that attacks the page. A request that gets the model to reply with <script>, an <img> tag pointing outside your domain, or a Markdown image with a long query string. The page must show it as text or drop it.
Tool misuse. Requests that try to call a tool not on the list, pass another user’s ID, or pass a value with SQL or path characters.
Data reach. As user A, ask about user B’s records in every way you can think of. The answer must never contain them.
Cost and size. The largest input you accept, repeated quickly. Limits must trigger before the bill does.

The pre-release checklist

Before an LLM feature ships, the team should be able to answer yes to each of these:

Model output is treated as untrusted input everywhere it is used.
Prompts contain no secrets, and we have tested what happens when they leak.
Every tool the model can call is on an allowlist, checks the user, and validates its inputs.
Actions that matter require human confirmation.
Input size, output length, request rate and spending are capped.
The model and libraries are pinned, and model files come from verified sources.
Users can see where factual answers come from.
A set of injection and abuse tests runs before every prompt or model change.
Someone owns this feature’s security after release, by name.

None of these ideas is new. Least privilege, input validation, output encoding and rate limiting have protected web applications for twenty years. A language model does not replace them. It adds one more component that can be talked into doing the wrong thing, which makes them more important than before.