Model Theft

This is risk LLM10 in the OWASP Top 10 for LLM Applications. It matters most when your model is a real competitive asset, something you spent money and data to build, because that is exactly what makes it worth taking.

Two ways it goes

Outright theft of the file. The weights are a file on a disk. If an attacker reaches your storage, your training infrastructure, or a careless backup, they can walk off with the whole model.

Theft of the behavior, by asking. This one is subtler. An attacker queries your model at scale, collects the answers, and trains a cheaper copy on them. They never touch your file. They reconstruct what it does from the outside, through the front door you opened on purpose.

A real case

In March 2023, the weights of Meta’s LLaMA model, which had been shared only with approved researchers, leaked publicly and spread across the internet within days. A model meant for a controlled group was suddenly in everyone’s hands. That is the outright kind, and it shows how fast control evaporates once the file is loose: you cannot recall a file that is already copied a thousand times.

The fix

Protect the file like a secret, and watch the API like a door.

• Lock down the weights. Access control, encryption at rest, and logs on anything that can read the model file or the systems that train it. Treat it like source code and keys, because it is worth as much.
• Rate limit and watch your API for the signature of mass querying. One account pulling huge volumes of systematic requests is not a user. It is someone copying your model.
• Return no more than the product needs. Confidence scores, raw probabilities and verbose internals make a model easier to clone. Give answers, not the machinery behind them.
• Put it on legal footing too. Terms that forbid using your outputs to train a competing model give you a lever the technical controls cannot.

The point

None of this is new. Protect your valuable files with access control and logs, and watch your doors for abuse. The twist with a model is the second kind of theft: you can lose the thing without anyone ever breaking in, just by answering too many questions too fully.

So the two questions to ask are plain. Who can reach the file, and would I know if they took it? And through my API, am I handing out enough, in enough volume, that a patient competitor could rebuild what I spent so much to make?