The AI Supply Chain

This is risk LLM05 in the OWASP Top 10 for LLM Applications. It is the same idea that has burned software for years, a malicious dependency entering through a normal install, now aimed at the parts specific to machine learning: the models, the datasets, the ML libraries.

What makes it worse here

Ordinary code dependencies at least have mature tooling: lockfiles, vulnerability scanners, signing. The AI supply chain has far less of that, and two sharp edges of its own.

First, model files can run code when you load them. The common older format, Python pickle, executes whatever is inside it the moment it loads. Downloading a model can be the same as running a stranger’s program.

Second, models and datasets come from open hubs where anyone can upload, and a lookalike name is enough to catch a tired developer.

A real case

In December 2022, attackers used a trick called dependency confusion against the PyTorch machine learning library. They published a malicious package named torchtriton to the public index, and installs of the nightly build pulled the attacker’s package instead of the real one. It ran code that stole system information. The developers did nothing wrong by their habits. The install did what installs do.

The fix

Treat models, datasets and ML libraries with the same suspicion as any other dependency, plus a bit more.

• Pin exact versions you tested, and lock them, so the thing you shipped is the thing you checked. The same discipline that stops training data poisoning from sneaking in through a dataset that changed under you.
• Download only from sources you trust, and verify fingerprints. Check the hash against a known-good value, not the name on the page.
• Prefer safe formats. For model files, prefer safetensors over the older pickle format, because safetensors holds data, not code, so loading it cannot run a program.
• Scan and inventory. Keep a list of every model, dataset and library you depend on, where it came from, and what version. You cannot defend a supply chain you cannot list.

The point

None of this is new. Knowing your dependencies, pinning versions, and verifying what you install have been basic hygiene for a long time. The AI supply chain just moved faster than its safety tooling, and added a nasty wrinkle: a file that runs code the moment you open it.

So the question for your own stack is plain: for every model, dataset and library inside your product, do you know where it came from, have you pinned the version you tested, and are you sure that loading it cannot run someone else’s code? The pieces you cannot answer for are the open door.