“The Hugging Face hack” is a phrase I now hear the way I used to hear “the Equifax breach” — as though there were one event, one date, one thing that got fixed. There were, at minimum, five distinct incidents on five different mechanisms across three years, and collapsing them into one story is how a person ends up drawing the wrong lesson from the right file.
Straightening the record is not the point of this essay, though I will do it. The point is what the record does once it is in order: it stops reading like a list of vulnerabilities and starts reading like a single argument in three acts. Every time the open-model ecosystem closed one layer, the risk moved up to the next layer. That is not evidence anyone involved was careless — the opposite, as I will get to. It is what happens, reliably, when security gets bolted onto a format instead of designed into it.
This is an artifact question, not a behavior question. A separate piece in this section asks whether we can prove a model does only what we intended — a question about the thing’s mind, if it has one. This essay asks a narrower, more immediate question for a law firm: is the file on your disk actually what it claims to be, from whom it claims to be, doing only the arithmetic it claims to do? That question has a three-year paper trail, dated and public, worth reading end to end before anyone here touches a downloaded model again.
None of what follows argues for restricting open weights, gatekeeping a hub, or handing anyone veto power over what gets published. I made the opposite argument at length in this section on August 20, and nothing here backs away from a word of it. Openness is not the vulnerability. Trust without verification is — exactly as true of a closed vendor’s black box as of an open file you can at least inspect.
A checkpoint file is a program, not a document
Start with what “download a model” actually means, because the phrase is doing a lot of quiet work to make this sound safer than it is.
A trained neural network’s parameters — the numbers a model has learned — have to be written to disk somehow, and PyTorch’s native way of doing that is to serialize them with Python’s pickle module, typically wrapped in a .bin, .pt, or .pth file. Pickle is not a data format the way JSON or a spreadsheet is. It is a small, stack-based bytecode format, and its opcode set includes instructions that import a module and call a callable. Python objects can define a __reduce__ method telling pickle how to reconstruct them on load, and nothing stops that method from returning a callable like os.system paired with a shell command as its argument. When code calls torch.load() on such a file with an unrestricted unpickler, it unpickles it — and unpickling it runs that call. That is not a bug in pickle. It is pickle doing exactly what it was built to do. Loading a PyTorch checkpoint of unknown origin is, functionally, executing a program written by a stranger. (PyTorch has since narrowed the default: starting with version 2.6, released January 29, 2025, torch.load() sets weights_only=True unless a caller overrides it, which restricts the unpickler to tensors and a short list of primitive types and blocks it from importing anything else — PyTorch’s own documentation calls this a narrowing of the attack surface, not an elimination of it, since a caller can still opt back into weights_only=False and denial-of-service and memory-corruption risks remain even under the restricted loader. Every incident in this essay predates that change, and a great deal of code in production today — including code written before 2025 and never revisited — still calls torch.load() in ways that turn the restriction back off.)
JFrog’s security research team demonstrated exactly how exploitable this was on February 27, 2024, after doing what they describe as routine monitoring of models on the Hub: “we’ve been regularly monitoring and scanning AI models uploaded by users, and have discovered a model whose loading leads to code execution, after loading a pickle file.” Their write-up is plain about the consequence: “The model’s payload grants the attacker a shell on the compromised machine, enabling them to gain full control over victims’ machines through what is commonly referred to as a ‘backdoor.’” In one specific case they traced, a repository called baller423/goober2, “it appears that the malicious payload was injected into the PyTorch model file using the __reduce__ method of the pickle module. This method… enables attackers to insert arbitrary Python code into the deserialization process, potentially leading to malicious behavior when the model is loaded.” JFrog’s scan turned up “around 100 instances of such models” already sitting on the Hub — not proof-of-concept files sitting in a lab, but live uploads a data scientist could have pulled down and run that same week.
Say the plain version to a client and watch the reaction: downloading a machine-learning model and calling .load() on it is running an executable obtained from a stranger, with no more assurance about its contents than opening an email attachment from an unknown sender. Everyone in this profession understands the second sentence instinctively. Almost nobody translates it to the first.
The fix was bolted on, and the evasion arrived on schedule
Hugging Face’s response to the pickle problem had two parts, and the essay’s whole argument turns on the fact that only one of them actually fixed anything.
The first part was Picklescan, an automated scanner the Hub runs against every uploaded pickle-format file, flagging it “unsafe” when it finds calls to a blacklist of dangerous functions — os.system, eval, and the like. It is real and useful, and it is also, by design, a blacklist scanner bolted onto a format that was never made safe. It flags; it does not block. A user can see the “unsafe” label, download the model anyway, and run it, and nothing in the platform stops them.
The second part was safetensors, and this one is the genuine fix, not a patch. Hugging Face’s own account of why they built it does not soften the diagnosis: “The creation of this library was driven by the fact that PyTorch uses pickle under the hood, which is inherently unsafe… With pickle, it is possible to write a malicious file posing as a model that gives full control of a user’s computer to an attacker without the user’s knowledge.” Safetensors solves this structurally rather than procedurally. Per the project’s own README, it “implements a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy).” Structurally, a safetensors file is eight bytes giving a header length, then a JSON header mapping tensor names to their dtype, shape, and byte offset, then the raw tensor bytes themselves. There is no opcode stream in that structure. There is no __reduce__ equivalent, no instruction anywhere in the format that means “call this function.” It is declarative data — numbers and where to find them — not a program, and it cannot be made to execute code on load because the format has no execution semantics to exploit in the first place. That is categorically different from “we tell people not to run untrusted pickle files.” It is “the file cannot do the thing regardless of what anyone tells anyone.”
Hugging Face did not simply assert this. It commissioned an outside audit from Trail of Bits, together with EleutherAI and Stability AI, and its own post relays the result plainly: “No critical security flaw leading to arbitrary code execution was found… Some missing validation allowed polyglot files, which was fixed.” Even there, Hugging Face does not oversell the finding: “While it is impossible to prove the absence of flaws, this is a major step in giving reassurance that safetensors is indeed safe to use.” That is the right amount of confidence for a security claim, and it is worth noticing when a vendor gets that calibration right.
The rollout was not one event but two, roughly six months apart, and treating it as a single date will get an essay or a due-diligence memo wrong: safetensors became an installed-by-default dependency of the transformers library around May 2023, and it became the default format when saving new models only later, with transformers version 4.35.0 in November 2023. Anything uploaded before that second milestone — and a great deal of what was uploaded before the first — may still be sitting on the Hub in pickle format today, unsafe by construction, with a “please be careful” label attached to it.
Eleven months after JFrog’s finding, ReversingLabs found the seam between the fix and the thing it was bolted onto. On February 6, 2025, RL researchers disclosed what they named “nullifAI”: “Software development teams working on machine learning take note: RL threat researchers have identified nullifAI, a novel attack technique used on Hugging Face.” The technique exploited a mismatch between how Picklescan parses a file and how Python’s interpreter actually runs one: “The two models RL detected are stored in PyTorch format, which is basically a compressed Pickle file. By default, PyTorch uses the ZIP format for compression, and these two models are compressed using the 7z format, which prevents them from being loaded using PyTorch’s default function, torch.load().” That mismatched compression corrupted the file from Picklescan’s point of view without stopping Python’s pickle interpreter from working through it. RL’s own diagnosis is the sentence that matters most here: “Pickle file deserialization works in a different way from Pickle security scanning tools. Picklescan, for example, first validates Pickle files and, if they are validated, performs security scanning. Pickle deserialization, however, works like an interpreter, interpreting opcodes as they are read — but without first conducting a comprehensive scan to determine if the file is valid, or whether it is corrupted at some later point in the stream.” The scanner reads the whole file before deciding anything; the interpreter runs each instruction as it arrives and does not care whether the file ends cleanly. A malicious REDUCE opcode placed early enough executes before the corruption that would have made Picklescan give up ever arrives. RL found two models built this way, both delivering “a typical platform-aware reverse shell that connects to a hardcoded IP address,” and credits related earlier work by Checkmarx with identifying additional blacklist-bypass functions — a claim I have not independently verified beyond RL’s own characterization of it, so I report it one level removed rather than as confirmed by me directly.
RL’s stated conclusion is worth quoting because it is the whole thesis of this essay in one sentence from an independent source: “Pickle files present a security risk when used on a collaborative platform where consuming data from untrusted sources is the basic part of the workflow.” A blacklist scanner bolted onto an inherently executable format was always going to have a gap between what it checks and what actually runs, because that gap is not a bug in the scanner — it is the difference between validating a file and executing one, and no amount of scanner refinement erases that distinction. Safetensors erased it by removing execution semantics from the format itself. Picklescan could only ever chase it.
Two other incidents get folded into “the Hugging Face hack” in casual conversation, and I want to name them and set them aside rather than pretend they support this argument, because padding a security narrative with unrelated incidents is exactly the kind of thing that gets exposed by the first reader who knows the record. In June 2024, Hugging Face disclosed that its Spaces hosting platform had suffered “unauthorized access… specifically related to Spaces secrets” — an intrusion into Hugging Face’s own infrastructure exposing API tokens Spaces app authors had stored as environment variables, not a defect in any model file. And in late 2023, Lasso Security reported finding roughly 1,700 valid API tokens leaked across Hugging Face and GitHub through the ordinary mechanism of developers hardcoding credentials into public code. Both are real security failures. Neither is a model-supply-chain problem — they are the conventional credential-hygiene failure that could happen, and does happen, to any SaaS platform or code registry, with nothing AI-specific about the mechanism. I am leaving them out of the argument on purpose, not by oversight.
The backdoor that lives one layer above the file format
The third act is the one that should end any comfortable feeling that safetensors solved this.
On October 10, 2024, HiddenLayer’s research team disclosed a technique they call ShadowLogic, and their own framing states plainly what makes it different from everything above: “Instead of focusing on the model’s weights and biases, we began to investigate the potential to create backdoors in a neural network’s computational graph.” Every neural network, regardless of file format, is ultimately a graph — a specific sequence of mathematical operations the model executes to turn an input into an output. HiddenLayer’s technique edits that graph directly, inserting a hidden conditional branch: if the input matches a chosen trigger, force a chosen output, otherwise behave normally. They demonstrated it on three different kinds of models: ResNet, an image classifier, where a specific pixel pattern flips the classification result; YOLO, an object detector, where the simultaneous presence of a person and a cup in frame suppresses the “person” detection entirely; and Phi-3 Mini, a language model, where a particular token sequence forces a chosen output regardless of the actual prompt. In their own words: “This method can be easily implanted in pre-trained models, will persist across fine-tuning, and enables an attacker to create highly targeted attacks with ease. We call this technique ShadowLogic.”
Read that persistence claim twice, because it is the detail that should worry a firm more than any of the pickle findings. A backdoor that survives fine-tuning means a downstream user who takes a trusted base model and retrains it on their own data — the standard, recommended way to specialize an open model for a particular use — does not retrain the backdoor away. They carry it forward into a model they now believe is theirs.
And then there is the sentence that ends this three-act structure and forecloses the comfortable reading of everything before it: “One of the most alarming consequences is that these backdoors are format-agnostic. They can be implanted in virtually any model that supports graph-based architectures, regardless of the model architecture or domain.” Safetensors’ entire guarantee is that the file cannot execute code on load. ShadowLogic does not ask the file to execute anything. It changes what the model itself computes once loading has already finished successfully and safely, by every measure safetensors was built to satisfy. HiddenLayer’s own summary of the category is precise: “The emergence of backdoors like ShadowLogic in computational graphs introduces a whole new class of model vulnerabilities that do not require traditional code execution exploits. Unlike standard software backdoors that rely on executing malicious code, these backdoors are embedded within the very structure of the model, making them more challenging to detect and mitigate.” That is the format fix, cleanly stepped around, exactly one layer up from where it was built to defend.
Two things need saying about ShadowLogic with more care than most coverage gives it. First, this is responsibly disclosed offensive security research demonstrated by the researchers who found it, not a reported breach with a real victim — I found no evidence of an in-the-wild ShadowLogic campaign, and this essay should not be read to imply one. What it demonstrates is that the technique works on production-relevant architectures, which is a category of risk with a working proof, not an anecdote — a different and in some ways more useful thing to know than a single incident report. Second, HiddenLayer sells a commercial detection product marketed against exactly this threat, and I have not independently verified that product’s efficacy; a vendor’s claim about the danger of a problem it also sells the solution to is testimony that should be discounted by the size of that interest before it is repeated as fact. Both cautions hold at once: the underlying finding is sound and format-agnostic backdoors are real, and the company that found them has a financial reason to want you frightened enough to buy something.
The honest defense of this record, and why it does not rescue the fix
A fair reader will notice something I have not disputed: every one of these fixes arrived fast. JFrog published in February 2024, and Hugging Face already had Picklescan running and safetensors already built. The safetensors audit was external, published, and candid about its own limits. OpenSSF shipped a working, Sigstore-based signing standard for model artifacts a little over a year after the pickle finding went public. That is a genuinely fast response for a loosely coordinated open ecosystem reacting to independent research rather than to a court order, and it deserves to be said plainly rather than buried under the criticism: this is roughly what a functioning immune system looks like. Nobody sat on the JFrog finding. Nobody argued safetensors wasn’t worth building.
I do not think that concession rescues the fixes as adequate, and here is why: an immune system that reliably arrives one step behind the pathogen is real, and it is also proof the underlying architecture keeps generating new attack surface faster than any single fix retires the old one. Fast patching of one layer, while the exploit migrates cleanly to the next layer up, is not a rebuttal of “bolted on rather than designed in.” It is what bolting on looks like when the people doing it are competent and fast. The lesson is not that the open-model ecosystem is careless — the opposite is demonstrated on every page cited above. The lesson is that a security property proven at the file-format layer tells you nothing about the layer above it, and nobody should have believed otherwise after the second act, let alone the third.
What a firm actually does with this
None of the above is a reason to stop using open models. It is a reason to stop treating “I downloaded it from a reputable hub” as due diligence. Concretely, for a firm running its own inference stack — and I have written elsewhere in this section about why a firm would want to — five things are checkable today, and none of them require trusting anyone’s marketing:
Prefer safetensors and refuse pickle-format checkpoints when an alternative exists. Most current releases on the major hubs ship safetensors by default; where a repository only offers a .bin or .pt file and a safetensors equivalent exists elsewhere, take the safetensors version. Where it does not exist at all, that absence is itself information about how actively the artifact is maintained.
Pin model revisions by commit hash, not by tag or branch name. A tag like main or latest can be repointed to different weights at any time, by anyone with write access to that repository, without your pipeline noticing the substitution. A specific commit hash is content-addressed — the hash is a description of the exact bytes — and re-pulling it later gets you the same file or an explicit error, never a silent swap.
Treat trust_remote_code=True as the loaded gun the name implies. Many Hub repositories bundle custom Python modules alongside the weights, and loading them with this flag set imports and runs that code on your machine — a mechanism with the same practical consequence as the pickle problem, deliberately re-opened by a configuration flag rather than by an exploit. If you do not know why a particular model needs it, that is reason enough not to set it.
Sign and verify, where the tooling now allows it. OpenSSF’s Model Signing project shipped version 1.0 on April 4, 2025, built on Sigstore, the same keyless-signing infrastructure the software supply chain already uses for package and container signing. Its own announcement states the gap plainly: “there is no indication that a model being uploaded there matches what was intended during training. Similarly, there is no means to trace the provenance of the model back to its creator.” Signing proves only this: the bytes you received are the exact bytes a specific, identified signer produced — origin and integrity, not safety, not correctness. A malicious actor who is also the trusted signer can insert a ShadowLogic-style backdoor, sign the result, and have it verify perfectly, because verification checks who produced the file and whether it changed since, not what the file computes. As of this writing, adoption on the major hubs is opt-in — not a gate any hub enforces before a model goes live.
Say plainly what remains unsolved, because pretending otherwise is worse than the gap itself. Graph-level backdoors are, as far as I can find, an open problem for the entire ecosystem, not a Hugging Face-specific failure. I found no evidence of a Hub-level graph-backdoor detector in production as of this writing. A firm’s own diligence — checking hashes, avoiding pickle, refusing unexplained remote code, preferring signed artifacts where they exist — closes the layers this essay’s first two acts describe. It does not close the third. Anyone who tells a client it does is selling reassurance the engineering does not support.
This firm has direct exposure to exactly this risk, and I should say so rather than let it sit in a footnote: the self-hosted tier described elsewhere in this section runs on downloaded, open-weight artifacts, obtained the same way every incident above was obtained. I have argued for building that capability and stand by the argument. Discount my confidence accordingly, and read the sourcing instead.
The contract nobody is writing
Every gap above is also, separately, a contract-drafting failure, and it is worth a short detour even though it deserves its own treatment.
A typical vendor agreement for an AI product built on downloaded model weights says nothing about any of this. It does not warrant that the delivered artifact matches a specific signed hash. It does not give the licensee a right to receive that hash and re-verify it independently rather than take the vendor’s word. Where it does contain an indemnity for security defects, that indemnity is almost always scoped to the artifact “as delivered” — precisely the scope a ShadowLogic-style backdoor is built to survive, since persistence through the fine-tuning a downstream buyer is expected to perform is the whole point of the technique.
None of that is exotic drafting. Provenance warranties, a contractual right to the artifact’s hash at delivery and at any later re-verification, and indemnity language that expressly covers post-fine-tuning discovery are things a lawyer can put in an agreement today, using tools — Sigstore, commit hashes — that already exist and cost nothing to invoke. Almost no agreement I have seen does any of it, because the drafting convention in this industry still treats a model as a service output rather than as the executable artifact it actually is. That gap deserves a full treatment of its own rather than a paragraph here, and I will leave it there for now.
What would prove me wrong
A prediction with no way to fail is not a prediction, so here are the specific, observable conditions under which I would concede each piece of this.
On the escalation pattern being real and ongoing: if five years pass from JFrog’s February 2024 finding with no further escalation — no new layer of attack surface discovered above the computational graph, no new bypass of the current generation of defenses — I would take that as evidence the pattern has run its course rather than being structural, and I would say so.
On graph-level backdoors being an unsolved, ecosystem-wide problem: if a Hub-level graph-backdoor detector ships, is adopted at scale, and holds up under adversarial testing by researchers who did not build it — the way Reluplex-style verification has held up for the narrow properties it actually covers — that specific claim is falsified, and I would update the practitioner guidance above accordingly.
On signing being real but insufficient as currently deployed: if model signing via OpenSSF’s tooling or an equivalent becomes the enforced default on a major hub — required before a model can be downloaded, not merely available to a publisher who opts in — the “opt-in, not enforced” criticism above no longer applies, and that is a specific, checkable fact anyone can verify by trying to upload an unsigned model and seeing whether the hub accepts it.
I hold no stake in any particular hub, scanner, or signing vendor coming out ahead of another. I do hold the stake disclosed above: this firm runs on the kind of artifact this essay is about, and that should make you read every reassurance in it more skeptically, not less.
Sources
- JFrog Security Research, Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor (Feb. 27, 2024) — the pickle
__reduce__mechanism, theballer423/goober2example, and the ~100-model count, retrieved and quoted from JFrog’s own text - ReversingLabs, Malicious ML models discovered on Hugging Face platform (Feb. 6, 2025) — the “nullifAI” Picklescan-evasion technique and RL’s own diagnosis of the validate-vs-execute mismatch, quoted from RL’s own text
- Hugging Face, Audit shows that safetensors is safe and ready to become the default — the format’s design rationale, the Trail of Bits audit result as relayed in Hugging Face’s own post, and the format’s declared limits, quoted from Hugging Face’s own text
safetensorsREADME — the format’s own description of itself as non-executable, quoted directly- HiddenLayer, ShadowLogic: Persistent No-Code Backdoors in AI Computational Graphs (Oct. 10, 2024) — the computational-graph backdoor technique, its format-agnostic property, and the ResNet/YOLO/Phi-3 Mini demonstrations, quoted from HiddenLayer’s own text
- OpenSSF, Launch of Model Signing v1.0 (Apr. 4, 2025) — what Sigstore-based model signing proves and does not prove, quoted from OpenSSF’s own text
- Hugging Face, Space secrets security update (June 2024) — the Spaces infrastructure breach, cited only to set it aside as a conventional credential incident, not a model-supply-chain one
- Companion essays in this section: Protectionism in a Safety Costume, Run It Yourself, Then Let the Client Choose, Running Models on Your Own Hardware, The Model Will Be Free. The Electricity Won’t., and The Gap Between Intended and Proven on the behavioral half of this same problem
Speculative commentary on technology and the business of law — the technical claims are sourced above, and the opinions, predictions, and stated falsifiers are the author’s. Not legal advice. The author’s economic interest is disclosed in the text: this firm operates a self-hosted inference tier built on the kind of downloaded model artifact this essay describes, and that interest should discount the reassurances above accordingly. No client information appears in this article. Questions about anything here: Send us a message or 612-470-6529.