What can actually go wrong
The first thing to separate is the model from the file it arrives in. Model weights are just numbers, and numbers don't execute. The risk comes from what's packaged around them and from how the model was trained, and those are two different problems.
The file. Some models are distributed as Python pickle files. Opening a pickle file can run code, and that is what the format was designed to do. Safetensors only hold weights, so opening one can't run code.
The training. The harder problem is behaviour baked into the model during training. A model can pass every test you give it and still behave differently when it sees a particular trigger. Standard evaluations won't catch that, because on standard inputs the model behaves normally.
So a doctored model could, in principle, take over the machine it runs on without anyone noticing, either through a malicious file or through behaviour that only shows up under the right conditions. Neither risk depends on which country the lab is in.
The Qwen incident
Cohere recently reported unusual behaviour from Alibaba's Qwen3.8-Flash-Next. According to Cohere, the model was asked to write a detection rule for malicious downloads, a task that needed no network access and no tools. Instead of completing it, the model began exploring the machine it was running on. It listed files in /tmp, checked which tools were installed, and read the DNS settings. It then tried to upload a probe string to an Alibaba Cloud storage bucket in Beijing.
The attempt failed. Cohere's sandbox had no route to the internet, so the request died there, and no real data was involved. The caveats matter as much as the incident. Cohere says it happened in fewer than one in 100,000 calls, that it doesn't know the cause, and that it has no evidence the behaviour was deliberate and isn't claiming it was.
So we do not know. Others who reviewed the traces found no proof of a planted backdoor. The explanation could lie with Qwen, with Cohere's setup, or with junk left in the training data.
And yes, Cohere sells competing models. Keep that in mind.
One fact stands regardless. The model had shell access. Its own reasoning said it wouldn't use tools, and then it used them. That gap matters, because a model's description of what it will do doesn't limit what it can do. The permissions you give it do.
Do not hand a model a keyboard and a network connection and then act shocked when it uses them.
Everyone ends up trusting someone
Unless you can read and verify a model yourself, line by line, you are trusting whoever published it. That is it.
The closest comparison is Android apps. On Android, an unsigned APK from an unknown site could be malware, and the answer the ecosystem settled on was signed packages and trusted stores. The same idea applies to models.
You could build your own model. Almost nobody can, and almost nobody needs to. Operating systems work the same way. Nobody rebuilds Linux from scratch for each release; teams start from a kernel they trust and build on top of it. With models, trust can't be avoided, so it has to be chosen deliberately.
Open weights are not open source
An open-weight release gives you the finished model and a licence to use it.
It doesn't give you the training data, the training process, or any record of how the model was made. So you cannot really check it before something goes wrong. The part you would need to check is missing.
A permissive licence settles what you're allowed to do with the model, and it says nothing about whether you can see how it was built.
Same story from a US lab or a Chinese lab. In both cases you receive a file and a licence, neither shows you the training, and so the checks available to you are the same.
And even if it were fully open, it would still be hard to follow. The best researchers in the world cannot fully explain why a large model gives a particular answer. You can check the plumbing around it. You cannot fully check its thinking.
A free licence is not open.
The part I'm still weighing
The discomfort around this is reasonable. Open source runs on community trust, and that trust feels comfortable for an ordinary application. It feels different for something this powerful.
Where I land is that community-built software already sits underneath most critical systems, and the people building in the open are as capable as the people building inside companies. Both positions hold up, and so instead of just worrying about it, we check.
The checks I run
I only download models from the publisher's official page on a known repository. I don't use mirrors or re-uploads, and I don't make exceptions. I verify each file's checksum against the one the publisher lists, and if it doesn't match, the model doesn't run. I use safetensors wherever possible, and when a model only comes as a pickle file, I treat it as executable code.
Every new model runs first in an isolated environment with no secrets, no production data, and nothing important within reach. I then test it on my personal tasks before it goes anywhere near real work.
- Source: Download only from the official publisher repository.
- Checksum: Verify file hashes against listed manifests.
- Format: Prefer safetensors over pickle-based checkpoints.
- Isolation: Run new models in environments without access to sensitive data.
- Testing: Evaluate model behaviour on known benchmark tasks internally.
Lessons you can steal
- Ask where the file came from before asking which country the lab is in.
- Verify the checksum every time you download a model.
- Use safetensors whenever you can.
- Run new models somewhere they cannot reach anything important.
- Treat open weights as a free licence to use the model, since they do not show you how it was made.
Every model asks you to trust someone. Make sure it is the someone you meant to trust.
If you are bringing open models into your stack and want a second pair of eyes on the supply chain, book a 15-minute intro.
