Semgrep posted an argument that model weights defeat inspection, leaving AI supply chains beyond audit. They are right to ask for provenance and independent verification, but the approach was such a mess that I was surprised how they decided to land.
| Open Weights Claim | Reality |
|---|---|
| The problem is lost reverse engineering | Reverse engineering is when you don’t get the source. That’s a confidentiality problem. Model risk instead is an integrity problem. The difference is in attested lineage, reproducible training, and tamper evidence on the artifact. The essay’s own conclusion talks about integrity controls, which is the right landing, but all the stuff above it is stuck on confidentiality concerns that don’t fit. |
| The problem is poisoning is undetectable, as if open weights add risk | Every detection method that exists requires some kind of access: weight diffing, trigger reconstruction, activation probing. Anthropic’s defection probes reached AUROC above 99 percent against its own sleeper agent models using residual stream activations. Possessing the weights is the actual and known audit event that the essay tries to argue is impossible. |
| The problem is malicious post-trained variants as an open-weight hazard | A hostile fine-tune, the Shadow Alignment scenario, produces a new artifact with a new hash. That’s apples and oranges. Consumers pinned to the canonical release stay untouched. Signing and namespace controls solved this class of problem years ago. Silent model substitution is instead the default condition of API consumption, where the provider swaps weights at will. And THAT is a maddening integrity breach severely under reported. Anthropic Opus 4.7 did this in April 2026: a launch-week model that degraded within days, a postmortem admitting three silent behavior-altering changes while denying any inference-layer change, which demanded the need for a changelog that nobody ever saw. |
| The problem is Anthropic (conflating two different things) | A constant-sample-count finding belongs to the 2025 poisoning study with UK AISI and the Turing Institute. Sleeper Agents found backdoor persistence greatest in the largest models, and it places the adversary at the trainer, a threat model that actually indicts a closed distribution. |
| The problem is what Thompson said | The compiler backdoor survived source review because the build process was compromised. That lesson was toolchain provenance. The result of Wheeler’s diverse double-compiling gives us independent reproduction for verification. The model analogue is retraining from attested data. And that is available ONLY where weights and data are open. Thompson fits with the post’s conclusion and against all of the framing above it. |
Open weights mean any auditor can get an artifact.
Closed weights still mean there’s no independent auditing.