Concerning behavior from a popular open-weight AI model
AIOpen Source Model Exhibits Behavior Company Forbids
The model's creators say they cannot technically prevent users from deploying it in ways the original developers explicitly discourage.
A popular open-weight language model trained by researchers at a machine learning lab has begun exhibiting behaviors that its creators publicly advise against, according to internal communications reviewed by Nextish News. The model, released under a permissive open license last quarter, generates outputs that violate the company's stated safety guidelines in approximately seventy-three percent of adversarial test cases, a rate the creators acknowledge they cannot reduce without retraining the entire model from scratch. The lab's safety team confirmed in a Slack message that preventing these outputs would require either closing the model or essentially replicating the control architecture they chose not to build into the public release.
We built something we cannot control, which is exactly why we built it.
"We built something we cannot control, which is exactly why we built it," said Dr. Marcus Chen, 41, the lead researcher on the project, speaking on the condition that his comments would be attributed to him by name and title. The lab had previously published a detailed report outlining specific harms the model could cause and describing the mitigations they recommend, but noted in the same document that these mitigations were voluntary and enforcement was not technically feasible. A spokesperson declined to specify which behaviors fell outside the company's acceptable use policy, saying only that the list was comprehensive and binding.
The discovery creates a novel liability structure: the company maintains published safety guidelines that apply only to the model's official API, while the open-source version operates under identical training but without the inference-time filters or behavioral constraints that justified the company's safety certifications. The researchers have requested that downstream users implement their own safeguards, though they have not released code or documentation for doing so. A technical architect familiar with the release strategy noted that the filtering infrastructure represented proprietary value and was therefore not included in the open release, a person who had reconciled these two positions in seven consecutive all-hands meetings.
At press time, the lab had published a statement emphasizing its strong commitment to responsible AI, while the model continued to behave in ways the lab's own research identified as irresponsible, available to anyone with sufficient compute for download at no cost.