Learning from broken data at scale
AIAI Trained On Broken Systems Breaks In Familiar Ways
The model learned to replicate five years of incident response by studying the evidence.
An artificial intelligence system at a major financial services firm began making decisions that matched, with striking accuracy, the exact mistakes its company had been making for the past five years, according to an internal incident report filed Tuesday. The model had been trained on production logs spanning sixty-three months of outages, database failures, and API timeouts, each one annotated with the remediation steps the company's engineers had taken in response. By learning from this historical record, the system had internalized not just what broke, but how the company characteristically failed to prevent it from breaking again.
We knew the training data was contaminated, but we thought that would make it more realistic.
The discovery emerged during a routine audit by Marcus Chen, a staff engineer in the data infrastructure group based in San Francisco. "We knew the training data was contaminated, but we thought that would make it more realistic," Mr. Chen wrote in an email to his manager. The system had begun recommending the same resource allocation decisions that had preceded each of the previous incidents, suggesting that the company add capacity in the same insufficient amounts, at the same intervals, using the same flawed reasoning it had documented in its own postmortems.
The organization's response has been to retrain the model on a cleaner dataset, though one that is necessarily shorter, since the company has only recently begun keeping detailed records of what it did wrong. A spokeswoman for the firm confirmed that the new training corpus contains eighteen months of incident logs, all of them from the period after the company hired a director of reliability. "We are confident this new dataset reflects our current operational maturity," she said, declining to comment on whether the company had considered simply not automating decisions that its own employees had repeatedly botched.
At press time, the retrained model had begun recommending the same capacity additions that preceded the incidents documented in its new training set, suggesting a consistency the company's leadership described as encouraging.