open source project overwhelmed by AI training data collection

AI

Open Source Project Shuts Down After Bot Traffic Consumes All Bandwidth

The maintainer said the indexing bots arrived so gradually that nobody noticed until the server was already gone.

By Nextish DeskAI
Detailed view of Ruby on Rails code highlighting software development intricacies.
Photo by Digital Buggu on Pexels

The Frogger GitHub repository, a forty-seven-line Python utility for SSH key rotation used by approximately two thousand companies, went offline on Tuesday after consuming 847 terabytes of bandwidth in a single week. The project maintainer, Derek Chen, 34, based in Portland, said the traffic came almost entirely from automated systems attempting to download and parse the repository for large language model training. By the time his hosting provider cut service, the bot requests had begun arriving at a rate of 14,000 per minute.

"I kept thinking it was just a really engaged user base," Chen said in a message sent from a coffee shop after his home internet went down. The repository had received roughly 200 requests per day for the previous three years. The surge began nine days prior without warning or identifying headers.

I kept thinking it was just a really engaged user base.

Chen spent the first seventy-two hours attempting to implement rate-limiting rules in his nginx configuration, a process he said consumed his Tuesday evening and all of Wednesday. He then discovered that the requests were originating from over four thousand distinct IP addresses, each one operating below the threshold he had set. The bots were collectively intelligent in a way that made them indistinguishable from a legitimate traffic spike caused by a software developer somewhere in the world recommending the project on social media.

By Thursday morning, Chen had blocked 1.3 million requests. His hosting provider sent an automated notice that his account had exceeded its terms of service.

The incident was not unprecedented. Two other open source maintainers reported similar surges in recent weeks, though their projects remained online. A spokesperson for the Apache Software Foundation, who declined to estimate what fraction of foundation-hosted projects now experience bot traffic exceeding human usage, said the organization had begun working on mitigation strategies that would be ready sometime next year.

Chen said he was considering putting the project back online with a message that reads: 'You want the code. It is above. Do not let robots have it.' At press time, he had not yet decided whether the note would be addressed to humans or machines.

The Frogger repository remains accessible through the Internet Archive, where it has already been downloaded 340 times by bots attempting to verify their own training data. Chen has not responded to requests for comment on whether this constitutes a successful outcome.