Academic libraries around the world are being hit by waves of bots that harvest data from the web. The traffic slows down, and sometimes crashes, the sites meant for students, researchers and the public, as AI companies look for material to feed their language models.
In the United States, the University of North Carolina at Chapel Hill reported last year more than 11,000 searches for Finnish music, arriving from thousands of different IP addresses on a single day. The library catalogue was repeatedly made unavailable to students, faculty and staff. The figure, reported by CBC, comes from the university itself.
The core problem is that these systems ignore the rules. Well-behaved bots, such as those from Google or the Internet Archive, respect robots.txt, the file in which a site states what can and cannot be read. AI training bots often ignore it and can request thousands of documents per second, downloading them in every available format. Blocking IP addresses no longer helps, because a single company or bot farm can operate from thousands of them.
Libraries are left with two unpopular options. CAPTCHA tests and verification systems also block legitimate bots. Login requirements put barriers in front of people, too, on material that should be easy to reach. Experts warn that more advanced bots could learn to get around these defenses.
The deeper issue is the value of the data. Much of the libraries' material is in the public domain, freely usable, and that is exactly what makes it attractive: vetted content that is usually not behind paywalls. Peter Lewis, a Canadian researcher on trustworthy AI, describes an extractive use of common-good institutions, with commercial models later sold back to the public.
Large AI companies argue that collecting publicly available data for training is lawful under fair use and in the public interest. Facing criticism and copyright lawsuits, however, OpenAI, Google and Microsoft have announced a single opt-out signal in robots.txt that would cover several providers at once. The standard is still being defined: the common rule and its dedicated token is not yet active, and Microsoft will apply it from February 1, 2027.
The libraries' case shows how urgently such a tool is needed, though it is not enough on its own: without a mechanism that crawlers genuinely respect, protection still depends on the goodwill of whoever collects the data.

No comments yet. Be the first!