Protecting Library Websites from AI Bot Traffic

Bots scraping library websites for LLM training materials are causing big problems

Posted on February 26, 2025 in Podcasts

This month on Choice’s The Authority File podcast, Bill Mickey chats with Rosalyn Metz, Chief Technology Officer for Libraries and Museum at Emory University, about a frustrating new issue.

At a 2024 Charleston Conference panel, Rosalyn shared a quick anecdote on how spikes in AI bot traffic on Emory library’s website have led to service shutdowns. Now, Rosalyn joins the podcast to dig into this frustrating issue, exploring how her library has dealt with this traffic and the ethical implications of making a library’s catalog and metadata openly available on the Internet in the age of AI.

To start, Rosalyn explains the level of traffic the library has experienced in the past year and how staff have worked to distinguish between regular bot traffic and large language models crawling for content. Further, she chats about the protections her library and other institutions have instigated, whether that be turning to campus IT, seeking out vendor partnerships, or locking down library webpages completely. Sharing her concerns beyond interrupting site traffic and the user experience, Rosalyn discusses the implications of AI bots scraping unvetted institutional files and sensitive repository materials.

Listen now:

🎧 Like what you hear? Subscribe to The Authority File podcast on Apple Podcasts, Spotify, or YouTube.

---

About the guest:

Rosalyn Metz
Chief Technology Officer for Libraries and Museum
Emory University

Rosalyn Metz is the Chief Technology Officer for Libraries and Museum at Emory University in Atlanta. She leads a diverse cross-functional team that enhances technology infrastructure and digital strategies to support the university’s research and learning missions. With extensive experience in open-source library systems, she has played a critical role in implementing platforms like Fedora, ArchivesSpace, Blacklight, and other open-source projects within cloud-based environments.

Rosalyn has been actively involved in several open-source software communities, serving in leadership roles in Samvera, Fedora, and the Oxford Common File Layout (OCFL) Editorial Group. She is passionate about fostering collaboration within the library technology community and has worked on governance initiatives to create sustainable models for open-source projects. Her expertise in digital infrastructure, open infrastructure, and metadata management continues to shape the evolving landscape of academic libraries in the AI era.

Related Posts