Benchmarking AI Models in the Library: Critical Questions for Library Professionals and Users

How to select the best model for each job

Posted on November 24, 2025 in Blog Posts

AI literacy feels like a natural extension of information literacy. Like information, AI is now everywhere; it can be hard to discern fact from fiction, and we cannot always protect our users from poor-quality versions of it. Moreover, even the best of us are not immune to a well-organized AI scam. We can work to vet best-fit AI tools and resources for our libraries, but our users and library teams will still inevitably engage with freely available and built-in AI tools.

It is challenging to directly compare most AI tools apples-to-apples because the underlying corpus, training process, and intended uses can vary widely. However, practical benchmarking questions can help librarians compare models to one another or be used to guide users through the reference interview in order to select the best model for the task at hand. The goal isn’t to gatekeep AI use, but to empower thoughtful, informed engagement. Drawing on my own process of comparing a variety of AI resources and tools, I’ve distilled the benchmarking process into seven straightforward questions that can be compiled into a rubric or reference interview to determine if the AI tool under consideration is the best one for the job.

Is the tool free or licensed?

While not a perfect yardstick by which to measure all AI resources, tools paid for by the library or directly by an individual user may come with unique features, guarantees, or applications to accomplish the task at hand. Ideally, users will carefully review the terms and conditions for any model with which they engage. Otherwise, users should be encouraged to review the guarantees and disclosures for their selected model, as well as note where the free version of a particular AI product might differ from the paid version.

Depending on the goal librarians and users are hoping to accomplish, a freely licensed model might suffice for quick questions, short searches, simple analyses, and creating limited quantities of images or documents. However, users who need to process large amounts of data, generate a significant amount of content for export, or accomplish a difficult task might look at a paid model or dedicated AI tool so their engagement is not throttled or slowed down. Users of paid models might also be awarded additional benefits for their subscription, such as early access to improved models or more tailored responses to simple questions.

---

🔥 Stay up-to-date with LibTech Insights by signing up for our free newsletter. Just one weekly email with our new blog posts, top tech news stories, and other bonus content. Check out some posts from our archive:

• Using Notion to Organize Library Projects
• Top 5 Tips for Librarians Creating Surveys
• Keeping Up-to-Date on New Scholarship with OpenAlex

📹 Register now for our upcoming webinar, “‘Data’ and Indigenous Studies: Entry Points for Academic Libraries”

---

Can the user select the tool’s level of reasoning?

Along the same lines of selecting the right model for the user’s task, it is beneficial if models allow users to select the level of complex reasoning or effort dedicated to a model’s response. Many mainstream general-purpose models now include three levels of reasoning to help deliver the right caliber of response to user inputs: a basic conversational search for straightforward questions and answers; a thinking mode where the model takes more time to reason through a question and its subsequent answer; and a deep research mode where the model dedicates significantly more time and computing power to fully break down user inputs to deliver a report-like output, complete with sources.

For most standard AI searches and summaries, thinking and deep research are overkill. Combined with uncertainty about AI energy use and sustainability, librarians and users should responsibly choose the right level of reasoning for the job, especially because more complicated requests require more time and energy. However, thinking and deep research both have a place in the user’s arsenal for complex logical reasoning and problem-solving, assembling and organizing information on niche topics, and automating manual tasks, such as literature reviews.

Does the model prompt the user for more information?

In full disclosure, ChatGPT is the AI model that I use and prefer to all others on the market. This is not to say that it is a perfect model or that OpenAI is a perfect company. Simply, I find value in ChatGPT’s follow-up questions for conversational and complex queries alike. I do see more tools on the market now that ask clarifying questions to help improve user outputs, but there are still many tools that prioritize quick responses over more accurate responses.

I recently tested two AI models added to my library’s resources against ChatGPT in deep research mode to ask about current and forecasted trends in the cosmetics industry. The two new library resources got to work right away, while ChatGPT asked for clarifying information on the geographical and customer market to investigate, if there was a specific subset of cosmetics it should focus on, and whether current events, such as tariffs, should be considered for its analysis. Consequently, the ChatGPT report was more tailored for my (testing) needs, while the two library resource reports were very broad and generic. If I were a standard library user, initial clarifying questions might provide more accurate results to me faster than going back and forth with the two library resources to improve my information until I was able to whittle down to the right market, cosmetics subset, et cetera.

Does the tool share its reasoning process with the user?

While not ubiquitous, some AI tools display model reasoning and decision explanations—called Chain-of-Thought—to users as a response to a complex prompt, shedding light on the step-by-step information compilation process, external resources that might have been consulted, and decisions made along the way to create the output for a query. Standard users may not benefit from these long, detailed explanations, but those relying on AI for complex projects or iterative tasks might benefit from reviewing a model’s decision process.

For example, a user providing an AI tool with a Python script to update might be surprised if the script returned to them has numerous changes. Without access to the model’s reasoning, the user can ask the model why a decision was made to use a particular package or rewrite part of the code, but the jury is still out on the trustworthiness of this type of exchange. Chain-of-Thought scripts are still thought to be somewhat inaccurate, but they can provide users a forensic starting place to unpick a seemingly weird AI decision. Our coding user above might dig through their script and see that one of the packages in their code conflicts with a different package since they wrote the original script, thus resulting in the AI tool’s code changes.

Can the user review the sources that contributed to the model’s answer?

AI literacy is both an extension of information literacy and a locale where information literacy must be applied. Simple conversational AI rarely provides its sources to users, so it can be challenging to determine where it sources its information, much less how accurate that information might actually be. This is alarming, especially when we consider how widespread AI is and how users might be incorporating AI into their decision-making. As library professionals, we cannot curtail AI adoption or use, but we might be able to encourage users to seek out models that cite their sources so users can dig deeper on their own terms.

I often liken the AI deluge to the early days of Wikipedia. When Wikipedia was less regulated, educators and librarians discouraged the use of it as a research source, and instead directed researchers to use it as a starting point on the research journey. Anecdotally, students who were trained that Wikipedia should not be the sole or primary source for their research have grown into adults who might look to Wikipedia as a starting place and then jump to the additional sources linked in Wikipedia articles for more authoritative information. Similarly, I think there is an opportunity with AI to train users to contextualize AI and to reach for AI tools that natively reveal their sources. If we start training librarians and users now to seek out more transparent models, we have the opportunity to influence future AI tools and practices alike.

What is the quality of the included content?

Quality of information can be difficult to determine in an AI tool. Conversational outputs do not typically cite sources, so detailed facts can be tricky to confirm, and AI tool vendors may not disclose the underlying training data or knowledge base for their product. The best way for users to determine the quality of resulting outputs is to test AI models on topics that the users themselves are self-identified experts in before investing too much time and trust into a particular model.

Librarians and users can test models on anything from their favorite childhood TV show to a niche hometown band to family heritage to a nonsensical, made-up recipe to better understand how their selected model handles unexpected inputs. Good models will inform users that they’re not familiar with a particular concept, while less-ideal models will generate a response that sounds right but is not backed by legitimate facts or sources. Ideally, all AI models would cite their sources, so users could then determine if those sources were accurate and meaningful. In the absence of cited sources, though, testing how a tool reacts to something that it is unlikely to know is a quick way to qualify AI responses.

Is recent content included in the model’s answer?

While not applicable to all questions asked of AI, currency may play a big role in the accuracy of response for some applications. Users relying on an AI tool for simple explanations, brainstorming, or personal planning will likely not need a model to be updated with the latest news and information. However, including content from the web or incorporating the latest information into the model itself will benefit those relying on AI to describe rapidly developing trends, contextualize complex topics in current events, or capture the latest research developments.

Consider AI itself as a research topic. If a user wanted to analyze AI incorporation into household devices for a possible investment opportunity, they would benefit from the latest consumer news on AI in household devices, information on the current state of the stock market and investments, global context on AI investments, and an understanding of how trade tariffs might impact their investment. With the AI market developing rapidly and continually, time is of the essence. Comparatively, a user creating a packing list for an upcoming vacation would be able to successfully plan for their trip with generalized information provided about typical weather patterns, suggestions on what one might pack for five days and four nights in a tropical climate, and airplane cabin essentials without needing to understand the to-date global context of the travel industry.

*

Generative AI is here to stay. We cannot fully prevent bad AI or misguided applications from seeping into our lives, but we can work to make AI model benchmarking a natural extension of AI adoption and information literacy alike. Integrating comparative practices into our daily work doesn’t just strengthen our own discernment; it also empowers users to engage more critically and confidently with AI tools. Benchmarking may not be a silver bullet, but it does provide an opportunity for librarians and users to universally pause and consider their sources. 

Related Posts