AI Tools for Academic Libraries: AI Metadata Extraction

Evaluating tools for extracting and enhancing metadata

Posted on June 8, 2026 in OCUL's AI Tools for Academic Libraries

AI Tools for Academic Libraries is a bimonthly series from the Ontario Council of University Libraries (OCUL) and Choice that explores specific categories of AI tools through concise, practical evaluations. Authored by members of OCUL’s Artificial Intelligence and Machine Learning (AIML) Team, the series helps academic library staff understand, assess, and thoughtfully integrate AI into their workflows. Learn more about this series and read past installments.

Note: This installment of the series describes work that took place from September 2025 to March 2026.

---

Academic libraries are home to collections that span different time periods, languages, and formats. Making these collections accessible requires extensive time and specialized knowledge, and when these are in short supply, valuable resources may go under- or undescribed for long periods of time. Is there a role for artificial intelligence tools in catching up on cataloging, or doing a first pass at describing resources that are without metadata? As part of the OCUL AIML Initiative, the Government Documents Project sought to explore this question using a large collection of scanned government documents.  

In 2021, the University of Toronto and the Internet Archive began a mass digitization project of government documents, which were scanned and made available through the Internet Archive, and then moved into long-term storage. Across these 50,000 documents, there is significant variation in the completeness of the attached records. While some are associated with a full MARC record, many documents hosted on the Internet Archive have only a barcode for a title. While they are full-text searchable, uneven OCR quality and the sheer volume of documents make both known-item searching and discoverability challenging. This seemed like fruitful ground for OCUL’s work and the team led by Pieter Botha, OCUL’s AIML Technical Lead, and Jacqueline Whyte Appleby, Associate Director at Scholars Portal, with support from the OCUL Government Information Community, which has extensive expertise in the resource type.

This blog post offers a brief overview of several open-source generative AI models that were tested as part of the Gov Docs Project, specifically their competence in metadata extraction and how they might apply to different library or metadata needs.

Our process

All of the documents in this project were first run through an OCR tool, Chandra, and the results were saved in markdown. It was this markdown that was used to generate metadata, and the metadata outputs were only as good as the OCR inputs. The quality of the OCR is an essential consideration for all metadata generation work, although it might become less applicable in some contexts as vision models improve. 

Once we were satisfied with the markdown, we created a prompt, or set of instructions, and sent it to an AI model along with the markdown. Because most metadata elements come from the front matter, and many of these documents were hundreds or even thousands of pages long, we did not send all of the markdown; instead, we chose to send approximately the first 20 pages of each document (~7,500 tokens) as well as the last 10 pages of the document (~2,500 tokens). In the context of Canadian government documents, these last 10 pages were important because many of our documents were bilingual, with one official language in the first half of the text, and the same text in the other official language on the verso, i.e., the last 10 pages might be the first 10 pages of the French equivalent.  

When evaluating the metadata output, the most important factors to consider were accuracy and consistency. Almost all of the information needed for a metadata record is present in the documents, and for fields such as “publisher” or “ISBN,” we needed the AI models to look exclusively at the text they were given. For fields such as “summary” or “keywords,” there was more creativity involved, but again, these outputs needed to accurately represent what was in the text.

With such a large batch of documents, there were many series and subcollections, so we wanted to ensure consistency across similar documents. If a model processed 50 volumes of the same financial report, all of which looked very similar to a human reader, we wanted to see that the metadata reflected the cohesiveness of the subcollection. A model that gave excellent results 85 percent of the time, but big surprises for 15 percent of documents was less successful than a model that gave good or very good results 95 percent of the time.

About prompting

The capabilities of AI models are dictated by more than just their source data, algorithms, and training. As users of this technology, we can improve the quality of their responses through the development and refinement of the instructions we give them, usually called prompts. Each AI model has different “preferences” for receiving and executing prompts, so testing, evaluating, and adjusting prompts were a significant part of our work when selecting the right tool for our project. 

The longer the prompt, the more tokens it requires to run and return results, which has both financial and time costs, so balancing the level of detail with prompt size is an important piece of constructing prompts. We also found that while some models responded well to very explicit instructions, others could infer things and, indeed, gave more straightforward answers with fewer directions, so reducing prompt size had multiple benefits. Put simply, the models’ outputs were highly dependent on prompting (both the prompt itself and its alignment with the model’s preferences). 

Model findings

To illustrate the different ways models handle the same task, we will use “Nisga’a Land Claim Negotiations,” a pamphlet published by the Government of Canada, the Province of British Columbia, and the Nisga’a Tribal Council in 1992. This document details what negotiations were happening around the Nass Valley, why they were happening, and how each stakeholder group, including the public, could get involved. Pamphlets and small brochures make up about 5 percent of the collection, and they are an interesting example for models to use! They are usually smaller publications, so the model typically gets the full text to handle.

The following examples will showcase how each model handled the “keywords” field, since different library use cases could benefit from the different ways models approach instructions and output answers. Whether the desire is to enhance search using specific keywords or to provide a general sense of the subject matter, there is a model that could fit any scenario!

Qwen3 Models (4b, 30b, coder) 

• Pro: Good at following instructions and providing consistent results
• Con: Detail in responses ranges 

We tested a variety of Qwen3 Models, including the 4b parameter, 30b parameter, and coder models. All of these models preferred labeled categories and examples in their prompts (often called “few-shot prompting”), and the results they provided were fairly consistent.

The 4 billion parameter model (4b) impressed us initially with its speed and accuracy, but we found that this accuracy diminished as we introduced new document types. For example, early testing batches mostly included documents where the author name was preceded by the word “By,” but as we moved into other kinds of reports, we found it struggled to infer authorship without that designation. 

The 30 billion parameter model (30b) had similar issues and, in particular, seemed to struggle with identifying the language of a document. Our prompt instructed the model to identify languages used in a significant portion of the document, but the model often tagged languages that were only used in a single word or sentence. 

Qwen3-Coder performed the best of the three and was very consistent. One area where it struggled was in distinguishing an author from a publisher, which is challenging in the government documents context, but which some other models seemed to understand better.

All Qwen3 models wanted to use our prompt examples as outputs when they couldn’t find something, and it was through testing these models that we learned to drop a lot of examples from our prompts. 

Qwen3.5 Models (2b, 4b, 9b, 35b) 

• Pro: Consistently faster models
• Con: Requires additional prompting, especially for new document types 

We tested a variety of Qwen3.5 models, including the 2b, 4b, 9b, and 35b parameter models. When we began testing Qwen3.5 models, we initially started by reusing the prompts that worked with Qwen3 models. However, it quickly became clear that the Qwen3.5 models needed a lot of additional prompting to achieve the same results as the Qwen3 models. 

Both the 2 billion parameter model and the 35 billion parameter model would “think” too much—hallucinating and adding extra fields, or adding unasked-for explanatory text to results. 

The 4b model cast a wide net for results, which is not an inherently negative quality, but in our case, it affected the accuracy of the records. For example, when the “University of Toronto Libraries” stamp would appear in the OCR, 4b would include the “University of Toronto” as a “responsible entity” for producing this document. Most other models seemed to understand that the University of Toronto, not mentioned anywhere else in the document, was not an appropriate result.

We also found that the 2b and 4b models frequently ignored formatting instructions for fields such as publication date and ISSN.

Of the models we tested, the 9b had the most consistent results while being conscious of semantics, but it frequently lacked detail in fields such as “summary,” “keywords,” and even “responsible entities.”

Mistral Models (3b, 8b, 14b) 

• Pro: Very smart (picks up information) 
• Con: For our purposes, too creative. Hard to instruct

We tested Mistral 3b, 8b, and 14b. All of the Mistral models were relatively good at making inferences. For example, Mistral models were more likely to identify that a “working group” named in the front matter counted as a responsible entity. 

We found the 8b model to be the best at this kind of inference and association, but unfortunately, it struggled with some essential instructions. This model was the most likely to make up a title or add descriptive words to a title, and it frequently ignored instructions around the length or structure of the output. It made for a very difficult model to instruct, which is particularly unhelpful in the context of metadata extraction. For the 14b model, we could not convince it to return keywords or phrases that were a maximum of two words—it always wanted to create longer, more creative phrases! Finally, despite many iterations of both positive and negative prompts, all of the Mistral models continually resisted returning empty fields, although this frequently would have been appropriate.

GLM4.7 – Flash  

• Pro: Fast model, good at following instructions 
• Con: The more detail, the less quality of results (struggled with longer prompts) 

GLM4.7 Flash is a fast, compact model that worked particularly well with smaller prompts focused on 1-3 fields, which was not how we typically worked. With our larger prompts, we found that GLM4.7-Flash produced weaker, less detailed responses compared to other models.

Other models 

Some models simply did not perform well enough for our project use case to continue past preliminary testing. The testing of these models occurred in the fall of 2025, and, as with the results above, it is likely that some of these models have been changed or improved since then. 

Gemma3

• Pros: Worked well when information was explicitly labeled, relatively consistent  
• Cons: Merged fields together, required a lot of instruction for complex and nuanced document structures  

Granite4

• Pros: Good with patterns and standard sets of documents (e.g., 50 reports that have been structured the same way) 
• Cons: Missed content that was not explicitly labeled 

Phi4

• Pros: A very logical model that was good with semantics (e.g., differentiating author and editor) 
• Cons: Omitted a lot of detail and information without reason, consistently did not return some fields

Considerations

The integration of artificial intelligence tools into library metadata workflows requires significant consideration and ongoing evaluation. Different collections and use cases will naturally have different requirements for accuracy, consistency, speed, compute, and structure of output. Prompt engineering makes a vital difference in the quality of output, but the myriad of models available also all have their own inherent strengths and limitations, which are continually changing. Hearing from colleagues about their own experimentation and findings has been invaluable to us. While the specifics of our evaluations will likely become outdated quickly, we hope this overview of our findings may likewise be helpful to others. 

Acknowledgements 

The authors would like to extend their gratitude to the OCUL Government Information Community for their expertise and advice over the course of the project. Our results would not be what they are without your support. 

Related Posts