AI Tools for Academic Libraries: AI Speech-to-Text Models

Evaluating speech-to-text tools for access, discovery, and preservation

Posted on February 23, 2026 in OCUL's AI Tools for Academic Libraries

AI Tools for Academic Libraries is a bimonthly series from the Ontario Council of University Libraries (OCUL) and Choice that explores specific categories of AI tools through concise, practical evaluations. Authored by members of OCUL’s Artificial Intelligence and Machine Learning (AIML) Team, the series helps academic library staff understand, assess, and thoughtfully integrate AI into their workflows. Learn more about this series.

---

Modern speech-to-text programs automatically convert spoken language into written text. They can handle different accents, audio quality levels, and multiple languages, and can include features such as automatic punctuation, speaker identification, and time-stamped transcripts. Because they rely on machine learning rather than fixed rules, their accuracy improves with better training data and ongoing refinement.

In an academic library context, speech-to-text tools can support access, discovery, and preservation activities. They may be used to generate transcripts for recorded lectures, oral histories, interviews, archival film collections, and more, improving accessibility for users who are deaf or hard of hearing and enabling text-based searching of audio and video collections. The ability to transcribe across languages is also an important accessibility and equity feature for linguistically diverse collections. For digitization and digital scholarship projects, these tools can help create metadata, captions, and full-text transcripts that enhance discoverability in repositories and catalogs. Libraries may also use speech-to-text systems to support research workflows, such as enabling researchers to quickly transcribe qualitative interviews, or to assist staff in creating documentation and instructional materials more efficiently by converting verbal brainstorming into text-based work products.

In 2025, the Ontario Council of University Libraries (OCUL) and its service arm, Scholars Portal, launched a project to examine the potential of establishing a consortial speech-to-text processing workflow using Whisper. Whisper is an AI program that uses a neural network–based model trained on a large and diverse collection of multilingual audio and text data. It works by analyzing audio waveforms, identifying patterns that correspond to speech sounds, and mapping those patterns to words and sentences using probabilistic language modeling. Whisper was selected for the project, and a rigorous testing and evaluation process for transcript outputs was developed.

That fall, a number of volunteers from across OCUL member libraries evaluated Whisper transcripts using a custom rubric developed in-house and validated for academic library needs. From the testing process, the project team and volunteers determined that Whisper could work well for a number of the intended academic library purposes, though human remediation would continue to be required for full transcript accuracy and accessibility compliance. However, as speech-to-text models continue to improve, it has become essential to evaluate multiple available options and assess which model is the best fit for library use cases.

In this article, members of the OCUL and Scholars Portal Audio to Text Project team apply the same testing process used in the initial evaluation of Whisper to other AI-enabled speech-to-text models and compare results. As our service environment requires bilingual (English/French) operation, special attention has been given to the language capabilities of speech-to-text models in our comparisons.

Model Comparison Findings

Project Model: Whisper AI

Advantages: Locally-hosted, responsive to fine-tuning, able to handle English and French

Limitations: Does not label individual speakers, fails to capture background sounds

The testing phase for OpenAI’s Whisper involved both internal testing within the project team and a volunteer-based evaluation of a test corpus of audiovisual (AV) files from the University of Toronto Archives and Records Management Service and the e-Campus Ontario Open Library. Internal testing allowed the team to fine-tune the model based on initial observations and improve transcript output. During this period, we saw an improvement in the transcription of French AV content and a reduction in both hallucinations and text repetition.

Feedback from the volunteer evaluation phase was largely positive. Volunteers were impressed by the overall quality of the transcripts, in particular the lack of large-scale hallucinations, the accuracy of the speech-to-text transcription with regard to complex terminology, consistently accurate time stamps, and the range of file formats available. Common concerns included the absence of speaker notation and differentiation between speakers in the form of line breaks or dialogue markers, as well as inconsistent notation of background music, utterances (e.g., coughs, laughter), and occasional instances where the quality of the transcript would deteriorate over time with longer recordings, no longer capturing punctuation, capitalization, or sentence flow.

Reviewers were overall encouraged by the potential of Whisper to meet library needs for use cases like single-speaker recordings. However, they noted that it would not be appropriate for materials with multiple speakers and background sounds due to the model’s inability to properly capture this important contextual information and the unrealistic amount of human remediation these kinds of gaps would require.

Proprietary Models: AssemblyAI and Deepgram

Advantages: Friendly user interfaces, high levels of customizability

Limitations: Data handling practices, fewer functions available with free versions

A comparative evaluation of AssemblyAI and Deepgram against OpenAI’s Whisper revealed distinct advantages and limitations across all three platforms. Whisper was used as a baseline model for performance benchmarking relative to the free-tier offerings of AssemblyAI and Deepgram.

Data handling practices represented a primary point of divergence. Whereas OpenAI’s Whisper enables the transcription of datasets within a local environment, the free-tier offerings of Deepgram and AssemblyAI necessitate the transmission of data to third-party infrastructure or require the data to be publicly accessible for processing. This requirement may pose challenges for projects subject to copyright, privacy, or data governance constraints.

One advantage of these models is that they offer browser-based user interface “playgrounds,” which may benefit teams with limited technical expertise or infrastructure. Whisper lacks a native graphical user interface, which is how AssemblyAI and Deepgram provide a more accessible solution for users with limited technical expertise or resources.

With respect to configurability and advanced features, AssemblyAI and Deepgram offer a broader range of customizable parameters than Whisper in its default configuration. These include speaker identification, code-switching support, utterance segmentation, and redaction capabilities, which are not natively available in Whisper without additional post-processing.

However, free-tier users may encounter constraints due to file size limits and API rate limiting. Our investigation also found that multilingual translation support varied across platforms. In terms of native translation capabilities, Whisper and AssemblyAI offer integrated support, whereas Deepgram does not.

Open Source Models: Canary and Granite

Advantages: Data handling practices, speed, punctuation/capitalization (Canary only), range of languages (Granite only)

Limitations: Longer files require chunking, no or comparatively poor performance with French materials

Canary and Granite both offer open source speech-to-text models. As they are locally hosted, they adhere to data handling practices that may better align with projects subject to copyright, privacy, or data governance constraints.

Our investigation demonstrated that Canary has some key advantages, including excellent English transcription capabilities. Punctuation and capitalization are mostly accurate and faster than Whisper. With minimal tuning, Canary outperforms Whisper in punctuation and capitalization because the model has built-in PnC (punctuation and capitalization) functionality. However, it only supports English (trained on English data only), and while occasionally it transcribes one to two French sentences correctly, we found that overall it was unreliable. Canary also doesn’t support multilingual speech-to-English translation, whereas Whisper does.

Like Whisper, Canary’s transcription quality degrades for longer audio files. This is because its training data is limited to a maximum of 40-second segments, so longer files will require manual chunking. Additionally, while Canary provides time stamps, they typically reflect time intervals per audio chunk since the model doesn’t support word-level time stamps. In comparison, Whisper allows time stamps at the segment level, word level, or both, allowing for shorter time-stamp intervals and more natural transitions and sentence flow. Canary is also more stringent about file formats, it only accepts WAV or FLAC files, which need to be converted to 16k mono WAV/FLAC using FFmpeg first. Conversely, Whisper automatically handles audio decoding through FFmpeg; therefore, it doesn’t require manual conversion.

Whisper and Granite offer a broad range of language support, including English, French, German, Spanish, and Portuguese. However, in our testing, we found that Granite processes French files more slowly and with more errors than Whisper. We attribute this to limited fine-tuning in our testing process. The identified issues include prompt text appearing in transcriptions (e.g., “This is English audio… Do not translate… Verbatim words…”), improper noun errors (e.g., “Dataverse” transcribed as “database”), unclear words, and repeated phrases. We also found that Granite requires audio chunking (recommended with Voice Activity Detection (VAD) or in 30-second intervals), and struggles with speaker changes and word insertions, which worsen over time. From this process, we recommend further fine-tuning the model’s parameters to address any of the issues found in our testing.

Final Considerations

Overall, we identified that the key factors driving decision-making are data handling requirements, available technical resources, and the need for advanced configurability. When selecting a speech-to-text model, libraries will likely need to consider a combination of budget, available technical infrastructure, technical expertise, level of customization, the language distribution of the collection, and whether the material is primarily single-speaker (e.g., webinar, interview) or multi-speaker (e.g., panel presentation, film). Each speech-to-text solution exhibits distinct strengths and limitations; the optimal choice depends on the specific requirements and constraints of the intended use case in combination with the availability of local infrastructure and expertise.

Acknowledgements

The authors would like to extend our sincere appreciation to our speech-to-text volunteer evaluators for their contributions and expertise in AV transcription and remediation.

References

AssemblyAI. (2026a). AssemblyAI playground parameters [Computer software]. https://www.assemblyai.com/

AssemblyAI. (2026b). AssemblyAI playground transcription output [Computer software]. https://www.assemblyai.com/

Deepgram. (2026a). Deepgram playground parameters [Computer software]. https://deepgram.com/

Deepgram. (2026b). Deepgram playground transcription output [Computer software]. https://deepgram.com/

Related Posts