Balancing User Needs and Technology during Migration Planning

A step-by-step guide for prioritizing your users during a data migration

Posted on July 15, 2024 in Blog Posts

Authors:

Crissandra George
A librarian preparing for a large-scale data migration

Migrations are intimidating, time-consuming, and overwhelming. As with any big change, it can be difficult to balance a multitude of aspects: from technological requirements and system infrastructures to contextual considerations, such as organizational resources, users, and various other dependencies. A substantial amount of work is put into the premigration steps to aid in an ideally smooth migration and the implementation of a new system. Not to mention the post-migration work, which equally requires a lot of time, effort, and resources. Regardless of the time and effort put into all steps of the migration process, preparing for a migration requires a balance of not only purposefully investigating systems and technology, but also starting with users in mind from the first to the very last step before conducting the migration. 

Migration planning is the current task that I am facing at Case Western Reserve University, as well as other libraries within our state consortium. Outside of this consortial migration, many other libraries are finding themselves in the same position, especially if they have a system that is using Drupal 7, which is sunsetting in January 2025. Obviously, this puts a large sense of urgency on this task, but even with the rush of an end-of-life date, migrations rely heavily on users, both external users, who interact with the user-side of the platform, and internal users, such as librarians, technology staff who partake in the management of the system, and other administrative professionals. Internal and external staff guide many of the steps in preparing for a migration—from the selection of a new system to the features and functionality desired in the configuration of the new system—further demonstrating the significance of user research within the premigration process. Quite often, these aspects can get hidden behind required technological transitions and the ever-changing landscape of information systems, but if the people are forgotten in these aspects, the consequences can be detrimental. Purpose-driven migrations start with the users, and these users then guide the technology and, subsequently, all the migration planning steps that follow. 

Since I focus specifically on digital collections, the examples I use will be based on digital collections management systems and digital libraries, but I implore everyone to adjust this lens based on your specific circumstance and alter it as necessary.

Based on digital library resources on migrations that I have found to be useful (even if they refer to a specific platform) such as Wu, et. al (2020) and Crocken and Washington (2019), other literature within the library field, and in my own experience of currently going through these planning steps, I have found that breaking up the migration planning into several categories can make it more manageable. These categories, which largely mirror the articles above, are: 

  1. User Research and Assessment of Needs
  2. Contextual Considerations
  3. Digital Library Analysis
  4. Content Analysis
  5. Metadata Analysis
  6. Normalization of Metadata
  7. Content Migration
  8. Verification and Quality Control 
  9. Migration Resources. 

By no means do these steps have to happen chronologically. You will likely already know some aspects before others, but it is a good way to prepare and organize this large amount of information. For instance, steps 1–5 were completed while we were evaluating different platforms because a lot of the information from these analyses gave insight into the potential platform we were looking for, so reflecting on how these steps fit into your planning process is crucial. Again, I cannot stress enough the great assistance that Wu, et. al (2020) and Crocken and Washington (2019) provided in creating this list. 

Now, let’s dive deeper into each of these planning processes and how I have put them into action throughout this migration planning process. 

🌟 Subscribe to the LibTech Insights newsletter for weekly roundups and bonus content, including: 

User Research and Assessment of Needs

User research is critical to migration planning. This research will identify variables that will be used to evaluate each step of the migration from start to finish. Additionally, this research creates use cases that dictate the features and functionality that you will look for in a platform: needed analytics, display viewers, configuration of access, workflow processes, and more. Both internal and external user research will also help bring to the surface necessary contextual considerations that go hand-in-hand with the analysis of the current and future systems, the metadata, content types, and other migration choices. 

When beginning this step, I started with internal stakeholders across the libraries by sending out a survey that asked librarians and library staff their perceptions of the repository in its current state, what they would like to see going forward, and their overall satisfaction with the platform across multiple categories. This gave me the perspectives of librarians who help patrons use the digital library, technical staff who directly work with the administrative side, and collaborators who interact with the system throughout the workflow in a larger capacity. This greatly guided which functions were not working and needed improvement, such as the searching abilities of the platform, and general steps toward making digital collections available after digitization. By looking at both perspectives of interactions within the platform, it gave more specific insights that may have been ignored if the audience of the survey were smaller or only limited to users. 

With this information gathered, I then moved to various modes of collecting user information in collaboration with other librarians, such as the User Experience Librarian. We conducted a micro-assessment that brought to light different user reactions to the repository and showcased gaps in usability and visibility that I aim to fill with the migration. Paired with the micro-assessment data, other assessment analytics, and feedback from users, it was evident which current and future user needs were a priority for the potential new system. We also did a smaller-scale environmental scan to better understand the local, regional, and international community of users to fully ensure that we were making use cases that encapsulated all current potential users of the platform. This information was then used when researching and selecting a potential platform, in conjunction with several contextual considerations. 

Contextual Considerations

Contextual considerations require reflecting on the available technological resources at your organization and the type of system that is required for specific collections and user needs, including the size of your digital library and your current and potential user community. Additionally, understanding your users—from a community member to a student, staff, or librarian—is key. This determines the internal requirements for making content accessible online and how the material can be accessed. 

This part really challenged me to synthesize the user research and assessment information with the contextual aspects of my organization into concrete statements and priorities. I really put an emphasis on the abilities of the direct support staff of the repository in relation to the needs of our users. For example, all of our digital collections are openly available, so having the ability to restrict viewing to users was not as pertinent as it would be for a digital library that only allows registered users to access the content. Due to this, we also allow open downloading of all derivatives and metadata, so making sure these functions worked in a new system with enhanced features, such as a variety of formats for exporting the digital items and data, was imperative. Another issue I encountered was that our direct support staff is small, requiring vendor support for the system migration itself and for future configurations and custom developments needed for users. These are only a few examples of high priorities throughout the migration planning that combined a variety of both user research and contextual considerations, demonstrating this synthesis of gathered information. 

While taking into account these factors, we also made it important to note any administrative requirements for a migration, such as where final approval takes place—with a committee or the Library Director? Is there further approval needed from other stakeholders, such as IT or other university departments? This was especially important because many of my colleagues, and me, are new to the organization and the documentation for these processes was minimal. When taking into account the path of approval, it also exposed further internal stakeholder needs of leadership that we were unaware of, so having this input was extremely beneficial. 

Lastly, the migration and sustainable management of a library and information system are dependent on sufficient staff and collaboration across staff roles. Laying out these specific dependencies is crucial for ensuring that all necessary parties are aware, acknowledged, and present throughout the planning and implementation. This also ensures that each category is accompanied by an expert in the specific migration planning aspect, such as the inclusion of the Metadata Librarian in the metadata analysis and normalization process, to name one example. Laying out these roles also gave us the opportunity to gauge where more assistance or support was needed, whether it be content, metadata, migration, or technology. 

Digital Library Analysis

Analyzing the current system gives insight into infrastructure, size, and workflow processes that otherwise may be ignored when planning a migration. When analyzing this aspect of our system, we looked into our front end, back end, and indexing server. Seeing how all of these systems worked together and visualizing it with a schematic was key when explaining our system to potential vendors, but also when figuring out how to move our data. We also looked into the current size of our collections and their projected growth to gauge how much space we needed. Some platforms are limited by size, so it was crucial to have a concrete idea of how much would be migrated. 

By reflecting on our systems and their size, it was impossible to ignore how our workflows may be altered or adjusted in a new system. This brought up many discussions on how a change in the data model or systems during a migration could improve or alter established workflows and policies that we did not initially think about. 

Lastly, since we were working with a system that was nearing an end-of-life date, documenting the timeline based on the digital library specifications was extremely helpful for our own sake and for conversations with vendors. 

Content Analysis

Looking into the content that is currently made accessible through the digital library can give you a better understanding of what is needed in a future system. Investigating the amount and variety of content types will determine file requirements, viewer requirements, and even necessary processes. For example, do you have digitized materials that are nested under each other in the form of compound objects? This may require further post-migration processing or extra steps in the migration process due to the content type and format. Compound objects specifically have been a huge worry for us throughout the migration planning process and have required my team and me to investigate various ways to handle these complex parent-child object relationships. In addition to complex problems from system-specific objects, we expanded this examination across all objects, looking into how many books, pages, images, or audiovisual files we have and in which file type they reside, so we can have a larger understanding of the content and its location. Due to the hierarchies we currently have in place, we have been taking time to reflect on the relationships between the content and their organization (which in some cases contains even more files in XML format than as RELS-EXT files). 

Whether the collection hierarchy will remain the same or have to change is something that is best discussed earlier rather than later, because some systems do have limitations regarding hierarchical structures. At this stage, we are beginning these conversations and establishing how we want to address these adjustments. Examining the content and format has been largely the focus of many specifications that will need to be implemented during the migration to ensure access for users. 

Besides the structure of the digital objects and their collections, how the content becomes accessible is important to consider at this stage. What does the digitization workflow look like? Where do the reformatted copies reside before the ingest process, and will this have to be adjusted for the new system? Are derivative copies created upon ingest, such as a thumbnail, a file of a different quality (TIFF files may trigger the creation of JPG as well upon ingest), an additional metadata file in a different format, or more? Once ingested, where do these digital files live, and what does this storage location look like? All of these aspects need to be documented and considered throughout the migration process because these are the files and data that you will need to access to migrate to the new system. 

Documenting and planning these processes have been helpful for getting everyone to start thinking about how the migration will affect processes and collaborative workflows, especially related to the digitization and digital preservation storage locations. Thankfully, I am fortunate to have these colleagues working closely with me throughout this process to ensure efficient and proactive planning. All in all, analyzing the content does not just include the content types, but rather, how the content is made available, created, processed, managed, and stored. Making a schematic of this process and these locations is highly recommended and can help in communicating your technological needs to stakeholders who are not involved or familiar with the daily repository operations. 

Metadata Analysis

Ensuring the current system’s schema use, field requirements, controlled vocabularies, copyright, and metadata quality can save a lot of time during the migration process. New systems can sometimes mean changing to a new metadata schema, like MODS to DC, or using a different method for ingesting metadata into the system (such as Islandora Legacy’s use of TWIG Templates), so having this information handy can be a lifesaver. In my particular situation, previous librarians had created multiple crosswalks for our digital collections, so gathering this information was easy, but it helped to make sure we had it all in one place. 

Additionally, establishing field requirements will make the configuration of a new system or any adjustments much more manageable and consistent when migrating new objects. We have found this extremely helpful when testing out sandbox environments of platforms. This step also allows for reconsideration of field requirements down to the collection or content-type level, which can be very beneficial for current and future librarians who manage this system. Determining controlled vocabularies and copyright in the beginning can also factor into certain platform features and automated processes, so making sure you know how your current and future system will handle this is key for consistent data quality.

Some systems do allow for a crosswalk or configuration setting during the migration that directly maps a field (or multiple fields) from the old repository to a new field in the existing platform, which would allow any inconsistencies to be improved during the migration process. Other systems can also provide batch metadata editing capabilities for post-migration clean up, but sometimes it is preferable to make these changes prior to the migration, even if it can be time-consuming.

No matter which way is chosen to ensure consistent and improved data quality, it is crucial to document these metadata inconsistencies before the migration to serve as reminders for necessary changes that could be made premigration, during the migration process, or post-migration (if there are batch editing capabilities). Having this knowledge and understanding of how metadata operates within both systems is instrumental for planning the next steps in the migration process and how you plan to approach the normalization process.

Normalization of Metadata

Planning when and how to do metadata normalization can give you an opportunity to really dig into the timeline of the migration. Above, the mention of batch editing in the new system, but not the current one, can create a different scenario for metadata normalization than a system that already has batch editing. If your current system does have extensive batch metadata editing, then this could be a good time to make those revisions, especially if it will make the migration process smoother. On the other hand, if that is not the case, metadata normalization may have to wait due to time constraints. 

This is also a good time to look into tools that can assist in the metadata normalization process, such as metadata mappings that were made within your organization or other tools that are openly available. Some systems employ a searching server, like Apache Solr, that allows you to fully export data in a customizable manner, so knowing about these avenues for mass exporting metadata can also assist in the metadata normalization process. 

Additionally, OAI harvests and other avenues should be consulted in this process. Personally, I have been gathering all different normalization methods within one easy place to access, just in case we need multiple methods for different scenarios. This is a process that the Metadata Librarian and I continually discuss to ensure we are covering all our bases. We also have included our programmer in these discussions, because they have an extensive knowledge of scripts that could assist as well. While metadata normalization is definitely a complex process in the planning, this preparation will ensure that we can handle a variety of scenarios. 

This preparation also extends to file management and naming conventions. If this is plausible to complete beforehand, even better, but this is not always feasible due to the time-consuming nature of these steps. In the case where post–clean up will still be a large task, documentation of prior file naming, file management, and more are crucial to at least give context to the inconsistent content. Specifically, our Digital Preservation Librarian has been instrumental in documenting legacy naming conventions, which will definitely be helpful when migrating content over to the new system.

Content Migration

Migrating metadata is one step, but migrating the content is a separate process with many complexities. Knowing whether files can be migrated through an API or other processes can help guide migration choices. There is never just one way to migrate content, and sometimes you may have to use multiple methods to move the content to the new system. Some vendors will do this process for you, or it may be a full re-ingest of materials upon migration. Currently, this is something that is of huge interest to me within our migration planning as I document a variety of ways to migrate content, even down to granular levels. The reasoning for spending so much time on this is that there are many content formats and hierarchical levels within our digital collections. Compound objects are notoriously tricky to migrate due to these issues, so documenting multiple methods of migrations, from BagIt solutions to Solr exports with object links and APIs, ensures that I have many ways to attempt to migrate the content. 

Multiple methods for migrating content have been a huge topic of conversation at conferences like The Best Practices Exchange, which just recently occurred. Presenters talked about the struggle of having to get external hard drives with all their collections and metadata from vendors in order to migrate, while some moved them to various storage locations as an intermediary. Others were able to use easier solutions to quickly move the objects. Truly, no one migration is the same, which is why it is important to know all your options when it comes to migrating the digital objects themselves. 

Additionally, knowing the sources of objects and metadata of the current repository and how they map to the new one on a data model-level can be extremely important because not all systems operate the same way. For instance, the data model in Islandora Legacy vastly differs from the newest version of Islandora, Alma Digital, and others. Preparing the manner in which objects are migrated along with the path of the migration within the system ensures as much efficiency as possible and increases the migration’s success.

Content Verification

Content verification seems so far away during the planning process, but planning the approaches and metrics of content verification post-migration can give a baseline for what post-migration processing will look like. Setting up who will be a part of this process from the beginning will make sure that everyone involved is aware of their role. The verification process can look different for everyone. Some may choose to have multiple groups involved, while others may choose to limit who is involved. Either way, having this prepared can save a lot of wait time down the road. 

With our migration, we plan to include the main stakeholders in the digitization workflow for a much more specific and concentrated verification process that matches their position. For example, the Metadata Librarian will focus on the metadata format and display, while the archivists will focus on the specifics of the content. This is then broadened to others involved in the digital collections workflow for a more general overview of the system and migrated objects. Based on conversations with other institutions, some will split up the verification so 10 percent is reviewed and verified to confirm a successful migration of a collection, whereas others choose to include multiple groups to get a wider sampling. This is something that my colleagues and I continually discuss to see which would work best for our migration. 

Migration Resources

Lastly, compiling a list of potential migration resources from APIs, community forums, feedback from other organization’s migrations, and tools can decrease the amount of time spent searching for them later. Even just a simple list of these sources can be beneficial and even give light to issues you did not intend to occur. This is where I also like to add notes from other librarians who recently did a migration and the problems they encountered. It serves as a way to fully understand what could occur and the resources you have to fix it. I also have extensively compiled documentation from both the current and future systems and organized it into quick links to reference specific encounters. This has already been helpful in sandbox environments and even for discussions surrounding the specific steps of migration. 

And Now… Happy Migrating!

Overall, migrations are a large endeavor to encounter. Planning for them is even more time-consuming and tedious, but with these steps, it can become a bit more manageable. I am sure this is not the only migration planning resource available, but it is the one that has been most helpful to me throughout this process. It made me see the connection between users and a system migration, showing how purpose-driven migrations will always start with people—the users. From there, everything else is shaped around it. 

Having user research guide the selection and planning and examining contextual and technological considerations, the premigration process can be a great time to reflect on greater aspects of the digital library that may not have been explored otherwise. If anyone else is going through this same process, I wish them happy migrating! 

🔥 Sign up for LibTech Insights (LTI) new post notifications and updates.

✍️ Interested in contributing to LTI? Send an email to Daniel P. at Choice with your topic idea.

Related Posts