August 2023 – NDSA Blog

Catching up with past NDSA Excellence Awards Winners: Project Electron

In 2020, Project Electron received the NDSA Innovation Award in the Project category. It impressed the awards panel with its comprehensive adaptation and extension of traditional archival principles and workflows to digital materials. A multi-year initiative at the Rockefeller Archive Center, it sought to implement sustainable, user-centered, and standards-compliant infrastructure to support the ongoing acquisition, management, and preservation of archival digital records. The panel also appreciated the positioning of this initiative as an open-source and standards-based effort. This would allow maximum opportunities for its transferability to other programmatic contexts in a time when many archival institutions face significant challenges in supporting digitized and born-digital records and special collections. Headshot for Hillel Arnold

We contacted Hillel Arnold and found out how Project Electron has evolved and learned how it has impacted other work.

1) What have you been doing since receiving an NDSA Excellence Award?

We operationalized Project Electron in August 2019. Since then, we’ve continued to build on the infrastructure, methodologies, and expertise we built during the project.

One of the next big pieces of work we undertook was a complete rebuild of our discovery environment, locally branded as DIMES. The knowledge about building event-driven pipelines we picked up from Project Electron shaped this project, and also allowed us to complete it relatively quickly under challenging circumstances during the COVID-19 pandemic. You can read more about this project in the blog post announcing its launch.

Once we’d done all of that, the number of applications we needed to maintain had grown significantly, so we’ve also spent a fair amount of time improving our maintenance chops. As many folks know, I feel strongly that maintenance practices are both a core part of technology work as well as an enabler of strategic initiatives. We refined our continuous integration and continuous deployment pipelines, and instituted regular, efficient processes for dependency management. This continues to be an important area of focus for us, as we look for ways to implement DevOps methodologies and tools which help to expand our pool of developers among our colleagues.

Finally, we’ve continued to invest in building a User Experience and Accessibility program to support ongoing evaluation and improvement of these systems.

2) What did receiving the NDSA award mean to you?

For us, the Innovation Award was important because it provided external validation not only of what we were doing, but how we were going about it. It felt really good to know that other people saw value in what we were doing, even if they weren’t going to use our code, or if a completely different approach made sense for them. It also verified that our project values had been instrumental in directing us towards solutions that were reproducible and based in archival standards and practice. Most of all though, the award reaffirmed our participation in a community of digital preservation practitioners, which is incredibly important to the project team as well as the Rockefeller Archive Center as an institution. We have a lot to learn from each other!

3) How has Project Electron evolved since you won the Excellence Award?

As I mentioned above, Project Electron was an important platform for us in a number of ways. On the systems side, the approach we took has allowed us to extend the existing infrastructure to support additional workflows. So far we’ve built additional services to support a data pipeline for archival description, creation of image derivatives and IIIF manifests, and we’re in the process of building out services to support the ingest of digitized AV content. We’ve also spent some time improving the infrastructure’s scalability so we can process large files and large packages of files.

Going into the project, we always knew we would have to build more than just software applications. So, we’ve also spent a significant amount of time building a user community around the tools and working to support adoption across our donor organizations. In many cases this has involved implementing other tools such as DART to support the creation of BagIt bags by organizations.

We’re also trying to find ways to support records management processes in our donor organizations, since having an empowered records management function is key to successfully onboarding organizations because they are an effective way of mitigating concerns many of our donor organizations have about risk management. We’ve taken some broader approaches in this area too, such as spinning up a Records and Information Management Program to support these efforts, as well as establishing the Advancing Foundation Archives conference and community.

4) What do you currently see as some of the biggest challenges in digital preservation?

Climate change. In order for the work of digital preservation to be useful, there needs to be a future in which the records we’re preserving will be used. Thankfully, there is a growing conversation around issues of climate change and sustainability, and the ways that digital preservation is impacted by and can impact them. The things that make us skilled digital preservation practitioners (thinking about systems, data flows and disaster recovery) are also key ingredients in supporting sustainability, so we have a lot to offer. At the same time, significant changes in this area are going to require us to both work collectively across the entire archival sector, and also to develop partnerships outside of it.

Learn more about the other winners from the 2020 NDSA Innovation Awards!

Call for Nominations to the NDSA Coordinating Committee

NDSA will be electing three members to its Coordinating Committee (CC) this year, with terms starting in January 2024. CC members serve a three year term, participate in a monthly call, and meet at the annual Digital Preservation Conference. The Coordinating Committee provides strategic leadership to the organization in coordination with group co-chairs. NDSA is a diverse community with a critical mission, and we seek candidates to join the CC that bring a variety of cultures and orientations, skills, perspectives and experiences, to bear on leadership initiatives. Working on the CC is an opportunity to contribute your leadership for the community as a whole, while collaborating with a wonderful group of dynamic and motivated professionals.

If you are interested in joining the NDSA Coordinating Committee (CC) or want to nominate another member, please complete the nomination form by 11:59pm EDT Friday, September 8, 2023, which asks for the name, e-mail address, brief bio/candidate statement (nominee-approved), and NDSA-affiliated institution of the nominee. We particularly encourage and welcome nominations of people from underrepresented groups and sectors.

As members of the NDSA, we join together to form a consortium of more than 270 partnering organizations, including businesses, government agencies, nonprofit organizations, professional associations and universities, all engaged in the long-term preservation of digital information. Committed to preserving access to our national digital heritage, we each offer our diverse skills, perspectives, experiences, cultures and orientations to achieve what we could not do alone.

The CC is dedicated to ensuring a strategic direction for NDSA, to the advancement of NDSA activities to achieve community goals, and to further communication among digital preservation professionals and NDSA member organizations. The CC is responsible for reviewing and approving NDSA membership applications and publications; updating eligibility standards for membership in the alliance, and other strategic documents; engaging with stakeholders in the community; and working to enroll new members committed to our core mission. More information about the duties and responsibilities of CC members can be found at the NDSA’s Leadership Page.

We hope you will give this opportunity serious consideration and we value your continued contributions and leadership in our community.

Any questions can be directed to ndsa.digipres@gmail.com.

Thank you,

Bethany Scott, Vice Chair

Catching up with past NDSA Excellence Awards Winners: Dr. Dinesh Katre!

The NDSA Individual Excellence Award honors individuals making significant contributions to the digital preservation community. In 2019, Dr. Dinesh Katre was one of two awardees in this category. Dr. Katre was recognized for his work to advocate for and deploy the Indian National Digital Preservation Programme which provides a robust and comprehensive platform for the effective long-term preservation of digital materials. As Chief Investigator of the Programme’s flagship project to establish a Center of Excellence for Digital Preservation,

Headshot of Dr. Dinesh Katre, Senior Director & Head of Department, Centre for Development of Advance Computing (C-DAC), Pune, INDIA.

Dr. Katre led the process to develop a digital preservation standard for India. He also conceptualized, designed and led the development of DIGITĀLAYA, a software framework, which comprehensively implements the OAIS reference model. Katre’s efforts culminated in the first repository in the world to achieve ISO 16363 certification.

We recently caught up with Dr. Katre to learn more about the progress of his work on the Indian National Digital Preservation program and other projects over the last few years.

1) What have you been doing since receiving an NDSA Excellence Award?

I have spearheaded the working group constituted by Supreme Court of India, which has defined the Digital Preservation Standard Operating Procedure (SOP) for the Indian Judiciary which provides coverage for Supreme Court of India, 25 High Courts and 672 District Courts. The SOP provides guidelines and recommendations with regard to Information Governance (IG) policies for courts, digitization of judicial records, cloud infrastructure for establishing Judicial Digital Repositories, tools and technologies, standards, AI/ML based applications to leverage upon massive data repositories to modernize the Indian Judiciary for accelerating the justice delivery. Most interestingly, we conducted 5 rounds of surveys across all high courts and district courts to collect information on various aspects of digitization. Huge amount of data was collected and analyzed for developing the insights. I am pleased to inform that the Digital Preservation Standard Operating Procedure (SOP) has been approved by honorable Chief Justice of India and sanctioned by the Law Ministry for implementation across the judiciary. The SOP is available at the following URL: https://ecommitteesci.gov.in/document/digital-preservation/

2) What did receiving the NDSA award mean to you?

I have worked extensively towards establishing the Indian National Digital Preservation Program since 2008, which involved development of archival systems, tools, standards, and digital repositories to comply as per the ISO 16363. As a part of my research, I had organized an Indo-US workshop to study the international trends in digital preservation in collaboration with the experts from National Digital Information Infrastructure and Preservation Program (NDIIPP) of the Library of Congress. I knew that National Digital Stewardship Alliance (NDSA) was originally launched by NDIIPP in 2010. Therefore, it was an immensely inspiring and encouraging moment for me to receive the NDSA Individual Innovation Award in 2019.

3) What efforts/advances of the last few years have you been impressed with or admired in the field of data stewardship and/or digital preservation?

I am particularly impressed with the advances in Digital Humanities and Computational Archival Science (CAS) which leverage upon artificial Intelligence/machine learning technologies to enable automation of digital preservation procedures and knowledge extraction from the digital repositories.

4) How has your work evolved since you won the Excellence Award?

The NDSA award infused me with a great deal of confidence and courage to embrace the evolving technological landscape. Since then, I have initiated R & D on the development of intelligent archiving tools for automatic metadata extraction, ontology-based classification of records, document orientation detection, visual entity tagging in miniature paintings and information extraction from documents.

5) What do you currently see as some of the biggest challenges or opportunities in digital preservation?

Whether proprietary or open source, the major challenge is heterogeneity and inconsistency in the properties of file formats. The digital preservation domain has been relying on open-source file formats but we must understand that they are primarily evolved for the purpose of interoperability. Therefore, a wider consensus is required for defining comprehensive “Universally Intelligible & Interoperable File Formats (UNIIFormats, a term coined by me) for all major types of contents, which would be specially designed for the purpose of digital preservation. The proposed UNIIFormats should provide built-in support for self-description, knowledge markup, semantic linkability, searchability, accessibility, discoverability, authenticity, and backward & forward compatibility. One should have a choice of storing information in the proposed UNIIFormat, if it requires long term retention. Incorporating so many properties into a file format may sound a bit utopian but I feel that there has not been much evolution and advancement in the file formats as compared with other technological advancements.

Presently, producing these properties for the data requires you to avail separate, fragmented, and paid application services. It may be beneficial for business but detrimental for preserving the digital footprint of the human civilization. Post-processing of the data for preservation is very laborious, costly, prone to loss of information, errors, and mis-interpretation.

We also require to use AI ML techniques for creating knowledge services to leverage the massive data repositories, which can help in long term sustenance.

You may like to refer my presentation on “Digital Eternity: Innovating a Future for the Past” which is available at the following URL: https://www.youtube.com/watch?v=lpr4ypE88qI&t=3047s

6) Are you working on any new digital preservation related projects at the moment?

I am presently leading the Digital Preservation for the Indian Judiciary initiative and Digital Preservation of Sanskrit Encyclopedic Dictionary project which is supported under the Science & Heritage Research Initiative (SHRI) of the Department of Science and Technology (DST).

National Libraries and the Levels of Digital Preservation

This post is a follow up to the summary post about the survey the NDSA Levels Steering committee released this spring, with the hope of being able to better understand the different scenarios under which institutions deploy the NDSA Levels.

While the response rate was very modest, with only twelve individuals/institutions filling out the survey by the deadline date. We noted that among those twelve were three national libraries. Overall, the three national libraries have in common an organizational/conceptual disposition toward the levels. In one case, the NDSA Levels are used as a “planning tool” to aid with development of digital preservation policies. In another, levels serve as a “maturity modeling tool” to help articulate functional preparedness for digital preservation tasks. And in another, the Levels serve as a “conceptual framework” that undergirds a three-year Digital Preservation Strategic Plan.

As a very small sample of non-representational institutions, it might seem like a stretch to zero in on how national libraries are using the Levels. But consider – national libraries are often the ones setting policy for everyone else. And so their common use of a particular rubric is helpful to note, alongside their uses of other tools to fill in some of the gaps they consider not covered by the NDSA Levels. Because that’s not all they’re using! Other tools mentioned included TRAC, OAIS, and PREMIS, as well as the Digital Preservation Coalition’s Rapid Assessment Model (DPC RAM) to track progress and for continuous improvement.

Environmental impact also emerges as a key area for more attention. Two of the three national libraries take environmental impact into account when considering their digital preservation activities. But the NDSA Levels don’t provide adequate guidance in this area, according to comments. In two cases, environmental impact is only considered when looking at bitstream preservation, and the process of evaluating environmental impact is still in its early stages. Given our growing awareness of the impact of multiple copies and other high-bandwidth activities on carbon output, and the increasingly immediate need for institutions to grapple with both preservation and impact, perhaps this is an area for reassessment at a high level, with additional input from policy-setting institutions like national libraries.

We plan to continue checking out the results of our survey and share any compelling findings. Do you have insights about the NDSA Levels and how you use them? Please reach out to the “Levellers” group! We would love to hear from you.

As a reminder, our next open session is on August 16th at 11:30 AM – 12:30 PM. The topic will be discussing the idea of environmental sustainability in regards to the Levels of Digital Preservation. We are keen to hear how the community would like us to engage with this topic and if there is any work required to update the Levels with this in mind.