Recent Posts

2020 has been a tough year. I think many of us felt, and still feel, overwhelmed, depressed and hopeless. In that context, when I took my annual look back at the D-Team’s progress over the past year, I was expecting to see reduced activity and productivity. Instead, what I found was the opposite - this year we completed a number of large-scale initiatives, any one of which would have, in years past, occupied us for much of the year:

The Rockefeller Archive Center just wrapped up a little less than five month long project to separate our archival and bibliographic data, both in our collections management system and our discovery interface. You might have read a little bit about this in Katie Martin’s blog post describing the process of removing our library data from ArchivesSpace, but that only covers a portion of the entire project. The processing group working on the export scripts in mid-May, finished in mid-June, and we gathered requirements, built, tested, and launched the system from mid-June to the first week of October. Now that our new library discovery system has gone live, this post will provide more context to the project, outline the process, discuss the technical profile of the system, and finally share some lessons we learned along the way.

Following on the initial implementation of our data pipeline back in March, we’ve been looking closely at ways to increase the performance of the pipeline, by which we mean how many objects the pipeline is able to process in a given time interval. This has been a fairly lengthy process with a lot of learning along the way, so I thought it would be good to document some of what we’ve learned here, both for our future reference as well as for the benefit of someone else trying to do this kind of thing in the future.

Over the last few months, Patrick Galligan of the digital strategies team along with Amy Berish, Darren Young, and myself of the processing team have worked on a collaborative project to export JSON representations of bibliographic data and delete all corresponding resource, subject, agent, and top container records in ArchivesSpace. This was just the first step of a larger project to create a new interim discovery system for library data separate from our archival discovery system, DIMES.

As of this past July 1st, the Digital Programs team (we’re often colloquially referred to as the D-Team) is now officially known as Digital Strategies. This seemingly insignificant change of one word in our name reflects tectonic shifts, both to our team’s role as well as a larger organizational approach to technology and technical expertise.