Project Electron Update: Introducing Aurora 1.0

We are very pleased to announce the initial release of Aurora, an application to receive, virus check, and validate the structure and contents of digital records transfers. It provides a read-only interface for representatives of donor organizations to track transfers, so that they can follow their records as they move through the archival lifecycle. It also includes functionality for RAC staff to add or update organization accounts and users associated with them, appraise incoming transfers, and initiate the accessioning process. Aurora is built on community-driven standards and specifications, and we have released it as open source software. This is a major milestone for Project Electron, and we are excited to share it with the world. Many thanks to our partners at Marist College IT and to the Ford Foundation for their generous support of the project.

Aurora homescreen

We will continue to improve Aurora as we test and integrate it with a chain of other archival management and digital preservation tools.

Read more about Project Electron here.

What Aurora Does

We have designed Aurora primarily to enable ongoing transfers of records from active donor organizations in a way that regularizes and streamlines the process. Defining a consistent structure and set of required metadata elements for transfers of records facilitates automation of repetitive and routine administrative activities, allowing archivists to focus time and attention on the content and context of the records. Below is a summary of Aurora’s capabilities.

Transferring and Tracking Digital Records

At regular intervals, Aurora scans a directory structure where transfers are uploaded by donor organizations. Each transfer is first checked for viruses and then validated against the BagIt specification and the donor’s BagIt profile. Transfers that are not structurally valid, contain a virus, or are missing required metadata elements are rejected by the application and deleted immediately with notifications going to donors about the failure. The application displays transfer information in a searchable and sortable table where donors and archivists can track the status of transfers and view metadata. As transfers move through archival processes, their status will be updated in this interface, providing transparency for our donor organizations and increasing the visibility of archival labor.

Aurora dashboard interface

Aurora Dashboard Interface

Archival Appraisal

Once a transfer is validated, it moves to the appraisal queue, where appraisal archivists can review the transfers to ensure they are within a repository’s collecting scope. Archivists can accept or reject transfers, as well as add appraisal notes. When a transfer is rejected, it is removed from the appraisal queue, and its status is updated in the transfer table.

Aurora appraisal queue

Aurora Appraisal Queue

Archival Accessioning

Transfers accepted by an appraisal archivist move from the appraisal queue to the accessioning queue. Here, transfers are grouped by organization, record creators, and record type. Archivists can initiate the accessioning process by creating accession records for one or more transfers based on aggregated data from transfers grouped in an accession. Future work will integrate Aurora’s accessioning functionality with an archival management system (in our context ArchivesSpace) and a digital preservation packaging system (Archivematica).

Aurora accession record

Aurora Accession Record

User and Organization Administration

The application supports the management of organizations, user accounts, user groups, and their associated permissions. Archivists with permissions can declare user accounts active or inactive, associate them with a specific organization, or assign them to a group. Depending on the group that is assigned, users see different views of the application that are specific to their functions (such as appraisal archivist or accessioning archivist). The application also allows certain groups of archivists to create and delete organizations, as well as edit the BagIt Profiles and PREMIS rights statements for organizations.

Bag-It Profiles

Aurora provides a user interface for archivists to create, edit, and delete BagIt Profiles for organizations. These profiles allow us to specify metadata elements that must be present in each transfer, the repeatability of those elements, as well as control values for certain elements. This ensures that all transfers will include DACS single-level minimum required description, allowing us to automate processes on top of that metadata. The application also provides a JSON representation of the Profile.

PREMIS Rights Statements

Aurora provides a user interface for archivists to create, edit, and delete PREMIS Rights Statements, and associate them with specific record types. This automates the creation of rights statements for each transfer of records, since source organization and record type are among the minimum required metadata elements for a records transfer. Having rights statements linked to records defines what can and cannot be done with those records based on factors like copyright and our agreements with donors. The application automatically calculates the term of grant or restriction based on the ‘start’ or ‘end’ dates of the records in the transfer (also required metadata elements).

API

Aurora comes with a RESTful API (built using the Django Rest Framework). The application includes a browsable API interface, as well as command-line interaction. This is a key feature that allows Aurora to be integrated with other tools.

What’s Next for Aurora and Project Electron

We continue to push forward on Project Electron through the development of a set of microservices to enable the integration of Aurora with other systems such as ArchivesSpace and Archivematica, usability testing of Aurora, and solidifying our data model. For a detailed look at the project plan, check out the project milestones.

Interested in using our code? Check out Aurora on GitHub or download the 1.0-alpha source code here.

4 thoughts on “Project Electron Update: Introducing Aurora 1.0

  1. Are your donors responsible for bagging materials before upload? Have you made instructions (that you can publish/share) for that?

    • Yes, donors will create the bags that conform to the BagIt profile for their organization and push them to Aurora. We’re initially targeting this process toward large organizations that will need to automate the bagging process to work at scale, and because these donors will be pulling records and metadata from a variety of systems on an ongoing basis, we are letting them determine how they want to create the bags. However, we’ve shared an example bag profile and an example python script to create and validate bags which are available on the Project Electron repo here: https://github.com/RockefellerArchiveCenter/project_electron/tree/master/transfer

      As we roll this out to a wider range of organizations, we’ll be evaluating existing tools for creating bags, and will certainly share our findings!

      • Thanks!

        Also, any reason for the 2GB bag size limit recommendation? I’m looking at this for bags that will regularly be a few hundred GB.

        • Hi!

          Great question. For our use case, we expect transfers to be fairly atomic and regularized, and are anticipating receiving primarily text-based records. Keeping the size limit low will give us better metadata for groups of records to have a sense of what they are. The size limit can certainly be raised, but we haven’t done much stress testing for how long a much larger transfer would take over SFTP.

          We’re focused on a minimum viable product with this, but are certainly interested to see how Aurora might be adapted and adopted in other contexts.

Leave a Reply

Your email address will not be published. Required fields are marked *