Using APTrust

From aptrust
Jump to: navigation, search

Using APTrust

APTrust provides a complete demo environment where depositors can test depositing and restoring of objects. It is encouraged to use the demo environment first to get familiar with the system and process.

Before you do anything, you'll need access to your institution's S3 buckets and to APTrust's Web UI, PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata..

To use APTrust basic features, you'll receive the following for both, demo and the production environment.

  • AWS keys. These let you access your receiving bucket, into which you'll upload new bags for ingest, and your restoration bucket, from which you'll download restored bags. Each member has two sets of keys: one for the demo environment, and one for the production environment. You can request AWS keys from [[1]]. 
  • Tools for accessing S3 buckets. Amazon maintains client libraries in commonly-used languages to help you access S3 through the language of your choice. APTrust maintains a set of partner tools for uploading to, downloading from, and listing the contents of your S3 buckets.
  • A login for our Web UI. If your organization already has access to our web interfaces at https://demo.aptrust.org/ and https://repo.aptrust.org, then you have a local APTrust administrator who can set up an account for you. If no one in your organization has credentials to these sites, contact [[2]] to get set up.
  • An API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program. You'll need this only if you plan on accessing our member APIApplication Programming Interface.  Once you have a login for the Web UI, you may generate an API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program using by accessing your users profile page. Click "Generate an API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program" button and and note the generated key in a secure place. For help contact [[3]] for an API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program, and note that you'll need separate keys for the demo and production systems.

New Institution Ramp-up

Welcome to APTrust! By now you have signed the terms and conditions of deposit and are ready to start. To make your ramp-up to using APTrust as quick and easy as possible we have created a checklist to get you started.

# Step You should APTrust staff will
1 Review membership responsibilities Review the responsibilities of APTrust membership in all relevant documentation.
2 Provide user information Provide names, institutional email addresses, and their roles (admin or depositor user) the individuals will fill.

An "admin" user will have the ability to manage other users and allow deletion requests of other users. We employ a two-factor deletion policy that requires two users (admin and regular user) to delete data.

A "depositor user" is able to make deposits and access the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. web-ui and API

Use that information to setup accounts on the repository frontend and AWS backend systems.
3 Provide credentials to the institutional users Make sure to store credentials in a safe manner. Provide:

- Each institutional user (admin or depositor) with an account on the repository frontend systems called "PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata." (https://demo.aptrust.org [demo system] and https://repo.aptrust.org [production system]). Note: only admin users can delete files or objects.

- Each institutional user with individual AWS credentials to be able to deposit and restore objects to S3 buckets. Each user will receive an email with a link to an encrypted note that entails the credential information, AWS S3 restore and deposit buckets. The note will destroy itself after opening for the first time.

4 Read the documenation Read the documentation on how to use APTrust.
5 Login to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Now that you have read the documentation it's a good point to take a look at the repository frontend "PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata.". You may have realized that you haven't gotten credentials to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata.. That is because you will create them yourself. Go to each system https://demo.aptrust.org and https://repo.aptrust.org and enter the email address you have provided to use and click the "Forgot Password?" link below.
6 Create an API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program Once you are able to login to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata., you can create an API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program in order to use the repositories APIApplication Programming Interface. (You may not need it if you don't want to programmatically access the repository.) Login to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata., and click on "View Profile" in the top right corner on your name.
View Profile
Generate API Key
- Click on "Generate API Secret Key". The generated API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program will appear in the top. Make sure to store it safely as you won't be able to read it again. If you lose the key, you will have to re-generate a new key.
Generated API Key
7 Make a test deposit Use the "Easy Store" or "Partner Tools" bagging tools to create a test bag to be deposited.
8 Perform a test restore Use the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. web interface or an API call with the appropriate object identifier.

Technical Support

For technical support email to help@aptrust.org or connect to us directly through the APTrust Slack #aptrust_community.

Technical Workflows

Required Metadata elements

bagit.txt

This is requited by the BagIt specification, and should contain the following:

BagIt-Version:  0.97
Tag-File-Character-Encoding:  UTF-8

bag-info.txt

Valid APTrust bags MUST contain a bag-info.txt file with the following fields, which may be blank:

Fieldname Example Description
Source-Organization: University of Virginia This should be the human readable name of the APTrust partner organization. For example, "University of Virginia." You may be more specific, if you wish, specifying a specific college or library within the university, such as "Georgetown University Law Library." However, when APTrust restores bags, the source organization in the bag-info.txt file will be set to the name of the partner institution.
Bagging-DateISO 8601 UTC format Date (YYYY-MM-DD) that the bag content was prepared for delivery.: 2017-07-06 Date of bagging using ISO 8601 UTC format (YYYY-MM-DD).
Bag-Count: 1 Two numbers separated by "of", in particular, "N of T", where T is the total number of bags in a group of bags and N is the ordinal number within the group; if T is not known, specify it as "?" (question mark).  Examples: 1 of 2, 4 of 4, 3 of ?, 89 of 145.
Internal-Sender-Description: [Optional] Human readable description of the contents of the bag. This will appear as the bag description in our web UI. An alternate sender-specific identifier

 for the content and/or bag.

Internal-Sender-Identifier: [Optional] Internal or alternate identifier used at the senders location. This will appear in the web UI, and you can search for bags using this identifier.  A sender-local prose description of the contents of the bag.
Bag-Group-Identifier: [Optional] Several bags may share the same Bag-Group-Identifier to indicate that they are part of the same collection, or part of the same logical group. The Bag-Group-Identifier should be a UTF-8 string or a number in string format. Starting in the second half of 2018, depositors will be able to search by Bag-Group-Identifier for Intellectual Objects in the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. UI and in the REST APIApplication Programming Interface. This is not yet implemented as of July 2018. This tag is entirely optional. If the tag is missing from the bag-info.txt file, the bag is still valid.
An example file could look like this:
Source-Organization: University of Virginia
Bagging-Date: 2018-02-27
Bag-Count: 1
Internal-Sender-Description: Twitter captures of the events of August 12 2017 in Charlottesville, VA
Internal-Sender-Identifier: AUG12
Bag-Group-Identifier: Cville-Aug-12

This file MAY contain additional fields; however, APTrust will not preserve the additional tags in bag-info.txt. Because bag-info.txt may contain a Payload-OxumThe "octetstream sum" of the payload, namely, a two-part number of the form "OctetCount.StreamCount", where OctetCount is the total number of octets (8-bit bytes) across all payload file content and StreamCount is the total number of payload files. Intended for machine consumption. tag, APTrust regenerates this file when you restore a bag. Consider these two situations:

  1. You upload a bag containing 100 files to APTrust. Then you delete 10 of those files.
  2. You upload a bag containing 100 files to APTrust. Then you upload a new version of that same bag, overwriting 10 files.

When we restore either one of these bags, the Payload-OxumThe "octetstream sum" of the payload, namely, a two-part number of the form "OctetCount.StreamCount", where OctetCount is the total number of octets (8-bit bytes) across all payload file content and StreamCount is the total number of payload files. Intended for machine consumption. value, which shows the number of bytes and number of files in the payload directory, will not match the Payload-OxumThe "octetstream sum" of the payload, namely, a two-part number of the form "OctetCount.StreamCount", where OctetCount is the total number of octets (8-bit bytes) across all payload file content and StreamCount is the total number of payload files. Intended for machine consumption. of the original bag-info.txt file, and your BagIt validator will show the bag as invalid. For this reason, we regenerate bag-info.txt when restoring the bag.

We do preserve and restore all tag files other than bag-info.txt and bagit.txt, so if your bag includes tags that you want to preserve, put those tags into other tag files.

aptrust-info.txt

This file MUST be present and MUST contain the following tag fields.

  • Title: Human readable title for searching and listing in APTrust. This cannot be empty.
  • Description: A human-readable description of the bag.
  • Access: One of three enumerated access conditions. [“Consortia”, “Restricted”, “Institution”]. These access restrictions describe who can see the object metadata, including the object's name and description, a list of its generic files and events. APTrust does not currently provide access to the objects themselves, except when the owning institution restores a bag it owns. In other words, no matter which access setting you choose, no other institution can access your intellectual object. The general public cannot see any information in the APTrust system.
    • Restricted: Metadata about this object is accessible to the institutional administrator (at the depositing institution) and to the APTrust admin. No one else can even see that this object exists in the repository.
      • Institution: All users at the depositing institution can see metadata about this object.
      • Consortia: All APTrust members can see this object's metadata.
  • Storage-Option: This tag is new as of July 2018. It indicates how and where you want APTrust to store your bag. Because this tag was not part of the original APTrust BagIt specification, this tag is not required. If omitted, Storage-Option defaults to "Standard," which was the only storage option that existed prior to 2018. See the section on Storage Options below for more information. The Storage-Option tag supports the following values:
    • Standard: The bag's contents will be store in S3 in Northern Virginia and Glacier in Oregon. APTrust will perform fixity checks on the S3 files every 90 days.
    • Glacier-OH: Files will be stored ONLY in Glacier, in AWS's Ohio region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.
    • Glacier-OR: Files will be stored ONLY in Glacier, in AWS's Oregon region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.
    • Glacier-VA: Files will be stored ONLY in Glacier, in AWS's Northern Virginia region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.


Bagging specifications

Following a quick checklist for valid bags. For detailed specification go here: Bagging specifications


Valid bags meet all of the following criteria:

  • The bag was submitted as a tar file, without compression
  • Bag name follows the pattern <institution.edu>.bag_name[.b###.of###].tar.
  • Bag untars to a directory whose name matches the name of the tar file, minus the .tar extension.
  • Bag contains an md5 or sha256 manifest (or both)
  • Bag contains the data directory
  • Bag contains bagit.txt, as described above
  • Bag contains bag-info.txt as described above
  • Bag contains aptrust-info.txt as described above
  • All data files are in the manifest, and all checksums matched
  • All tag files mentioned in the tag manifest are present, and checksums match (you may omit tag files from the tag manifests)


Deletion

! This feature is available only to institutional administrators. !

In order to delete an object an institutional administrator has to access the APTrust web interface ( https://demo.aptrust.org or https://repo.aptrust.org ). There is no API endpoint to delete files programmatically. APTrust also uses a double fault deletion system which means all deletion requests will need to be confirmed via email.

Intellectual objects

To delete intellectual objects use the red DELETE button (see below) on the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Web UI.

PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Web UI

After you click Delete, the web interface will send an email to other institutional administrators at your institution, asking for someone to confirm the deletion request via a link provided in the email. Once the link has been clicked, APTrust will create a delete request for each generic file in this intellectual object. You'll see the delete requests listed under the Processed Items tab of the Web UI.

Important Note: If at all possible, the system will send the request to a user other than the one who made the request, ensuring all deletion requests are seen by at least two people. However, if your institution has only one institutional administrator, they will be able to both request and confirm the deletion of an intellectual object or generic file. It is therefore highly encouraged that all institutions have at least two institutional administrators.

Individual Files

You can also delete individual files:

  • Locate the Intellectual Object whose file(s) you want to delete.
  • Click the View Preserved Files button.
  • Click on the name of the generic file you want to delete.
  • Click the delete button at the bottom of the page.
  • Confirm the deletion of the generic file by email (typically done by a different user)
PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Web UI

Once the deletion request has been confirmed, you will see a single delete request under the Processed Items tab.

Restrictions

  • You can delete only objects and files that belong to your institution.
  • You must be an institutional admin to delete a file or object.
  • If a pending request exists to restore an intellectual object, or to send that object to DPN, the system will not allow you to delete the intellectual object or any of its files until those pending operations are complete.
  • The deletion request must be reviewed and confirmed by another institutional administrator from that same institution.

Effect of Deletion on Metadata

When you delete a file, PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. creates a deletion PREMIS event for the file that includes the date and time of deletion and the email address of the user who requested the deletion. PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. keeps the generic file records, changing it's state from 'Active' to 'Deleted.' PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. also keeps all prior PREMIS events and fixity records related to the file, and the file record remains accessible through both the Web UI and the member API. In addition, PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. keeps a Work Item record that shows when the deletion was requested and when it was completed. Work Items are also available through the Web UI and the member API.

Special Circumstances

Deletion requests for both files and objects can be made by members of the APTrust technical staff. However, those deletion requests are subject to the same requirements; specifically, that deletion request must still be reviewed by an institutional administrator at the institution to which the file or object belongs. APTrust staff do not receive deletion confirmation emails.

Ingest

To store an item in APTrust, simply bag it in valid APTrust format and upload it to your institution's APTrust receiving bucket. Your receiving bucket for the demo system is:

aptrust.receiving.test.<institution.domain>

The demo system is for testing your workflows and bagging tools. It currently accepts bags no larger than 100MB. It will ignore files larger than that in the receiving buckets.

The receiving bucket for the production system is:

aptrust.receiving.<institution.domain>

Replace <institution.domain> with your organization's domain name. For example, virginia.edu, jhu.edu, etc.

You'll need AWS keys to complete the upload. If you don't yet have them, contact support@aptrust.org. You can upload bags with [Amazon's S3 CLI tools], or by integrating one of [Amazon's S3 client libraries] into your own tools, or by using APTrust's partner tools.

Note that APTrust does take responsibility for your bag until it is fully ingested. Fully ingested means we have stored all of the bag's contents in our long-term storage areas and have recorded an Intellectual Object record for the bag, Generic File records for each file in the bag's payload, and PREMIS events describing the ingest of the object and the ingest, identifier assignment, and initial fixity values for each of the object's files.


Monitoring the Ingest Process

APTrust periodically scans the receiving buckets and adds new bags to the ingest queue. You can follow the status of all work requests by clicking the Processed Items tab in our Web UI.

PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Work Items


Click on an item to view status details.

PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Work Item Detail

You can also programmatically retrieve the status of work items using the Items endpoint of the member APIApplication Programming Interface.

The ingest process follows these steps:

  • Fetch - We retrieve the bag (the tar file) from your receiving bucket.
  • Unpack - We untar the bag.
  • Validate - We make sure all files are present and match the checksums in the manifests. (The BagIt spec allows you include some custom tag files without mentioning them in the tag manifests. If we find these files, we allow them, but we obviously can't validate their checksums.)
  • Store - We copy the files to long-term storage in Virginia.
  • Record - We tell PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. what we ingested, recording all generic files and PREMIS events.
  • Replicate - We copy all files to Glacier in Oregon and tell PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. where we put them.
  • Cleanup - If all succeeded, we delete the original tar file from your receiving bucket.

Once a bag enters the work queue, ingest can take anywhere from one minute to several hours. The two factors that affect ingest time are bag size and system load. When APTrust is flooded with ingest requests, several hours may elapse between your bag appearing in the work queue and Step 1 of the ingest process. Bags containing large amounts of data always take a long time to process, because we have to retrieve the bag from the receiving bucket and calculate md5 and sha256 checksums on all of the bag's contents.

Uploading Multiple Versions of the Same Bag

See Updating, below.

Ingest Timeline

The ingest process can take anywhere from one minute to 24 hours to complete. During periods when the system is under sustained heavy load, ingest may take 2-3 days.

The two factors that determine how long an ingest will take are:

  1. The size of the bag.
  2. The overall load on the ingest servers.

When you upload a bag to your receiving bucket for ingest, our ingest services may take up to one hour to notice the new bag. At that point, ingest services add the new bag to a work queue for processing, which includes the following steps:

  1. Copy the bag from the S3 receiving bucket to a processing area (a hard drive attached to the ingest server). This can anywhere from a few seconds for a small bag (under 10 MB) to several hours for a large bag (over 1 TB).
  2. Validation can take a few seconds for a smaller bag (under 10 MB) or several hours for a larger bag (more than 200 GB). The most time-consuming part of bag validation is fixity calculation. We calculate both md5 and sha256 checksums on every file. When APTrust validates the bag, it ensures the following:
    1. all files are present,
    2. all checksums match the payload and tag manifests (either an md5 or sha256 payload manifest is required; tag manifests are optional),
    3. the bag contains no extraneous or illegal files,
    4. no payload files are missing from the payload manifests.
  3. Store the bag. This involves copying the bag's files to S3 in Northern Virginia and to Glacier in Oregon. This step can take a minute or less for smaller bags (less than 10 MB) and over 24 hours for large bags containing many files (more than 1TB and more than 5000 files). A 1TB bag containing 30,000 files usually takes longer to store than a 1 TB bag containing 2 files because of limitations in the number of concurrent upload connections supported by our current S3 network library.
  4. Record the bag. This step involves recording metadata in PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. about the bag and its contents. We store an Intellectual Object record for the bag, a Generic File record for each file in the bag, and a number of PREMIS events describing ingest, fixity generation, identifier assignments, etc. Since each file generates six PREMIS events, a bag of 100 files will generate about 700 records (100 files, plus 600 events), while a bag of 10,000 files will generate around 70,000 records. The record phase can take anywhere from 1 second to 20 minutes, depending on the number of files in the bag.

In addition to the four steps listed above, system load may have an impact on ingest time. If you upload a 5MB bag that would normally ingest in under a minute, and there are five 1TB bags in the work queue ahead of your bag, you may have to wait 48 hours or more for some of those large bags to process before the system even attempts to ingest your 5MB bag. This kind of backlog does not happen often but it does happen reliably. APTrust often gets large ingests from multiple institutions in the two weeks preceding the annual Spring meeting, and again in the the two weeks preceding the Fall meeting. In addition, APTrust depositors who are also DPN members may pass several terabytes through APTrust to DPN in the last two weeks of December to meet DPN's deposit deadlines.

Ingest and Stewardship / Responsibility for Materials

The depositor is responsible for their materials until a bag has been fully ingested into APTrust. Fully ingested means APTrust has validated the bag, stored its contents, and created metadata records in PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata., including a record for the Intellectual Object, and record for each of a bag's Generic Files, and PREMIS events recording the ingest of the object and each file.

There are several ways to know that a bag has been fully ingested, including:

  1. The ingest Work Item record for the bag. When this appears in the Web UI or the API results with Stage = Resolve and Status = Success, the bag has been ingested. Note that for bags ingested more than once, you should check the latest Ingest Work Item for the bag.
  2. A new PREMIS ingestion event has been recorded for the bag. Again, for bags ingested more than once, check for an ingestion event recorded AFTER you uploaded the bag to the receiving bucket.
  3. You can run the apt_check_ingest tool in the APTrust Partner Tools on your own computer to check our APIApplication Programming Interface. The tool will list ALL ingest attempts for a specified bag, and will tell you whether and when each ingest succeeded or failed. apt_check_ingest can print results in plain text for human consumption, or in JSON format for scripted machine use.

APTrust is responsible for the contents of the bag after the bag has been fully ingested.

If your workflows include deleting your local copy of the bag after upload to APTrust, they should not delete the bag until after APTrust shows it has been successfully ingested. If you delete your local copy of the bag, you should still keep a copy of the files that were packaged in the bag. APTrust was designed to be a component of your preservation strategy, not the sole steward or conservator of any content.

Ingest and Storage Options

As of Summer, 2018, the APTrust BagIt format supports a new tag in the aptrust-info.txt file called Storage-Option. The Storage-Option tag tells the APTrust ingest process where to store the bag's payload for long-term preservation. Because this tag was not part of the original APTrust BagIt specification, this tag is not required. If omitted, Storage-Option defaults to "Standard," which was the only storage option that existed prior to 2018. See the section on Storage Options below for more information. The Storage-Option tag supports the following values:

  • Standard: The bag's contents will be store in S3 in Northern Virginia and Glacier in Oregon. APTrust will perform fixity checks on the S3 files every 90 days.
  • Glacier-OH: Files will be stored ONLY in Glacier, in AWS's Ohio region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.
  • Glacier-OR: Files will be stored ONLY in Glacier, in AWS's Oregon region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.
  • Glacier-VA: Files will be stored ONLY in Glacier, in AWS's Northern Virginia region, and will be encrypted during storage. APTrust will not perform any fixity checks on these files.

Note that Storage-Option is a "sticky" setting and will apply to all subsequent versions of a bag you upload. For example, if you deposit a bag with Storage-Option Glacier-OH, all subsequent versions of that bag will be stored with the Glacier-OH storage option, even if the Storage-Option tag in aptrust-info.txt specifies otherwise. We do this to prevent versioning problems.

Consider a case where you upload one version of a bag in January and a newer version in June. The APTrust ingest process compares the files in the new version of the bag to those already in preservation. It overwrites preserved files that have changed in the new bag, and it copies into preservation files from the new bag that did not exist in the original. (The process is described here.) The bag update process considers the last-ingested checksums for each file to be authoritative.

If APTrust were to store the January version in Glacier-OH and the June version in Glacier-VA, it would effectively have two versions of the same intellectual object. The checksums for files updated in June would not match the older version of those files still stored in Glacier-OH from the original ingest. When you ask to have the bag restored, the system will say the bag is valid if it happens to restore the bag from Glacier-VA (the second ingest) because all the file checksums match. If the bag were restored from Glacier-OH (the first ingest), the system would say the bag is invalid because the checksums of updated files don't match and any files added in the June update would be missing.

In short, the stickiness of the Storage-Option tag is due to the fact that APTrust's metadata registry has no internal notion of versioning. A bag always has only one authoritative version.

If you upload a new version of an existing bag, and the Storage-Option tag in the new version does not match the Storage-Option tag of the original version, APTrust's ingest process will forcibly apply the original Storage-Option and store the new version of the bag in the same region as the original.

If you are sure you want to change a bag's Storage-Option, you must first delete the existing bag/intellectual object from APTrust and then upload the new version with the new storage option.

The Standard storage option makes sense for objects and files for which:

  • You want high redundancy. APTrust stores six copies of each file across multiple AWS regions availability zones.
  • You want geographic diversity. These files are stored in S3 in Virginia and Glacier in Oregon.
  • You want regular fixity checks. APTrust runs fixity checks on these files every 90 days.
  • You expect to restore the object. Items in Standard storage can typically be restored within a few hours at no cost to the depositor.
  • You want to push items through APTrust to DPN. You can do this with Standard storage but not with Glacier-only storage (due to cost issues).

Glacier-only storage makes sense for items you want to preserve at minimal cost and which you do not expect to retrieve, except in cases of emergency or disaster. The characteristics of the Glacier-only options include:

  • Reduced cost. Glacier-only storage is significantly less expensive than S3 storage.
  • No fixity checking. Fees for retrieving Glacier files every 90 days would be prohibitive.
  • Longer retrieval / restoration time. Restoring bags from Glacier involves the extra step of temporarily moving files from Glacier to S3 before reconstructing a bag. This step usually takes several hours.
  • Encryption at rest. All Glacier files are encrypted at rest and decrypted upon retrieval. AWS manages the encryption keys as well as the encryption/decryption of data.
  • Potential additional fees for excessive data retrieval. AWS charges steep fees for retrieving more than 5% of Glacier storage volume in a single month. AWS pro-rates the 5% per-month cap down to the hour. Assuming there are 720 hours in a month, fees would begin to accrue when a depositor retrieves more than 1/720th of its 5% allotment in a single hour. APTrust may pass these fees on to depositors.
  • No "send to DPN" option. Because of the cost of retrieving items from Glacier and rebagging files in DPN BagIt format, you cannot push Glacier-only bags from APTrust to DPN.

DPN Ingest

We currently support DPN ingest for APTrust member institutions who are also members of DPN.

More complete documentation will be coming soon.

Restoration

Restoration of content can be triggered by using the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. web interface or an API call with the appropriate object identifier. This will initiate a process that repackages the objects back into a bag using the current APTrust BagIt profile. This will allow depositors to maintain a single set of packaging/restoration scripts based on the current bag format.  As with the submission bags, content in the data directory will always remain as it was sent to APTrust in the first place.  Tag files and top level files that are part of the wrapping bag will always reflect the current APTrust BagIt profile so depositors do not have to keep track of what bag version was used in the past for a specific item. This achieves the following goals:

  • Allow institutional admins to request that a specific Intellectual Object and all active child Generic File object files are repackaged for download to their local institution.
  • To provide a simple downloadable distribution package that conforms to the current APTrust BagIt profile so depositors only have to maintain a means for translating the current version of the APTrust BagIt spec.
  • To return the exact same bits for files deposited in APTrust for preservation.
  • To return files in the data directory with the same name and relative path used in the original submission bag.

When content restoration is requested, a distribution package will be created consisting of the original files and metadata written to a BagIt bag conforming to the current APTrust BagIt profile.  The files in the data directory will be the exact same name and bits that were sent to APTrust in the submission bag and the metadata written to tag files in the bag adhering to the current APTrust BagIt format.  This allows partners to only have to be able to parse a bag in the current APTrust format and give us the flexibility to migrate our content models and metadata more freely in the future.

Note: The files in the restored bag will not have the same owner id, group id, and permissions as the files in the original submission.

Restoration using PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Web-UI

To restore a bag locate the intellectual object to retrieve and click the Restore Object button.
PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Work Items


This adds a restoration task to the work queue, which you can monitor by clicking the Processed Items tab, or by calling the Items endpoint of the Member API.

PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Work Item Detail


The restoration process can take anywhere from a few minutes to a few hours, depending on the amount of traffic our system is handling and the size of the bag. Larger bags always take longer to restore, because we calculate md5 and sha256 checksums on the bag's contents.

When we restore a bag, we retrieve all of the intellectual object's files from long-term storage, verify the checksums, reassemble the bag, write the manifests, tar up the bag, and leave it in your receiving bucket. When that's done, the Processed Items list will show your bag with a green background, with Action = Restore, Stage = Resolve, and Status = Success, like the first item in the list above.

It's up to you then to retrieve and delete the item from your restoration bucket. Your restoration bucket for the demo system is:

aptrust.restore.test.<institution.domain>

For the production system, it's:

aptrust.restore.<institution.domain>

Replace <institution.domain> with your organization's domain name. For example, virginia.edu, jhu.edu, etc. You can download your bags using Amazon's S3 CLI tools, or by integrating one of Amazon's S3 client libraries into your own tools, or by using APTrust's partner tools.

Format and Content of Restored Bags

According to the "Other Files" section of our old documentation, APTrust did not preserve or restore custom tag files. That changed as of March 29, 2016.

When you restore a bag that was ingested after March 29, 2016, the version you get back will have the same contents and format of the bag you originally uploaded, except: Individual files you deleted through our Web UI will not be restored. The restored bag will include the following manifests, even if they were not present in the original bag:

  • manifest-md5.txt
  • manifest-sha256.txt
  • tagmanifest-md5.txt
  • tagmanifest-sha256.txt

All tag files will be listed in the tag manifests, even those that were omitted from tag manifests in the original upload.

If you uploaded multiple versions of the same bag with the same name, you'll see the following:

  • The restored bag contains the latest (last uploaded) version of each file.
  • Files that were not included in the later versions of a bag you uploaded multiple times, but were present in earlier versions, will be present in the restored version unless you manually deleted them through our Web UI. (Our policy not to delete your content. You have to do that deliberately.)
  • We regenerated the bag-info.txt file to prevent possible conflicts in the Payload-OxumThe "octetstream sum" of the payload, namely, a two-part number of the form "OctetCount.StreamCount", where OctetCount

is the total number of octets (8-bit bytes) across all payload file content and StreamCount is the total number of payload files. Intended for machine consumption. value. Because files in the bag may have been updated or deleted between initial ingest and restoration, the Payload-OxumThe "octetstream sum" of the payload, namely, a two-part number of the form "OctetCount.StreamCount", where OctetCount is the total number of octets (8-bit bytes) across all payload file content and StreamCount is the total number of payload files. Intended for machine consumption. value of the original bag-info.txt file may no longer be valid, so we regenerate the entire bag-info.txt file, including only the minimum required tags. This means you may lose some valuable tag data. If you want to preserve tags, put them in a tag file other than bag-info.txt and bagit.txt.

Bags ingested prior to March 29, 2016, will be restored with the same contents as above, except:

  • No custom tag files will be restored, because we didn't preserve them.

Restoration Timeline

The time it takes to restore a bag depends on these three items:

  1. The size of the bag. Bigger bags take longer.
  2. The overall load on the APTrust servers.
  3. The number of files in the bag being restored.

A small bag (under 10 MB) with a handful of files may be restored in less than a minute. A large bag (1 TB), may take 12 hours, and even longer if it contains tens of thousands of files.

When you click the restore button to restore a bag/intellectual object, the request goes into a work queue. The system usually acts on the requests within 20 minutes. So if you're restoring a 50MB bag, the process may take 30 minutes: twenty minutes before the system begins to restore the bag, plus ten minutes to assemble it and put it into your restoration bucket.

If the restoration work queue contains many other pending requests, or if APTrust's servers are under heavy load, it may take longer to start working on your restoration request. The system's busiest times tend to be during the two weeks preceding the annual APTrust spring meeting, the two weeks preceding the fall meeting, and the last two weeks of December. During those times of heavy load, a restoration that takes 30 minutes at any other time of year may take 8 hours or more.

Updating

APTrust does not version bags. If you want to keep multiple versions of a bag, use a naming convention. For example: virginia.edu.bag_of_photos, virginia.edu.bag_of_photos_V2, virginia.edu.bag_of_photos_V3

When you upload a bag that has the same name as an existing bag, this is what happens:

  1. If a file in the new bag has the same name as a file in the old bag and the size or the md5 checksum or the sha256 checksum has changed, we overwrite the old file with the new one. You cannot recover the old file.
  2. If a file in the new bag has the same name as a file in the old bag and the size and checksums have not changed, we do nothing.
  3. If a file in the new bag did not exist in the old bag, we save it. If a file in the old bag is not present in the new bag, we do not delete it.

This table shows what happens when you upload a new version of a previously ingested bag.

Old Bag New Bag What is stored Reason
bag-info.txt bag-info.txt (changed) new version Contents in new version have changed
data/document.pdf data/document.pdf (unchanged) old version The document did not change
(file not present) data/new_image.jpg new version File did not exist in old bag, but it's here now
data/old_image.jpg (file not present) old version Although this file has been deleted from the new bag, we will not assume you want to delete it from storage. File deletion must be a deliberate act of the depositor.

This update policy has three important implications:

  1. If you want to delete files from an ingested bag / intellectual object, you must do that deliberately. Currently, you can delete only through our Web UI.
  2. When you restore the bag described in the table above, you'll get back both old_image.jpg and new_image.jpg (unless you manually delete one of them before you restore).
  3. You can update metadata in a bag by uploading only the metadata, as long as there's at least one file in the data directory and the bag is otherwise valid. This may be useful for bags that contain 100GB of data and 100KB of frequently-updated metadata.

In the registry (PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata.), updating (overwriting) an existing file in a bag causes the following to happen:

  1. A new PREMIS ingestion event appears with the date of the new ingest.
  2. Two new PREMIS fixity generation events appear, one with the md5 checksum of the new file, and one with the sha256 checksum.

The PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. file page for the updated file will show the new checksums for the file at the bottom of the page, along with the date on which the new checksums were calculated. Below those will be the older checksums, and the dates they were calculated.

Future fixity checks on the updated files will test against the latest fixity value of the updated files.

Ongoing Fixity Checks

APTrust automatically performs fixity checks on each preserved file every ninety days. We perform the check agains the sha256 checksum that was calculated upon ingest. If a fixity check succeeds, we record a PREMIS event with the date of the check, with the outcome marked as "succeeded", and the actual fixity value from the check preserved in the outcome detail. If the check fails, we record a PREMIS event with the same information, but the outcome is marked as "failed."

Our daily alerts notify administrators at depositing institutions of any failed fixity events related to their files. Our reports allow depositors to find failed fixity events at any time through the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Web UI.

Fixity checks do not alter the AIP or any of its component files. Failed fixity checks generate alerts to both APTrust staff and the depositing institution.

When a fixity check fails, indicating a missing or corrupt file, we retrieve another copy of the same file from a different storage location (Glacier in Oregon), validate the fixity on that copy, then overwrite the bad copy with with the valid copy. This process currently requires intervention from APTrust staff. We have scripts to repair missing and corrupt files, but an APTrust staff member must manually run the scripts. We ran them in 2017 to fix a number of files that were incorrectly copied to long-term storage upon ingest, so we know the scripts work.

Administrative Workflows

Defining a collection

Key management

Each member institution requires security credentials to access their S3 storage buckets for ingest and restore. AWS identity and access management (IAM) is used to create and manage keys.

File Access

An institution is provided with two sets of access credentials, one for the production system and one for the demo system. At the time of institution setup access credentials are shared with the designated institutional administrator. The credentials are securely shared using privnote.com which provides a one-time link to an encrypted message which destroys itself after opening it. At this point it becomes the institutions responsibility to keep access credentials sufficiently secure. The institution is required to notify us if a key is compromised and needs to be revoked and a new one issued.

As stated in our security principles we granting least privilege—that is, granting only the permissions required to perform a task. Hence access keys only allow institutions to access their four ingest and restore buckets (demo and production) with limited access to write, read and download objects (files) from their buckets. It does not allow access to any other AWS service.

API Access

An institution has access to APTrusts' repository frontend "PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata." which allows for various read operations through it's member APIApplication Programming Interface. During on boarding an institution provides a list of users (Name and Email) that shall have access to the repository. An APTrust administrator creates user accounts on the demo and production PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. system and notifies the institutions users to enter their provided email address and click the "Forgot Password?" link on the repositories home page. This generates an email with a one-time link to create their password. We advise users to use a secure password but do not require any minimum password complexity.

At the time of writing the API only provides read endpoints, hence no data modification is possible using API access.

Future developments will improve security and practices as stated in https://www.pivotaltracker.com/story/show/154729518

Types of access from APTrust Staff

There are various types of access to the material deposited into APTrust. These can be broken down into authorized and unauthorized uses by APTrust Staff.

Authorized

  • Ingest
  • Update
  • Deletion (only with permission from admins at depositor institution)
  • Restore
  • Create and revoke legitimate user accounts (PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata.)
  • Create and revoke legitimate AWS access accounts
    • all AWS access accounts resolve to an individual

Unauthorized

  • Delete/alter production materials without depositor request/authorization
  • By policy, viewing restricted metadata and files from depositors is not authorized (unless required for system administration and maintenance)
    • Recommended practice: Depositors use encryption for sensitive materials
  • Alter any depositor content (i.e. metadata and file)
  • Moving/expose/distribute a depositor's content without express authorization from depositor

In general UVA staff adheres to the following UVA policy:

http://uvapolicy.virginia.edu/policy/IRM-012

Security

Adminonly_Technical Documentation

Technical Documentation

Architecture

APTrust Architecture
APTrust Architecture
  • EC2 Instances: Serves the foundational server environment where all non-native AWS services, software and applications are deployed. Major components of the infrastructure running on EC2 instances are:
    • Exchange (Content Processing Scripts): Go language services that manage work queues to monitor for the arrival of content, process content, register metadata with PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. in it's PostgresSQL database and move content to preservation storage. Additionally a set of Go scripts will also manage work queue around file life-cycle and processing (fixity) as well as restoration by re-packaging intellectual objects and related generic files back int an APTrust BagIT bag.
    • PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. - Our Rails app provides a “registry” describing what IntellectualObjects and GenericFiles are in our repository, along with checksums for those files and PREMIS events describing actions taken on those objects and files. PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. also keeps track of WorkItems, which are requests to do something with an object of file. DPN replication and restore requests have their own table in PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata., dpn_work_items. Finally, PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. stores blobs of compressed JSON in the WorkItemState table, which is described below. PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. stores its data in Multi-AZ Postgres RDS instance that contains local or workflow data about object processing, user data for admin interface authentication, as well as generic file metadata.

APTrust services run on apt-demo-services for our demo environment, and on apt-prod-services for our production environment. Each environment has separate receiving, restoration, and storage areas in S3.

  • S3 (Northern VA) - Partners upload new bags to the receiving buckets here, which we query every hour or so. Partners download restored bags from the restore buckets here, which is where our apt_restore service drops restored bags for depositors. The bucket called aptrust.preservation.storage is where we store ingested files for the long term. That bucket is accessible to the APTrust admin account only. The buckets are distinguished by use:
    • Receiving Buckets: Each APTrust member has an individual S3 bucket designated for the upload of submission packages to APTrust and to facilitate the hand-off of content. Access to each bucket is restricted to a designated institution who have PUT and LIST permissions or the APTrust processing scripts which have full permissions
    • Preservation Bucket: A single S3 bucket is used for central preservation storage. Files to be preserved are placed here with pointers to the file stored in the corresponding PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. object along with any relevant metadata.
    • Restoration Buckets: Each APTrust member has an individual S3 bucket designated for the download of distribution packages for restoration. Access to each bucket is restricted to the designated institution who has LIST, GET and DELETE permissions to that bucket as well as full access to the APTrust processing scripts.
  • AWS Glacier (Oregon) - This is where we store replication copies of all ingested files, and is accessible to the APTrust admin account only.
  • apt-prod-service & apt-demo-services - These servers run the processes that perform ingest, file deletion, bag restoration, DPN ingest, and ongoing fixity checks.
  • NSQ (runs on apt-prod-service and apt-demo-services) - cron jobs like apt_queue query PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. for outstanding WorkItems and push the WorkItem IDs into the proper NSQ topics. For example, the ID of a WorkItem requesting a deletion goes into the apt_file_delete topic in NSQ.
    • NSQ pushes WorkItem IDs to the workers that subscribe to its channels. apt_file_delete subscribes to the apt_file_delete channel, so it gets the IDs of file deletion WorkItems.
  • The workers query PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. to get the full WorkItem record associated with the ID that NSQ pushed to them. The workers also query PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. for data associated with that WorkItem, including:
    • The IntellectualObject associated with the WorkItem, if there is one.
    • The GenericFile associated with the WorkItem, if there is one.
    • The WorkItemState associated with the item, if there is one.
Pharos DB ER Diagram
PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. DB ER-Diagram
WorkItemState is a Rails object that contains a blob of JSON describing the state of a WorkItem as it progresses through the queues and workers. The APTrust services are like a series of conveyor belts in a factory. As a WorkItem and its associated IntellectualObject or GenericFile passes from one worker to the next, it is accompanied by a manifest, which the worker fills out as it goes. The manifests are different for each process (ingest, restoration, deletion, etc.) because the work that needs to be done differs.

For example, each of the ingest workers adds information to an ingest manifest as it does its work. The first worker, apt_fetch, records where on the local file system it stored the tar file it just downloaded. It also records the names and checksums of all the GenericFiles it found inside the tar file, and any validation errors it encountered.

When a service like apt_fetch is done working on a WorkItem, it converts its manifest to JSON and sends that data back to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. to be saved in the WorkItemState table. PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. compresses the data before saving it, because it can get quite large, especially in the case of ingest, and tends to compress to about 10% of its full size.

When the next worker pick up the WorkItem, it pulls the WorkItemState record from PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. and from that it knows all it needs to know to do its job intelligently. For example, both apt_store, which copies files to long-term storage, and apt_record, which tells PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. what files have been stored, can and do stop work on an item after encountering too many transient errors. (Transient errors are almost always network errors or problems with disk I/O.) They will record the WorkItemState in PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. before requeuing the task in NSQ.

The next worker to pick up the task then loads the WorkItemState from PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. and knows what work has been done and what has not. For example, it’s common for the system to record 500 of an IntellectualObject’s 1000 files on the first ingest attempt before running into some network problem. The next apt_store worker to pick up the ingest request will know it doesn’t have to store the first 500 files, and it starts its work at file #501.

Most services write to two logs in the /mnt/efs/apt/logs directory: one called <service>.log, and one called <service>.json. The .log file is meant to be human-readable. The .json file is meant to be machine-readable. In cases where a service was not able to send a WorkItem’s JSON state back to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. (in the form of a WorkItemState object), you should be able to find the JSON in the .json log. A special tool called apt_json_extractor can quickly pull individual JSON records out of very large JSON logs.
APTrust Software Stack
APTrust Software Stack

Security

Introduction

The Academic Preservation Trust is operated by assignment from its Governing Board to the University of Virginia as a cost-recovery activity of the University of Virginia Library in a spirit of collaboration, openness, and transparency. The underlying preservation-storage infrastructure of APTrust is provided by cloud-based vendors whose services are available to the APTrust through the University of Virginia's access to relevant Internet2 Net+ contracts. Over time, access to cloud services may be available to APTrust through other UVA purchasing "vehicles."  APTrust's descriptions of specific services include identification of which contracts are involved with each (see APTrust Services and Fees List). Security of vendor-supplied services is generally provided by those vendors and has been reviewed as part of the establishment of ways to purchase the services by the contracting entity (such as Internet2 or the University of Virginia--see, for example, section 5 of the Internet2 contract information related to AWS services). Our focus in this section is on the services and technologies that APTrust directly controls.

[CHRISTIAN: AS DEVELOPMENT OF THIS DOCUMENT CONTINUES, PLEASE SEE MY OUTLINE AT https://docs.google.com/document/d/1hxbR0QnfZAnrrWSCqdAoF5w_NMPyN8SUgNgl-zyX2iI/edit?usp=sharing FOR CATEGORIES WE NEED TO ADDRESS BASED ON AN EVOLVING NDSA DOCUMENT]

Security Principles

  • APTrust's IT environment is tightly controlled and access is on a minimum necessary basis.
  • Least Privilege: APTrust follows the standard security advice of granting least privilege—that is, granting only the permissions required to perform a task. Otherwise we default to DENY ALL. This means that all access is denied by default and only granted to specific users if absolutely necessary.
  • Only designated operators have shell access to servers.
  • We use Multi-Factor-Authentication (MFA) wherever available.
  • We use password-less logins (SSH keys).
  • Password credentials are stored encrypted.
  • Defense-in-depth: we employ multiple layers of security controls (defense) to provide redundancy in the event a security control fails.

Authentication & Administration

Authentication is defined as validating a user's identity and basic-level authorization through determining if a user is allowed to access the system.

All services that APTrust provides and third-party services that we use require authenticated access. Wherever possible the means of authentication are securely stored using encryption. If encryption is not possible, we strongly limit access to resources that use plain-text authentication (such as environment variables on infrastructure).

Account Provisioning

Account provisioning consists of the creation of necessary accounts (PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. user account, AWS IAM accounts for S3 access) to use the APTrust preservation repository.

Only designated APTrust administrators create accounts and provide credentials to the account owners in a secure fashion. Many institutions have been receving only two institutional AWS IAM accounts (for demo and production) to provide access to S3 storage (ingest and restore buckets). As of 2018, any new institutions will get individual user accounts and generic institution accounts only on request. This provides a better audit trail and enhances security while lowering a blast radius. The APTrust administrator provides credentials per privnote.com.

Accounts are only created for users that have been specifically identified by a designated institutional administrator. User emails are required to be under the institution's domain, e.g. Institutions: University of Virginia User: Christian Dahlhausen; Email: christian.dahlhausen@virginia.edu

APTrust Staff:

Institutional Staff:

Account Access Termination

Institutional member accounts: Once credentials have been provided the institution is responsible to store them safely and report if a compromise of the credential has been likely. If an credential is compromised the APTrust administrator deactivates the credentials and issues new ones to the institution. The institutional member is responsible for sub-accounts on APTrust PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata., either subscriber accounts or institutional member accounts. This includes but is not limited to:

  • PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. accounts and API keys
  • AWS access keys
  • Google document access

APTrust staff accounts: When there is APTrust staff attrition one of the APTrust administrators deactivates/deletes all accounts that pertain to access of internal and external systems. This includes but is not limited to:

  • APTrust server accounts (SSH keys)
  • AWS credentials
  • Passpack account
  • Google Apps account
  • PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. account
Account Deprovisioning

Account Deprovisioning details the disposition of the account in the service, and management of the user-generated data (if any) related to the account.

this particularly pertains to subscriber institutions when the member institution leaves APtrust...

Roles and Authorization

Following designated user groups with varying level of authorization. The table follows the CRUD (create-read-update-delete) to denote permissive authorization.

Role Description PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. AWS S3 Google Apps Wiki ???
Anonymous Users A non-authenticated or unidentified visitor to the site. - - - R
Authenticated Users Any user that has an account on any of our systems. This includes vendors. CRUD
Institutional Member Any user that is an institutional member of APTrust by definition of the depositor agreement. CRUD1 CRUD2 - CRUD
Institutional Admin Any user that is an institutional member of APTrust by definition of the depositor agreement. Inst. Admin has full access to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. and is able to create and delete user accounts for it's institution. CRUD1 CRUD2 CRUD
APTrust Staff CRUD
-- General Staff General administrative staff R R R CRUD
-- Technical Staff Technical staff with some access to APTrust systems. CRUD CRUD RU CRUD
-- Administrator/Operator Technical staff with full administrative access. Has full access for troubleshooting and management of the applications. Is limited to a few lead or Sr roles in APTrust CRUD CRUD CRUD CRUD
Access Control Granularity
Access Permissions in APTrust PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata.
Consortial Institution Restricted
APTrust Consortial Members Institution-Specific Restricted None
Role Type Own Other Own Other Own Other Own Other
Anonymous Discover No No No No No No No No
View No No No No No No No No
Edit No No No No No No No No
Authenticated Discover No No No No No No No No
View No No No No No No No No
Edit No No No No No No No No
APTrust Admin Discover Yes Yes Yes Yes Yes Yes Yes Yes
View Yes Yes Yes Yes Yes Yes Yes Yes
Edit Yes Yes Yes Yes Yes Yes Yes Yes
Insitutional Admin Discover Yes Yes Yes No Yes No No No
View Yes Yes Yes No Yes No No No
Edit No No No No No No No No
Institutional User Discover Yes Yes Yes No Yes No No No
View Yes Yes Yes No No No No No
Edit No No No No No No No No

Access and Identity Tools

AWS Identity and Access Management (IAM)

AWS Access credentials

For access of ingest and restore storage buckets (S3) a user requires AWS Access Credentials (AWS ACCESS KEY and AWS SECRET ACCESS KEY). These are generated using AWS IAM. At the time of writing each member institution has one institutional account with two sets of access credentials. One set is for APTrust's production environment and one set is for the demo/development environment. The credentials are shared with the institutional administrator (for APTrust) using privnote. After the credentials have been provided the privnote destroys itself and the institutional administrator is responsible keeping the access credentials secure. APTrust makes stores each credential additionally in Passpack as a backup.

AWS Access policies

Each IAM account has access policies that describe the level of access to services. Member institutions have limited access on their S3 storage buckets and are only allowed certain actions on the data objects. Institutional accounts do not have access to any other AWS services besides S3.

Passpack

Passpack is an online password management tool that stores all credential data encrypted. APTrust has a main business account (aptrust) that manages all credentials pertaining APTrust used services. Each member of the APTrust operations and development team has their own account to which some of the password entries are shared with. An admin account has access to all APTrust related credentials and is only accessible by one of APTrusts designated administrators. Passpack entries are encrypted using Double Lock -- AES-128 bit encryption by default. It is possible to use higher encryption per credential (Triple Lock -- AES-256 bit encryption). We encourage using the higher encryption standard when new entries are created. APTrust staff is storing AWS access keys of member institutions in that datastore as well.

Workflow: A user is creating a Passpack entry and transfers ownership to the main APTrust admin account. That way the admin can re-share the credentials to other members of the team and manage editing permissions. 

Ansible Vault

Sensitive server and service information is stored using Ansible Vault.  Vault encrypts any text file with AES-256 encryption. It uses a single password to encrypt a file. When running Ansible playbooks and roles Ansible prompts for the encryption/decryption password in order to read the vault file(s).  For further information on how to use it, read here.

The Ansible Vault password is stored in Passpack. 

Auditing

Logging

PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. Application Server Logs

The Ruby application server logs activity in its application logs. Most errors are presented to the user on the web frontend itself, some are only logged in the application logs that are not accessible by the end-user. APTrust staff is spot checking server logs occasionally and errors are reported by Logwatch per email.

Nginx Webserver Logs

The web server logs access to PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. and errors from the Ruby application server. These are parsed and evaluated by Logwatch as well and APTrust staff is notified per email if non-regular activities occur.

Exchange - Ingest/Restore/Fixity Services logging
APTrust keeps extensive logs from each micro service from the Exchange suite. Below an example excerpt from downloading an object from ingest in order to be processed on the server. Each result dictionary tracks data and timestamps about the step in the ingest process.
 1 -------- BEGIN aptrust.receiving.templateuniversity.edu/templateuniversity.edu.10822_1046084.tar | Etag: fef0a540264e38912e9f6c4bc3268d9e | Time: 2018-04-09T16:16:07Z --------
 2  {
 3   "WorkItemId": 181485,
 4   "S3Bucket": "aptrust.receiving.templateuniversity.edu",
 5   "S3Key": "templateuniversity.edu.10822_1046084.tar",
 6   "ETag": "fef0a540264e38912e9f6c4bc3268d9e",
 7   "BagPath": "/mnt/lvm/apt/data/templateuniversity.edu/templateuniversity.edu.10822_1046084.tar",
 8   "DBPath": "/mnt/lvm/apt/data/templateuniversity.edu/templateuniversity.edu.10822_1046084.valdb",
 9   "FetchResult": {
10     "Attempted": true,
11     "AttemptNumber": 1,
12     "ErrorIsFatal": false,
13     "Errors": [],
14     "StartedAt": "2018-04-09T16:16:05.215809131Z",
15     "FinishedAt": "2018-04-09T16:16:06.6678436Z",
16     "Retry": true
17   },
18   "UntarResult": {
19     "Attempted": false,
20     "AttemptNumber": 0,
21     "ErrorIsFatal": false,
22     "Errors": [],
23     "StartedAt": "0001-01-01T00:00:00Z",
24     "FinishedAt": "0001-01-01T00:00:00Z",
25     "Retry": true
26   },
27   "ValidateResult": {
28     "Attempted": true,
29     "AttemptNumber": 1,
30     "ErrorIsFatal": false,
31     "Errors": [],
32     "StartedAt": "2018-04-09T16:16:06.906390067Z",
33     "FinishedAt": "2018-04-09T16:16:07.442069031Z",
34     "Retry": true
35   },
36   "StoreResult": {
37     "Attempted": false,
38     "AttemptNumber": 0,
39     "ErrorIsFatal": false,
40     "Errors": [],
41     "StartedAt": "0001-01-01T00:00:00Z",
42     "FinishedAt": "0001-01-01T00:00:00Z",
43     "Retry": true
44   },
45   "RecordResult": {
46     "Attempted": false,
47     "AttemptNumber": 0,
48     "ErrorIsFatal": false,
49     "Errors": [],
50     "StartedAt": "0001-01-01T00:00:00Z",
51     "FinishedAt": "0001-01-01T00:00:00Z",
52     "Retry": true
53   },
54   "CleanupResult": {
55     "Attempted": false,
56     "AttemptNumber": 0,
57     "ErrorIsFatal": false,
58     "Errors": [],
59     "StartedAt": "0001-01-01T00:00:00Z",
60     "FinishedAt": "0001-01-01T00:00:00Z",
61     "Retry": true
62   },
63   "Object": {
64     "state": "A",
65     "created_at": "0001-01-01T00:00:00Z",
66     "updated_at": "0001-01-01T00:00:00Z",
67     "ingest_deleted_from_receiving_at": "0001-01-01T00:00:00Z"
68   }
69 }
70  -------- END aptrust.receiving.templateuniversity.edu/templateuniversity.edu.10822_1046084.tar | Etag: fef0a540264e38912e9f6c4bc3268d9e | Time: 2018-04-09T16:16:07Z --------
Preservation Storage Logging

Activity on the preservation bucket is logged using AWS standard logging to a bucket named `aptrust.preservation.logging` for deeper auditing purposes and security.  This is in addition to any logging already provided by locally coded content services.

AWS Cloudtrail Logs
AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. CloudTrail provides event history of your AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command line tools, and other AWS services. This event history simplifies security analysis, resource change tracking, and troubleshooting.[1]
APTrust keeps and "AuditTrail" log that stores activity in all regions, all management activities and all activities on all S3 buckets. The log is continuously stored in an S3 bucket "cloudtrail-logs" and is used as a complete audit trail of all AWS related activities. The trail is currently not processed or triggers any alarms.

Monitoring and Alerting

APTrust is using multiple monitoring tools to ensure the systems stability and insight into resource usage.

Icinga2

The Icinga2 monitoring suite is based on de-facto-industry standard Nagios but has a couple of additional features. It alerts APTrust staff by Slack notification about service interruptions or performance issues upon which the ops team can react. Each alert can be "acknowledged" using the Icinga2 web interface. Documenting text can be added to the acknowledgement. Icinga2 also provides a history of past alerts for auditing purposes.

Grafana & InfluxDB

InfluxDB provide time-series data about resource usage and performance. Grafana is a web-frontend and dashboard to visualize that data. Data is fed from an Icinga2 plugin that runs on every instance. It also polls AWS directly. With the time-series data the operations team can identify trends in resource usage and act accordingly. It aids in scaling decisions and resource usage over time.

Fail2ban

Fail2Ban scans SSH and Nginx log files to identify malicious actions. Once IP's have been determined to be malicious the tool puts them in a `jail` (server-local IPtables firewall jail) to prevent any further malicious activity from that IP.

Logwatch

Logwatch is a customizable log analysis system. Logwatch parses through your system's logs and creates a report analyzing areas that you specify. A daily cron job runs the program and sends out a summary email from each node. The operations staff is spot reviewing the logwatch report emails. An example as follows:

---------- Forwarded message ----------
From: Cron Daemon <ops@aptrust.org>
Date: Sun, Nov 5, 2017 at 7:00 PM
Subject: Cron <root@apt-demo-repo2> /usr/sbin/logwatch
To: ops@aptrust.org



 ################### Logwatch 7.4.0 (05/29/13) ####################
        Processing Initiated: Mon Nov  6 00:00:02 2017
        Date Range Processed: yesterday
                              ( 2017-Nov-05 )
                              Period is day.
        Detail Level of Output: 5
        Type of Output/Format: stdout / text
        Logfiles for Host: apt-demo-repo2
 ##################################################################

 --------------------- Cron Begin ------------------------

 Commands Run:
    User root:
          cd / && run-parts --report /etc/cron.hourly: 24 Time(s)
       /usr/sbin/logwatch: 1 Time(s)
       test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew: 2 Time(s)
       test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ): 1 Time(s)
       test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ): 1 Time(s)
    User ubuntu:
       . $HOME/.profile; /var/www/demo.aptrust.org/pharos/current/bin/pharos_notify.py >> /var/www/demo.aptrust.org/pharos/current/log/cron_pharos_notify.log 2>&1: 1440 Time(s)
       /etc/psql_backup/pg_backup_rotated.sh >> /var/log/psql_backup.log 2>&1: 1 Time(s)

 ---------------------- Cron End -------------------------


 --------------------- httpd Begin ------------------------

 144.53 MB transferred in 22982 responses  (1xx 0, 2xx 22944, 3xx 36, 4xx 2, 5xx 0)
        4 Images (0.02 MB),
        4 Documents (0.04 MB),
     8463 Content pages (41.26 MB),
        1 Redirects (0.00 MB),
    14510 Other (103.21 MB)

 Requests with error response codes
    400 Bad Request
       /w00tw00t.at.ISC.SANS.DFind:): 1 Time(s)
    404 Not Found
       /a2billing/admin/Public/index.php: 1 Time(s)

 A total of 5 ROBOTS were logged

 ---------------------- httpd End -------------------------


 --------------------- pam_unix Begin ------------------------

 cron:
    Sessions Opened:
       ubuntu: 1441 Time(s)
       root: 29 Time(s)

 sshd:
    Sessions Opened:
       cd3ef: 1 Time(s)

 sudo:
    Sessions Opened:
       root -> root: 36 Time(s)
       root -> ubuntu: 6 Time(s)


 ---------------------- pam_unix End -------------------------


 --------------------- Postfix Begin ------------------------

 ****** Summary *************************************************************************************

    5.010K  Bytes accepted                               5,130
    5.010K  Bytes sent via SMTP                          5,130
 ========   ==================================================

        1   Accepted                                   100.00%
 --------   --------------------------------------------------
        1   Total                                      100.00%
 ========   ==================================================

        1   Removed from queue
        1   Sent via SMTP

 ****** Detail (1) **********************************************************************************

        1   Sent via SMTP ---------------------------------------------------------------------------
        1      aptrust.org

 === Delivery Delays Percentiles ============================================================
                     0%       25%       50%       75%       90%       95%       98%      100%
 --------------------------------------------------------------------------------------------
 Before qmgr       0.01      0.01      0.01      0.01      0.01      0.01      0.01      0.01
 In qmgr           0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
 Conn setup        0.17      0.17      0.17      0.17      0.17      0.17      0.17      0.17
 Transmission      0.29      0.29      0.29      0.29      0.29      0.29      0.29      0.29
 Total             0.48      0.48      0.48      0.48      0.48      0.48      0.48      0.48
 ============================================================================================

 ---------------------- Postfix End -------------------------


 --------------------- SSHD Begin ------------------------


 Users logging in through sshd:
    cd3ef:
       216.197.76.234 (c-va-3d8452d8df-23486-1.tingfiber.com): 1 time


 Received disconnect:
    11: disconnected by user
       216.197.76.234 : 1 Time(s)

 ---------------------- SSHD End -------------------------


 --------------------- Sudo (secure-log) Begin ------------------------


 cd3ef => root
 -------------
 /bin/sh                        -  36 Time(s).

 cd3ef => ubuntu
 ---------------
 /bin/sh                        -   6 Time(s).

 ---------------------- Sudo (secure-log) End -------------------------


 --------------------- Disk Space Begin ------------------------

 Filesystem                                 Size  Used Avail Use% Mounted on
 udev                                       2.0G   12K  2.0G   1% /dev
 /dev/xvda1                                  32G  4.0G   27G  14% /
 fs-97ff5bde.efs.us-east-1.amazonaws.com:/  8.0E  3.7G  8.0E   1% /mnt/efs/apt


 ---------------------- Disk Space End -------------------------


 ###################### Logwatch End #########################
AWS Cloudwatch

By default all instances in Amazon Web Services have basic Cloudwatch monitoring enabled which provides seven pre-selected metrics at five-minute frequency and three status check metrics at one-minute frequency[2]. All production instances (EC2 and RDS) have detailed Cloudwatch enabled which include additional metrics.

Currently there are no Cloudwatch alarms enabled on these metrics but rather used for a post-mortem analysis or a second layer of metrics. APTrust uses a combination of Grafana and Icinga2 metrics for troubleshooting and alarms.

Business Continuity

Interopability and Portability

Backup

Disaster Recovery

Risk Management, Threats, and Mitigations

Risk Management describes casting and evaluation of risks together with the identification of procedures to avoid or minimize their impact. Due to the repositories' architecture and usage of Amazon Web Services (AWS) some of the threats and mitigations are covered by a Shared Security Responsibilty Model[1] and hence shared between APTrust and Amazon Web Services (AWS) as infrastructure provider.

The system architecture and operations policies of APTrust are based on the threat model which was formalized in a 2005 paper published by the LOCKSS team, Requirements for Digital Preservation Systems: A Bottom-Up Approach, and periodic reviews of code, configuration and policies. The identified threats are not unique to digital preservation repositories but sometimes require different mitigation strategies due to the long time horizon preservation systems are build and maintained for.

The paper identified the following threats:

Media Failure

All storage media must be expected to degrade with time, causing irrecoverable bit errors, and to be subject to sudden catastrophic irrecoverable loss of bulk data such as disk crashes[2] or loss of off-line media.
Mitigation

- APTrust stores multiple copies of data objects in multiple locations (Virginia and Oregon) to mitigate localized failures.

- Regular fixity checks ensure the accuracy and authenticity of the data object.

- The storage backend used, AWS Simple Storage Service (S3), are designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities[3]. A second tier storage Amazon Glacier also redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives. Glacier performs regular, systematic data integrity checks and is built to be automatically self-healing[4].

- Storage media is managed by AWS and underlying "disk crashes" mitigated by the vendor as part of their shared security responsibilities[1].

Hardware Failure

All hardware components must be expected to suffer transient recoverable failures, such as power loss, and catastrophic irrecoverable failures, such as burnt-out power supplies.
Mitigation

- AWS manages all of our infrastructure as Infrastructure as a Service (IaaS) which covers monitoring, management and replacement of faulty hardware.

- If hardware outages occur APTrust is able to recover quickly due to it's automated provisioning and configuration management. New server instances can be provisioned quickly.

- Due to continuous monitoring and management of hardware by AWS, failures are rare.

Software Failure

 All software components must be expected to suffer from bugs that pose a risk to the stored data.
Mitigation

- APTrust adheres to software development guidelines that include regular unit and integration tests with written software. We employ a continuous integration service (Travis CI) that runs unit tests after every git commit. Failing unit tests are fixed immediately after they become apparent. Travis CI actively notifies the APTrust team via Slack about success of failure.

- In addition to unit tests APTrust applies an integration tests to make sure services are running and interacting as a system as expected. Integration tests are run locally before every git commit.

- The release process consists of a deploy on a demo/staging environment first. After successful deployment there, software is deployed into production.

- For destructive actions like deletion of ingested intellectual objects APTrust implemented a two factor deletion process that requires an institutional administrator to approve a deletion request before the object gets deleted.

Communication Errors

Systems cannot assume that the network transfers they use to ingest or disseminate content will either succeed or fail within a specified time period, or will actually deliver the content unaltered. A recent study "suggests that between one (data) packet in every 16 million packets and one packet in 10 billion packets will have an undetected checksum error."
- All communication channels the APTrust repository uses are configured to use TLS/SSL[5] to communicate which ensures secure communication as well as data integrity in transit[6].

- Ingest/Submission: Before data is transmitted for ingest to APTrust, a tag manifest file that includes checksums for each individual file in the bag is included and used for verification of data integrity during the ingest process. Bags without manifest files are invalid and not being ingested.

- Restore/Dissemination: When content restoration is requested, a distribution package will be created consisting of the original files and metadata written to a BagIt bag conforming to the current APTrust BagIt profile.  The files in the data directory will be the exact same name and bits that were sent to APTrust in the submission bag and the metadata written to tag files in the bag adhering to the current APTrust BagIt format.  

Failure of Network Services

Systems must anticipate that the external network services they use, including resolvers such as those for domain names [39] and persistent URLs [44], will suffer both transient and irrecoverable failures both of the network services and of individual entries in them. As examples, domain names will vanish or be reassigned if the registrant fails to pay the registrar, and a persistent URL will fail to resolve if the resolver service fails to preserve its data with as much care as the digital preservation service.
APTrust uses currently only one DNS provider (Network Solutions), it is possible that the DNS may fail and no secondary DNS will be failed over to. This may affect the repository services.

Ingest: The ingest services are running on a single node that does not rely on DNS as all services are accessing itself locally. Failure of either is presumed to be transient, and thus to delay but not prevent ingest.

Restore: The restore process is triggered by the repositories web frontend PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. which in case of resolver issues wouldn't be reachable and prohibit new restores. Failure of either is presumed to be transient, and thus to delay but not prevent restore.

Risk: Since all services and infrastructure resides in a cloud environment that has redundancy and adheres to SLAs long outages are unlikely. In case of a DNS outage system administrators could define local resolvers to keep internal services running. The external repository frontend may be compromised.

Media & Hardware Obsolescence

All media and hardware components will eventually fail. Before that, they may become obsolete in the sense of no longer being capable of communicating with other system components or being replaced when they do fail. This problem is particularly acute for removable media, which have a long history of remaining theoretically readable if only a suitable reader could be found.
APTrust is solely running on AWS infrastructure which is subject to the shared responsibility model[7]. AWS assumes responsibility of underlying hardware replacement and obsolescence. APTrust is informed of degraded hardware and if migration is necessary.
AWS Shared Responsibility Matrix

Risk: The assumed risk is low as APTrust is using Infrastructure as a service (IaaS) which makes management of physical hardware obsolete and hardware failure a non-issue.

Software Obsolescence

Similarly, software components will become obsolete. This will often be manifested as format obsolescence when, although the bits in which some data was encoded remain accessible, the information can no longer be decoded from the storage format into a legible form.
All software used in the APTrust repository is open source and most heavily community supported. We employ regular update and upgrade of software libraries to avoid security issues or out of date software. Wherever possible APTrust tries to keep a copy of software dependencies with its software.

Data format obsolescence is a risk which mostly lies with the depositors. The repository is kept up-to-date and functional as far as preservation activities go.

Since all the software is free and open-source, no financial provision need be made for its replacement or upgrade.

Risk: The risk of software obsolescence is assessed low. All software components are managed and updated on a regular basis.

Operator Error

Operator actions must be expected to include both recoverable and irrecoverable errors. This applies not merely to the digital preservation application itself, but also to the operating system on which it is running, the other applications sharing the same environment, the hardware underlying them, and the network through which they communicate.
Software:

Each software component of the repository that is developed by APTrust staff is unit tested with every commit. Before ingest, restore or fixity services are being deployed integration tests are executed to ensure predictable functionality and avoid bugs.

Infrastructure & Configuration:

APTrust strives to use automation wherever possible. With the advent as infrastructure-as-code and application of software engineering principles to infrastructure management many risks are mitigated or lowered by carefully crafted and tested automations.

Security:

APTrust employs several security principles throughout the development, deployment, and administrative processes. As part of the defense-in-depth principle each environment (production, demo, and test) are physically (deployed on different servers) and virtually (AWS security groups prohibit network access) separated. Each environment has its own set of credentials, even if an operator were to deploy a demo environment on a production instance, none of the access credentials would work hence avoid possible destructive actions

Risk: By applying software engineering principles and a suite of tests before anything gets put into production the risk of error is assessed as low.

Natural Disaster

APTrusts content is stored in geographically diverse locations. The primary S3 storage is part of the US-Standard region and routes copies across the Northern Virginia and Pacific Northwest[8].

A disaster affecting Northern Virginia may interrupt ingest, restore and web frontend services but due to automated provisioning of most of the infrastructure can be completely re-provisioned within an hour in a different geographic location.

Risk: Given that the risk of a natural disaster in Northern Virginia is relatively low,[9] the risk of affecting ingest/restore services is low.

External Attack

As documented in APTrust Security each instance is configured with minimal outside access (if at all) to prevent malicious actions. All communications among each node and service are encrypted using SSH and TLS. Database instances are not publicly accessible from the outside and are firewall controlled only allowing certain instances access. The surface available to an external attacker is thus minimized.

Each server instance is patched on a regular schedule (every 24 hours), security updates are applied immediately. Non-security updates may require administrator intervention but are usually applied bi-weekly.

Firewall (AWS security groups) are reviewed monthly.

The following precautions are taken to prevent unauthorized access to APTrusts web interface:

  • - Access per SSL/TLS only
  • - Firewall rules deny all except web.
  • - Administrative shell access is limited to designated APTrust operators.
  • - Access only with an account and password credential
  • - Separation of administrative and user accounts per institution

Risk: A security update could be released that is faulty. If an attacker would gain access to the web interface using someone elses access credentials he would be able to issue deletion or restore requests. Deletion requests however require a second factor (administrators permission). The risk is assessed moderate.

Internal Attack

Members of the APTrust staff have delineated roles, responsibilities and authorizations regarding changes and access.

  • Administrative access to nodes is limited to 3 employees. Only two employees have superuser (sudo) access.
  • All access is logged.
  • Non-technical staff has administrative access to the web interface. Staff that has administrative access has been vetted.

Risk: There is a risk that staff could act maliciously. However the risk is limited to only two employees with sudo, system level access, and three administrative access. All employees are vetted by background checks before being hired and malicious intend is unlikely. The risk is moderate.

Economic Failure

APTrust is funded by it's sustaining members by a yearly fee as well as the University of Virginia, which funds above the nominal fee per year. APTrust keeps to a budget and maintains enough reserves to sustain beyond a 1-year period without any incoming funds. Loss of APTrust Repository sustaining members would reduce funding but not affect its operations.

APTrust has been economically sustainable for the past 4 years.

Risk: The risk of economic failure is low as the organization is healthy and operates economically. However loss of multiple sustaining members would increase the risk of failure. The current risk is assessed as low.

Organizational Failure

APTrust has not formalized a succession plan yet but is well aware of the risk. In case of an organizational failure deposits could reside in AWS and custody and costs could be transferred to it's owner/sustainable member.

APTrust has drafted a succession policy that, once approved can mitigate this risk.

Risk: The risk is assessed as moderate.

Relevant Documents

  1. Requirements for Digital Preservation Systems: A Bottom-Up Approach
  1. 1.01.1 https://aws.amazon.com/compliance/shared-responsibility-model/
  2. TALAGALA, N. Characterizing Large Storage Systems: Error Behavior and Performance Benchmarks. PhD thesis, CS Div., Univ. of California at Berkeley, Berkeley, CA, USA, Oct. 1999. <http://www.eecs.berkeley.edu/Pubs/TechRpts/1999/CSD-99-1066.pdf>
  3. https://aws.amazon.com/s3/faqs/
  4. https://aws.amazon.com/glacier/faqs/
  5. https://en.wikipedia.org/wiki/Transport_Layer_Security
  6. https://en.wikipedia.org/wiki/Message_authentication_code#Message_integrity_codes
  7. This shared model can help relieve customer’s operational burden as AWS operates, manages and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the service operates. The customer assumes responsibility and management of the guest operating system (including updates and security patches), other associated application software as well as the configuration of the AWS provided security group firewall. Customers should carefully consider the services they choose as their responsibilities vary depending on the services used, the integration of those services into their IT environment, and applicable laws and regulations. The nature of this shared responsibility also provides the flexibility and customer control that permits the deployment. As shown in the chart below, this differentiation of responsibility is commonly referred to as Security “of” the Cloud versus Security “in” the Cloud. https://aws.amazon.com/compliance/shared-responsibility-model/
  8. Preservation and Storage#Geographic Diversity
  9. https://www.washingtonpost.com/news/capital-weather-gang/wp/2013/08/29/d-c-s-maryland-suburbs-have-among-nations-lowest-disaster-risk/?utm_term=.151d09279e15

Monitoring

Member API

The member-api allows depositors to programmatically query for status of events, files, intellectual objects or items in a work queue.


Preservation and Storage

APTrust breaks bags into individual files upon ingest and saves each file to Amazon's S3 storage in Northern Virginia and to Glacier storage in Oregon. We maintain a central registry describing all intellectual objects, which files belong to each object, and where those files are stored. The registry is a SQL database that is replicated across multiple AWS availability zones in Virginia, and to Oregon. We also store daily and weekly snapshots of the database on an EBS (disk) drive in Amazon's Virginia data center. (This is the SQL database underlying our PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. application, which provides access to data through both a Web UI and a REST APIApplication Programming Interface.)

In addition to the SQL database, we tag each file stored in S3 and Glacier with the following metadata:

  • Content Type - This is stored in as a mime type. E.g. image/jpeg, text/plain, etc.
  • Bag - This is the name of the intellectual object to which the file belongs.
  • Bag Path - This is the original path of the file within the bag (SIP) when we received the bag for ingest.
  • Institution - The domain name of the institution that owns the file. For example, virginia.edu, ncsu.edu, etc.
  • MD5 - The md5 checksum of the file.
  • SHA256 - The sha256 checksum of the file.

If APTrust were to lose its copies of the SQL database in both availability zones in Virginia, and the copy in Oregon, and the backups, we would still know where all of our preservation files are. We can locate and retrieve all AIPs and reconstruct our registry (except for PREMIS events) based on the metadata attached to those files.

This is an example of the metadata attached to each of the files stored in our Virginia/S3 and Oregon/Glacier preservation storage areas.

The image to the right shows an example of the metadata attached to each file.

DPN

Software Development Styleguide

The Software Development Styleguide documents practices and policies that APTrust engineering staff adheres to.

Support and Maintenance

Operations procedures and detailed technical documentation.

Partner Tools

APTrust Partner Tools can help you validate bags and manage your AWS buckets. Use the links below download the tools.

Version 2.2-beta

This version fixes some minor bugs and standardizes exit codes across all of the tools. It also replaces the underlying S3 library used for uploads and downloads with the official Amazon S3 library. The apt_upload program includes some breaking changes from the last release. Read the README.txt file included in the package.

OSX (64-bit) (v 2.2-beta - 12/04/2017)

Linux (64-bit) (v 2.2-beta - 12/04/2017)

Windows (64-bit) (v 2.2-beta - 12/04/2017)

Version 2.1

The version 2.1 tools are identical to the version 1.03 tools, except for the following additions:

  • apt_check_ingest, which can tell you the ingest status of a bag from the command line. (Added Nov. 17, 2017)
    • 2017-11-28: Added -etag option, added etag to text output, and improved readability of text output.
  • apt_validate, which can now validate both APTrust and DPN bags. (Added Feb. 17, 2017)
    • 2017-11-28: Fixed EXCHANGE_HOME/GOPATH error message that prevented the tool from running.
    • 2017-11-28: Fixed validation of untarred path on Windows.

The 2.1 packages include json configuration files that tell the validator how to validate APTrust and DPN bags.

You won't need to create or validate DPN bags. APTrust does that for you when you push items through us to DPN. The DPN validation config is included as an example of how to write a configuration file to validate bags whose requirements differ from the standard APTrust requirements. Although the configuration does not cover every possible BagIt option, it is a solid first step toward a fast, stand-alone, configurable bag validation tool. It runs on Mac, Linux, and Windows, has no installation dependencies, and is known to have good performance on large bags.

For help with the new validator, simply run the apt_validate command with no options or parameters.

OSX (64-bit) (v 2.1 - 11/28/2017)

Linux (64-bit) (v 2.1 - 11/28/2017)

Windows (64-bit) (v 2.1 -11/28/2017)

Version 1.03

OSX (64-bit) (v 1.03 - 12/14/2015)

Linux (64-bit) (v 1.03 - 12/14/2015)

Windows (64-bit) (v 1.03 - 12/14/2015)

Each archive contains the five executable files listed below. There is no installer. Simply put the executables where you want them and run them from the command line.

Tool Description
apt_check_ingest Command-line tool to check the ingest status of a bag.
apt_validate Validate bags (tarred or untarred) before uploading them for ingest (Version 1.03 has a bug that marks some bags with invalid tag manifests as valid. This validator also does not identify some invalid file names. Version 2.0 fixes these bugs.)
apt_upload Upload bags to your receiving buckets for ingest
apt_list List the contents of your receiving and restoration buckets
apt_download Download restored bags from your restoration buckets
apt_delete Delete restored bags from your restoration buckets

Configuration

All of the tools except apt_validate require a simple config file with five name-value pairs. The config file format and requirements are the same for 1.03 and 2.0 tools. Note that quotes are optional, and comment lines begin with a hash mark.

# Config for apt_upload and apt_download
AwsAccessKeyId = 123456789XYZ
AwsSecretAccessKey = THIS KEY INCLUDES SPACES AND DOES NOT NEED QUOTES
ReceivingBucket = 'aptrust.receiving.universityname.edu'
RestorationBucket = "aptrust.restore.universityname.edu"
DownloadDir = "/home/josie/downloads"
AptrustApiUser = "archivist@example.edu"
AptrustApiKey = "f887afc5e1624eda92ae1a5aecdf210c"

If you prefer not to put your AWS keys in the config file, you can put them into environment variables called AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

ReceivingBucket: S3 bucket that will hold your uploaded APTrust bags that are awaiting ingest.

RestorationBucket: S3 bucket that will hold your restored APTrust bags.

DownloadDir: The local directory in which to save files downloaded from your APTrust restoration bucket. The APTrust config currently does not expand ~ to your home directory, so use an absolute path to be safe.

AptrustApiUser: The email address for logging in to APTrust's PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. REST APIApplication Programming Interface.

AptrustApiKey: Your API keyAPI key is a code passed in by computer programs calling an application programming interface (API) to identify the calling program for the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. REST APIApplication Programming Interface. This key must match the user email. (That is, joe@example.com cannot log in with a key that was issued to beth@example.com.)

If you save your config file as ~/.aptrust_partner.conf in Linux/Mac or as %HOMEPATH%\.aptrust_partner.conf under Windows, you will not have to specify a --config option when you run the tools. Otherwise, run the tools with the --config file pointing to the full path of your configuration file.

Help

You can view any program's built-in documentation by passing the --help flag.

If you run into problems, send a message to help [at] aptrust.org.

Reporting

APTrust reporting comes in a variety of different manners, including standard reports, alerts, and various email notifications.

Reports

At present, there is one major report available for viewing or download as a pdf. The report provides a breakdown of the content of a single institution. It includes:

  • number of ingested intellectual objects,
  • number ingested generic files,
  • total number of premis events generated for the institution's content,
  • total number of work items generated for the institution's content,
  • total number of bytes preserved,
  • average file size for that institution,
  • a breakdown of the amount of content ingested by file type.

The report can be found at https://repo.aptrust.org/reports/overview/:instititon_identifier on the production site and at https://demo.aptrust.org/reports/overview/:institution_identifier on the demo site, under the 'Reports' tab of the top navigation.

Alerts

There are a variety of alerts available through the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. web application for institutional administrators to view. The types of alerts available for viewing are:

  • Failed fixity checks
  • Failed ingests
  • Failed restorations
  • Failed deletions
  • Failed DPN ingests
  • Stalled DPN replications
  • Stalled work items

The alerts can be found at https://repo.aptrust.org/alerts on the production site and at https://demo.aptrust.org/alerts on the demo site, under the 'Alerts' tab of the top navigation.

Email Notifications

As of September 2017 email notifications that inform institutional administrators of any failed fixity checks or successful intellectual object restorations as they happen. The emails will come once in a day for a batch of failed fixity checks or successful restorations, rather than one email per event so as not to clog email inboxes.

Two Factor Deletion

Deletion of an intellectual object triggers an email sent to institutional administrators at the parent institution in order to confirm the deletion request.

DPN

APTrust is a node in the Digital Preservation Network APTrust is a node in the Digital Preservation Network (DPN)