5.1.1.3 Effective mechanisms to detect bit corruption or loss

From aptrust
Jump to: navigation, search


5.1.1.3 Effective mechanisms to detect bit corruption or loss
Status Ready for review
Compliance Rating Fully compliant
Responsible

The repository shall have effective mechanisms to detect bit corruption or loss.

Supporting Text

This is necessary in order to ensure that AIPs and metadata are uncorrupted or any data losses are detected and fall within the tolerances established by repository policy (see 3.3.5).

Examples for Meeting the Requirement

Documents that specify bit error detection and correction mechanisms used; risk analysis; error reports; threat analysis; periodic analysis of the integrity of repository holdings.

Discussion

The objective is a comprehensive treatment of the sources of data loss and their real-world complexity. Any data or metadata that is (temporarily) lost should be recoverable from backups. Routine systematic failures must not be allowed to accumulate and cause data loss beyond the tolerances established by the repository policies. Mechanisms such as checksums (MD5 signatures) or digital signatures should be recognized for their effectiveness in detecting bit loss and incorporated into the overall approach of the repository for validating integrity.

Evidence Provided

The APTrust system retrieves files for fixity checks on a 90-day basis to ensure data is accurate and complete. At the time of deposit/ingest of new bags our system generates a MD5 and SHA256 hash of each individual file in the bag, compares it with the bag manifest, and stores it in the metadata if it matches the manifest. This ensures that the bag was received correctly and the files are in the exact same state as they were prior to submission and transfer. If a calculated hash doesn't match the one in the supplied bag manifest, the bag is not ingested and marked as failed. Depositors will need to submit a corrected bag per the Bagging Specifications.

Alerts are displayed on through the PharosPharos is APTrusts web interface to manage deposits and inspect deposit outcomesETag The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. web interface to institutional administrators. The repository also notifies depositors about failed fixity checks and successful restorations through Email Notifications.

Information on restoring content can be found in the Restoration section.

See the page on Risk Management, Threats, and Mitigations for information on the identifying, preventing, and mitigating risks.