Re-ingestion attempts are happening too quickly flooding the logs

[lagoan]

The new functionality that re-adds items that failed to be ingested floods the logs. This happens because the retries happen right after the failed attempt with only a 10-second delay.

We need to add a way to delay the attempts from PMPY to reingest the entities. We talked about doubling the time between retries. With a max wait time of a week.

[jefferya]

I'm not sure where to add this comment. I think the approach good. In case it helps, I'll share what was done with the CWRC preservation workflow. Adding delays helped with the CWRC preservation however failures still happened. A choice was made to assume failures would happen and development went toward both an audit tool to detect failures and report item preservation status plus a mechanism to preserve individual failed items.

[lagoan]

Thanks for sharing this @jefferya. Yes, we need to assume failures will continue to happen and we do need some sort of audit tool. Would you be able to provide design documentation for the approach used in CWRC?

[jefferya]

Would you be able to provide design documentation for the approach used in CWRC?

The design of the current approach is at the following link: https://github.com/ualbertalib/cwrc_preservation#reporting--auditing-cwrc_audit_reportrb. The approach is simplistic. The audit report lists all IDs in CWRC and Swift with an indication whether preservation was successful or absent (either due to a failure or due to the CWRC object being updated after the last preservation run) plus a quick check if the Swift object size makes sense.

Here is a summary:

• a custom metadate field is added to the swift object at ingest time (not into the package but added to the swift object metadata) to indicate CWRC resource version
• Audit process:
  • API call to get a list of CWRC resources and their associated version
  • API call to get a list of Swift objects with the resource version from the Swift object metadata
  • compare the two lists aligning by ID and version (not very efficient because quickly done)
  • output a CSV report indicating
    • if matching ID and version then preservation was successful and up-to-date
    • if matching ID but outdated Swift version then either
      • a preservation error
      • or a newly added/updated CWRC object since last preservation run
    • if only in CWRC then either
      • a preservation error
      • or a newly added CWRC object since the last preservation run
    • if only in Swift then this is CWRC deleted object (policy is to retain deleted within preservation, at present)
    • quick check of size (i.e., if size is below a minimum then likely the preservation package is an HTTP error as opposed to real content
    • the CSV: "#{cwrc_pid},#{cwrc_version},#{swift_id},#{swift_cwrc_version},#{swift_bytes},#{status}"

There are assumptions and trade-offs in this approach that might be solvable in the new CWRC version (e.g., db row created with checksums of the files within the archival package) but I'm not sure how an audit report might look like in the context of OLRC.

Note: this details of the approach are

• likely not compatible with the upcoming version of CWRC.
• may not be compatible with OLRC