GREI 3: HDV Task - Improve OAI-PMH Harvesting

[cmbz]

Overview

"Our proposed project will significantly improve the widely-used Harvard Dataverse repository to better support NIH-funded research. A critical measure of the GREI program’s success is to standardize the discoverability across generalist repositories.

To help with this, we propose to improve the existing harvesting functionality in the Dataverse software based on the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) standard, and coordinate with other repository packaging standards to share or move metadata and data. Dataverse already supports the Bags as defined by the Research Data Alliance (RDA) Research Data Repository Interoperability Working Group.

Here we proposed to improve the support for Bags, test it for NIH-funded datasets, and explore and define the appropriate standard to use to move the metadata and data across generalist repositories This will help with a sustainable and succession plan - if one repository cannot support anymore a specific dataset, it will allow to easily move the dataset to another repository without losing any information about the dataset." (Source)

Issues

New Features

Support harvesting from DataCite's OAI-PMH API

• Scope, define, and size work needed to extend the Dataverse harvesting client to support harvesting by identifier or 'locally defined set' #330
• Add support for OAI-harvesting from DataCite dataverse#10909

Spikes

• Collection: Keep track of list of issues that we want to address as part of 1.4.1 #25
• NIH AIM:4 YR:2 TASK:1B | 2.4.1B | (started yr1) Resolve OAI-PMH harvesting issues #85
• Spike: Review if inability to harvest all records from certain repositories will be or has been resolved when other harvesting-related issues are addressed dataverse.harvard.edu#92
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=287243 (SRDA)
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=295542 (CIFOR)
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=322153 (Borealis)
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=329544 (DataverseNO)
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=330637 (Roper Center)
• Policy Question: Do we have an End-of-Life, End-of-Service policy for Harvesting feature? #100

Harvesting Issues (Year 3)

In Progress

• Harvest metadata from SRDA dataverse.harvard.edu#243 - essentially done (we have harvested their metadata and their datasets are searchable by our users). we are keeping the issue open because we asked the devs at SRDA to improve something on their end, so this is for us to remember to check on the progress.
• Re-harvesting datasets from Roper Center dataverse.harvard.edu#204 - this is pending on Roper devs. fixing something on their end. We have reestablished contact and communicated what the issues are.
  • https://help.hmdc.harvard.edu/Ticket/Display.html?id=330637
• Re-harvest from Borealis Repository dataverse.harvard.edu#172 - ongoing effort
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=322153

Complete

• Really partial references when harvesting dataverse#8632
• For harvesting, allow advanced feature of using listRecords vs ListIdentifiers dataverse#10936
• Allow Harvesting to use arbitrary sets dataverse#10937
• Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse#8129
• Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse#8578
• Harvested dataset display is broken on develop branch dataverse#10611
• Refactor reindexing of harvested datasets  dataverse#10734
• Feature Request/Idea: Sanitize languages controlled vocabulary values dataverse#8243 - complete
• Revisit/reimplement the concept of a "Harvested file".  dataverse#8629
• Harvest: Do not list a set as available in ListSets until it has been successfully exported at least once. dataverse#3322
• Harvesting: OAI sets are not updated when datasets are deleted. dataverse#8005
• Purge all the stale Nesstar harvested dataverses in production dataverse.harvard.edu#153
• Unable to create harvesting client for SRDA repository dataverse#7624
• Unable to remove datasets that Harvard Dataverse Repository harvested from Swedish National Data Service  dataverse.harvard.edu#142
• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. dataverse#3437
• 3437 OAI records for datasets that still exist, but are temporarily un-indexed should not be marked deleted dataverse#10222
• Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values dataverse#9992
• XML declaration missing from OAI when using oai_ddi dataverse#10329
• Use Abacus Data Network's new server URL to harvest the installation's metadata dataverse.harvard.edu#251
• Add harvesting feature to handle missing controlled values dataverse#10323 - almost done (currently in QA)
• Fix redirects to the original repository for "Generic OAI Archive"-type Harvesting Clients dataverse#10254 - for the most part, the fix already exists (added to the prod. 6.1 patch - that's what made it possible to harvest from SRDA, next item on the list)
• Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse#8578
• Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse#8129
• Are Dataverse installations' harvesting sets exporting regularly? dataverse#5392
• Dataverse should return machine-readable metadata to requesting clients/servers (content negotiation) dataverse#3699
• Feature Request/Idea: Standardize standard license configuration dataverse#8512
• Harvest: Exporting all IQSS datasets has several failures. dataverse#3276
• Harvest: Harvest all from IQSS in JSON format had 68 failures out of 19726 dataverse#3298
• Harvest: Harvesting appears to just die silently intermittently, using Princeton set in prod dataverse#7074
• Harvest: OAI client should follow http->https redirect since oai server validation allows http. dataverse#4831
• Harvesting - Data Citation Values Missing dataverse#3290
• Harvesting run infinitely stays at IN PROGRESS and DELETE IN PROGRESS dataverse#7052
• Harvesting: Manage Sets page goes deaf after a while, making create set popup not save, does nothing until refresh page. dataverse#3316
• Harvesting: Manage sets, entering a longish string into set name throws exception, throws away set. dataverse#3314
• Harvesting: Manage Sets, sets defined with dataverse scope become less valid when subdv's are added. dataverse#3315
• Should the Geographic Bounding Box allow optional coordinates and multiple boxes in the XML? dataverse#7108
• Some elements in the DDI XML response have an incorrect sequence dataverse#7127
• Spike: Investigate how Dataverse stakeholders and users need to collect and use funder metadata dataverse#4859
• Support MARC-XML for Export and Harvest dataverse#2721
• Add support for exporting more metadata formats via XSLT crosswalks dataverse#5960

Pending

These issues will be sized and prioritized once the In Progress issues are closed.

• Add import for oai_datacite ("OpenAire") format (this will allow Dataverse to harvest this format too) dataverse#7727
• Refactor the OAI code, Step 2 dataverse#8842
• Dataverse should use ISO codes for languages when exporting and importing metadata. dataverse#7243
• Consider making export time stamps more granular dataverse#8630
• Consider reorganizing metadata import code dataverse#8631
• Re-harvesting ICPSR datasets dataverse.harvard.edu#63
• Investigate why Harvard Dataverse is unable to harvest from International Potato Center and ICRISAT Research Data Repository dataverse.harvard.edu#259

Additional Harvesting Issues (Year 4)

These issues will be prioritized during GREI Year 4 Planning.

• Access Rights metadata in OpenAIRE metadata export is being misapplied dataverse#5920
• Align or merge DataCite metadata exports dataverse#5889
• As a data repository, I need to harvest additional metadata in OAI_DC records dataverse#4176
• As a Dataverse collection admin, I want to use the UI to link harvested datasets into my collection dataverse#4025
• As a Dataverse installation administrator, I need clear indications of status of harvesting attempts so that I know which ones I need to troubleshoot dataverse#9294
• Feature request: create a harvested dataset via the API dataverse#7330
• Feature Request: enable static OAI-PMH sets for the collection tree dataverse#9344
• Feature Request/Idea: Add an image to harvested dataset display dataverse#9438
• Feature Request/Idea: Data Access Place add URI to support linkages dataverse#9079
• Harvester exporter uses short version of XML dataverse#8778
• Harvesting : message "javax.ejb.EJBTransactionRolledbackException, Exception thrown from bean: java.lang.NullPointerException" dataverse#9318
• Harvesting : message "javax.ejb.EJBTransactionRolledbackException, Exception thrown from bean: java.lang.NullPointerException" dataverse#7546
• Harvesting: Fails OAI validation on several tests. dataverse#8815
• Harvesting: Search datasets from specific sources dataverse#10195
• Migrate OAI-PMH options to JvmSettings and make log path configurable dataverse#10042
• Move harvestingclient_id from the dataset to dvobject and use it directly for files dataverse#9686
• oai_dc export fails, OAI-PMH error dataverse#8306
• PM.Epic: (Potentially) Modify the design of the harvesting Framework and/or metadata exports dataverse#8703
• Problems with Harvested Records dataverse#9261
• Publisher tag in oai_dc format dataverse#8739
• Saved Search inconsistencies around unpublished datasets dataverse#7453
• Support for Importing and Exporting DCAT Metadata dataverse#1592
• Usage of datestamp from Record during harvest using oai_dc format dataverse#8693
• Special characters in dataset metadata prevent Dataverse repositories from harvesting from Harvard Dataverse dataverse.harvard.edu#72
• Schedules returned by the "List All Configured Harvesting Clients" endpoint are a day off dataverse#10396

Related

• Investigate Solr performance issues (again) dataverse#10469 (Solr indexing maybe affecting performance)

Resources

• Project: NIH GREI - Generalist Repository Ecosystem Initiative #124
• Epic: GREI 3 - Search and Browse #117
• List of repositories to harvest: here
• NIH OTA
• Google Drive Folder
• Monthly Status Reports
• APPROVED - NIH GREI Coopetition Plan Year 3
• DRAFT: Year 3 Realignment Proposal Plan
• Spreadsheet with info about Dataverse repositories that Harvard Dataverse is harvesting metadata from

[cmbz]

Status: January 2024

Several issues relating to harvesting other repositories' metadata were resolved for Harvard Dataverse. A number of Dataverse harvesting-related issues and bugs were closed, see list:

Completed

• Purge all the stale Nesstar harvested dataverses in production dataverse.harvard.edu#153
• Unable to create harvesting client for SRDA repository dataverse#7624
• Unable to remove datasets that Harvard Dataverse Repository harvested from Swedish National Data Service  dataverse.harvard.edu#142
• Harvesting: OAI sets are not updated when datasets are deleted. dataverse#8005
• OAI-PMH error handling: meaningful errors in XML for verb processing dataverse#10205
• 3322 Empty OAI sets should not be listed in ListSets  dataverse#10228
• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. dataverse#3437
• 3437 OAI records for datasets that still exist, but are temporarily un-indexed should not be marked deleted dataverse#10222

In progress

• Re-harvesting datasets from Roper Center dataverse.harvard.edu#204
• Feature Request/Idea: Sanitize languages controlled vocabulary values dataverse#8243
• Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values dataverse#9992

[cmbz]

Status: February 2024

The following OAI-PMH harvesting improvements shown below were made this month.

Completed

• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. dataverse#3437
• 3437 OAI records for datasets that still exist, but are temporarily un-indexed should not be marked deleted dataverse#10222

[cmbz]

Status: March 2024

Meetings

A meeting was held on 2024/03/13 to:

• Discuss current GREI harvesting issues defined here: GREI 3: HDV Task - Improve OAI-PMH Harvesting #171.
• Discuss larger issues and spikes requiring substantive design and refactoring work and coordination
• Define a plan of action for GREI Year 3 (and ideally, Year 4) to complete the known remaining harvesting issues and any new, in-scope issues that may arise.

Action Items

The following action items were identified:

• Create an ordered list of non-Dataverse repositories that need harvesting @jggautier
• Add additional issue/RT Tickets related to harvesting to the tracking issue @jggautier
• Create list of all Dataverse installations we should be harvesting from and identify best metadata format for each @jggautier
• Re-create harvesting clients using most appropriate metadata format in order to harvest as much metadata as possible from each client, following draft policy @jggautier
• Modify policy issue to reflect discussion: Policy Question: Do we have an End-of-Life, End-of-Service policy for Harvesting feature? #100 @scolapasta
• Create "repository harvest profile" for each non-Dataverse repository that indicates the type of harvesting-appropriate metadata exports it supports. Use these profiles to inform the harvesting process for each repository. @jggautier
• Order the existing Pending issues list in order of priority @landreev
• Identify the current, remaining harvesting issues in the dataverse repo and add them to the tracking issue in a new section @cmbz

Other Work Towards Improving Harvesting

• @jggautier updated the GitHub issue at Align or merge DataCite metadata exports dataverse#5889 with a proposal for sending more metadata to DataCite

Completed

• Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values dataverse#9992
• XML declaration missing from OAI when using oai_ddi dataverse#10329
• Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values dataverse#9992
• Use Abacus Data Network's new server URL to harvest the installation's metadata dataverse.harvard.edu#251
• Add harvesting feature to handle missing controlled values dataverse#10323
• Fix redirects to the original repository for "Generic OAI Archive"-type Harvesting Clients dataverse#10254

[cmbz]

Status: April 2024

• @jggautier continued the March 2024 action item of re-creating harvesting clients using the most appropriate metadata format in order to harvest as much metadata as possible from each client. But he stopped due to technical issues being investigated, described at Investigate Solr performance issues (again) dataverse#10469, that are affecting harvesting.

[cmbz]

Status: June 2024

• @jggautier emailed the contacts of all repositories we've been emailing about Harvard Dataverse harvesting from their repositories. In most cases he let them know that we continue to work on challenges that prevent Harvard Dataverse from harvesting from these repositories, and in a couple of cases, such as what's described in #243, he asked about the status of related work that they are doing so that Harvard Dataverse is able to harvest from them.

[jggautier]

@landreev, @cmbz and @scolapasta, today or sometime this week I'm thinking of following up with contacts of the repositories who've emailed our support email addresses about harvesting, to let them know that we're continuing to work on improvements related to how Dataverse indexes records, and some of those improvements may affect harvesting. Most of the emails sitting in my RT queue are harvesting related:

[cmbz]

@jggautier Great thanks! When you do, please update the June status comment, too #171 (comment)

[cmbz]

Status: July 2024

• Closed: Spike: Investigate how Dataverse stakeholders and users need to collect and use funder metadata dataverse#4859

[cmbz]

Status: August 2024

Updates

• Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse #8578
  • We continued discussion and development work for extending the list of languages.
• Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse #8129 and As a data repository, I need to harvest additional metadata in OAI_DC records dataverse #4176
  • We continued discussion for improving the metadata included in Dublin Core metadata available when harvesting using OAI-PMH
• Support for Importing and Exporting DCAT Metadata dataverse #1592
  • We continued soliciting feedback from the Dataverse community about importing and exporting metadata using the DCAT metadata standard
• Support MARC-XML for Export and Harvest dataverse #2721
  • We continued soliciting feedback from the Dataverse community about importing and exporting metadata using the MARC-XML metadata standard

Harvesting Issues

• Currently in QA: Support the full ISO 639-3 list of languages #10578 dataverse#10762
• Currently in Review: Remap oai_dc fields dc:type and dc:date dataverse#10737
• Currently in This Sprint: Open DP: Devops Transition, 2024-08-28 to 2024-09-11 #318

[cmbz]

Status: September 2024

• Related PR merged to remap oai_dc fields dc:type and dc:date
  • Remap oai_dc fields dc:type and dc:date dataverse#10737
• Other issues include:
  • 10734 Refactoring and optimization of importing and reindexing of harvested content  dataverse#10836
  • Feature Request/Idea: Standardize standard license configuration dataverse#8512
  • Support the full ISO 639-3 list of languages #10578 dataverse#10762

[cmbz]

Status: October 2024

• The team began work to support harvesting from DataCite's OAI-PMH API. This functionality offers a straightforward way to use DataCite's OAI-PMH API to harvest metadata from datasets published by repositories that do not offer their own OAI-PMH feeds
  • Scope, define, and size work needed to extend the Dataverse harvesting client to support harvesting by identifier or 'locally defined set' #330
  • Add support for OAI-harvesting from DataCite dataverse#10909
• Several projects will immediately benefit from this new functionality, including GREI, CAFE, and the Bertarelli Foundation collections.

[cmbz]

Status: November 2024

The following harvesting-related PRs were merged:

• A fix for the harvesting regression/tests introduced in 10836 dataverse#10990
• Exclude others namespace from harvesting "oai_dc" metadata prefix dataverse#10837
• Fix lookups of Harvested datasets with lower-case versions of persistent identifiers in the database dataverse#11003
• Harvesting : Fix DDI Import for otherId dataverse#10772
• Update dev-usage.rst - Add Details for Accessing and Saving Harvestin… dataverse#10996
• A fix for the harvesting regression/tests introduced in 10836 dataverse#10990

Supporting development work continued to support harvesting from DataCite's OAI-PMH API. See:

• Add support for OAI-harvesting from DataCite dataverse#10909
• https://github.com/IQSS/dataverse-HDV-Curation/issues/582#issuecomment-2447340986