Mismatch between export and import DDI functions

[lubitchv]

We tried to export DDI xml metadata from one instance of dataverse and import that metadata into another. We noticed that not all the fields there imported, for example Related Publications (relPubl). The reason seems to be in mismatch between export and import DDI functions. Some fields are missing in export and not exported and hence although import function has them, they cannot be imported.
It seems also that export is not always DDI complaint and has fields in wrong sections. There is also misspelling, for example, in export there is estSmpErr field but in import EstSmpErr.
Another issue is more problematic. Whole sections of metadata are not exported, such as "Astronomy and Astrophysics", "Life Sciences" and "Journal" metadata. I do not know if it is possible to fix since it does not look like that DDI standard has corresponding fields that can be used.

[djbrooke]

Thanks @lubitchv for the report.

@jggautier do you feel the first part of this is a duplicate of #3297? And for the custom metadata blocks, we'd only expect those to be exported as part of the JSON export, correct?

[lubitchv]

@djbrooke I started to work on this issue since we need to export and import metadata. Json export/ import seems to import with files and we need only metadata. Also json import seems not to work with published existing doi that needed to stayed unpublished (datasets) after import. But it is already another issue.
I hope to submit pull request soon. It will not solve all the problems, just part of it since some of them are impossible to solve, since the some fields are not DDI compliant.

[djbrooke]

That's great @lubitchv, thanks for your continued work on Dataverse! It was really neat seeing the Data Curation Tool in action last week. I'll assign this to you for now, and we'll be happy to provide any feedback or answer any questions as you work towards a PR.

[jggautier]

do you feel the first part of this is a duplicate of #3297? And for the custom metadata blocks, we'd only expect those to be exported as part of the JSON export, correct?

I'm not sure. Since #3297 is about indexing fields from OAI-PMH harvested DDI metadata, is the code used when harvesting DDI metadata the same code used by the API endpoint for importing DDI metadata? If it's not the same code, are both methods for importing DDI metadata (with the API endpoint and over OAI-PMH) mapping DDI elements to Dataverse fields in the same way?

And for the custom metadata blocks, we'd only expect those to be exported as part of the JSON export, correct?

I think so. I wouldn't expect to be able to map fields from the "Astronomy and Astrophysics", "Life Sciences" and "Journal" metadatablocks to DDI Codebook elements.

[landreev]

My reading of #3297 is that it is indeed a duplicate of this issue. The core of it, as reported back then was the same thing: mismatch between what we export, and how we import.
I would not describe #3297 as "about indexing fields from ... harvested DDI metadata"; harvesting and indexing are mentioned in that issue, but only as symptoms of the main problem (or at least that's my reading of it).

[landreev]

The question from the original description of the issue, about exporting metadata from other custom blocks as DDI - just to confirm what others have said above, DDI is a narrowly specialized format, purposefully designed for quantitative social sciences. There shouldn't be any assumption that we could possibly use it for all of our custom metadata.
There are cases where it is possible to map some metadata fields designed for other disciplines into DDI fields. But they should be considered exceptions.
The scope of this issue is to standardize the rules used in the DDI export and import; so that an export-import round trip does not result in a loss, if at all possible.

[jggautier]

Hi @lubitchv. Since the PR #6669 is merged and shipped in Dataverse 4.20, can this issue be closed? Are there any details that should be added to new or existing issues?