Remap oai_dc fields dc:type and dc:date #10737
Conversation
The `oai_dc` export and harvesting format has had the following fields remapped: - dc:type was mapped to the field "Kind of Data". Now it is hard-coded to the word "Dataset". - dc:date was mapped to the field "Production Date" when available and otherwise to "Publication Date". Now it is mapped only to the field "Publication Date". - dc:rights was not mapped to anything. Now it is mapped (when available) to terms of use, restrictions, and license.
pdurbin
added
Feature: Harvesting
Size: 10
|
@jggautier heads up that this relates to this issue in that we are now adding "dc:rights" to the |
|
@tcoupin I'm requesting a review from you because you modified the dc:date login in the following pull request and I changed it (as explained above): |
|
Re: dc:date - should it be mapped to the same field as https://guides.dataverse.org/en/latest/api/native-api.html#set-citation-date-field-type-for-a-dataset ? That is publicationDate by default. |
This comment has been minimized.
This comment has been minimized.
|
@qqmyers well, Publication Date is what @philippconzett asked for in the issue (#8129). |
|
@philippconzett 's notes also point out that this date potentially going to be interpreted as the citation date. Since we allow configuring that in the local installation, it seems like it could be confusing to hardcode it for harvesting. If the harvester used the field from that setting, citations would be consistent in the local display and harvesting sites, and it would default to publicationDate as requested in the issue. |
|
I don't have a strong opinion about it. |
|
I think @qqmyers's suggestion for dc:date makes sense. |
|
I've taken @tcoupin's role on Dataverse issues, so I am looking at this for him. Part of the context for the change he implemented (mapping dc:date to Publication Date if Production Date is empty) was that when Dataverse harvests another OAI-PMH repo, dc:date is mapped to productionDate and this production date is then used in the citation of the harvested dataset. #8733 and #8732 were both part of an effort to guarantee the coherence between citation dates when harvesting another Dataverse. So I agree with @qqmyers's suggestion on Hardcoding There might be an alternative solution where there is always at least a |
This comment has been minimized.
This comment has been minimized.
|
@plecor thanks. One thing to consider with "dc:type" is that types other than datasets (like software and workflows) are coming... ... so maybe we can revisit "dc:type" once that pull request is merged. To all, I pushed some tests to exercise export and setting the citation date. Now I'm trying to see if there's a small change I can make to DublinCoreExportUtil to get the citation date out. I can get just the year (YYYY) with code like this... String citation = version.getCitation();
// We're looking for ", YYYY, " in a citation like this:
// Finch, Fiona, 1999, "Darwin's Finches", https://doi.org/10.5072/FK2/WSSYBE, Root, V1
Pattern pattern = Pattern.compile(", (\\d{4}), ");
Matcher matcher = pattern.matcher(citation);
matcher.find();
String yearInCitation = matcher.group(1);
writeFullElement(xmlw, dcFlavor+":"+"date", yearInCitation);... but I need the full YYYY-MM-DD version to put in the the |
|
I dug a little more and our citation code is focused on returning just a 4 digit year for the date. This would be a change from what we do now (YYYY-MM-DD) @philippconzett @plecor @qqmyers what do you think? Should we change The spec Philipp found seems to say it's ok. Check out the year 1650 as an example at https://www.base-search.net/about/en/faq_oai.php#dc-date
|
|
Since the citationDateFieldType is part of the Dataset, I'd think at some point it could/should be part of the DatasetDTO and JSON export, thereby being available to other exporters (will the SPA or other client need this info (in the JSON returned from the dataset api) at some point?). If that's too much for now, I think the idea of parsing it from the citation as YYYY makes sense, assuming that's sufficient for how people want to use that field. Alternately, I think you could 'go around' the exporter SPI interface and get the full value directly pretty easily as well, e.g. with something like: This would not be the only current exporter doing that (e.g. the DDI exporter grabs the ExportInstallationAsDistributorOnlyWhenNotSet Setting it needs). |
@qqmyers I gave this a try but |
|
Are you thinking of ? citationDataType.getName() should get the name of the field type and then dto2Primitive should get the full value from the field itself in the metadatablock, which I think should always be the full date (unless there's a field that is YYYY only?). |
|
Ah, @philippconzett, just to clarify, in my screenshot where I pointed out Here's a screenshot of that panel and that Restrictions field noted in green: |
|
@jggautier Thanks for clarifying. Is there a way to figure out what information we deliver to harvesters like BASE Bielefeld? See the initial post in #8129. |
|
@philippconzett, do you mean if there's a way to figure out what we currently deliver to harvesters like BASE Bielefeld or what we should deliver? Sorry if that sounds like a dumb question lol. I've been focused a lot on how decisions are made as well as what decisions are made, which maybe led to me overthink your question! If it's currently, you wrote in #8129 about what we currently deliver when it comes to rights metadata, which is that we don't deliver anything since Dataverse doesn't provide the field dc:rights. I can say that nothing's changed about this since you opened #8129 a few years ago. So when I asked if it's appropriate that this PR being merged will close #8129, and that there's discussion in that GitHub issue that isn't addressed, I was mostly thinking about your comments related to dc:rights and degrees of open access. For what we should start delivering to harvesters like BASE Bielefeld, you mentioned the guidance at https://www.base-search.net/about/en/faq_oai.php#dc-rights, which recommends the two vocabularies you pointed to earlier today: the info-eu-repo-Access-Rights vocabulary and the COAR-Access-Rights vocabulary. I think it'll be helpful to consider what I wrote in #5920, where I wrote about what we've learned and challenges about how the info-eu-repo-Access-Rights vocabulary is being included in the OpenAIRE exports that Dataverse creates. This might be a matter of scoping and timing, too, right? We could create a new GitHub issue specifically about the use of the info-eu-repo-Access-Rights and COAR-Access-Rights vocabularies, that mentions what's discussed in #8129. So when #8129 is closed because this PR is merged, the unaddressed goals you mentioned in #8129 aren't lost and there's a place where the community can focus on how to address those goals. And if we can learn how @pdurbin and others made the decisions in this PR about what goes into dc:rights, it'll be easier to think about how effective those decisions are. |
|
Hi @jggautier! Thanks for disentangling this! I wasn't aware of #5920, but have read up about it now and added some comments there. I don't really know what this means for #8129. Maybe a temporary solution to make dataset metadata from Dataverse more visible in OpenAIRE could be a kind of "inverted" and slightly adapted version of what you described in #5920:
|
This comment has been minimized.
This comment has been minimized.
@jggautier back in 59850ce these lines were added as part of PR #3308: This was for Dataverse 4.5 ( https://github.com/IQSS/dataverse/releases/tag/v4.5 ) when harvesting was first introduced (in 4.x). It looks like the code was mostly worked on by @landreev and @sekmiller. In short, the logic has been here for 8 years and I can't find any comment on why we do it this way. @jggautier what you're showing in that screenshot is a reflection of the same, unchanged logic. The code above is for the "dcterms" flavor of Dublin Core. In this current PR, I copied the logic above for the "dc" flavor. I hope this helps! |
|
Ah, thanks @pdurbin! I updated the Dataverse crosswalk to reflect this, specifically the part of the crosswalk showing that what's entered in the Restrictions field is included in the DC Terms export, as dcterms:rights like you wrote. The crosswalk used to indicate this, and for some reason I don't remember now, in 2022 I edited it to read that it was "(Not mapped)". To learn more about why the predefined license, Terms of Use, and Restrictions are included in the DC Terms export, I tried to find the Functional Requirements Document mentioned in that PR you linked to. The Functional Requirements Document folder in our Google Drive has some info, but I haven't seen any FRDs so far that go into enough detail. I think we should:
|
pdurbin
changed the title
|
As discussed in Slack and elsewhere:
@landreev I'm going to unassign myself but please let me know if you'd like me to jump back on this branch and do any additional coding or testing! |
|
📦 Pushed preview images as 🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name. |
|
@pdurbin |
|
@landreev sure, I added two XML examples, before and after, to the description of this PR. |
There was a problem hiding this comment.
I have confirmed that the change does not affect harvesting. In normal practice, this should not even be a concern, as two Dataverses should never use the oai_dc as the format for harvesting from each other. But it's not entirely impossible that it will be a practical use case for somebody. Plus it would simply feel wrong, for Dataverse not to be able to import its own metadata exports. So, happy to report that 2 Dataverses can still harvest from each other using the format.
What this PR does / why we need it:
The
oai_dcexport and harvesting format has had the following fields remapped:dc:rights was not mapped to anything. Now it is mapped (when available) to terms of use, restrictions, and license.Deferred until this issue:As these are backward incompatible changes, they have been emphasized in the release note snippet.
Which issue(s) this PR closes:
Special notes for your reviewer:
Should these backward-incompatible changes be hidden behind a feature flag?
Suggestions on how to test this:
See rules above under "what this PR does". Also, below are some examples of before and after.
Before
dc:dateis mapped to the field "Production Date" when available and otherwise to "Publication Date".dc:typebecause it was entered in the "Kind of Data" field.dc:typewill be absent if "Kind of Data" isn't filled in.After
dc:dateis mapped the field "Publication Date" or the field used for the citation date, if set (see Set Citation Date Field Type for a Dataset).dc:typeis hard-coded to "Dataset".Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No.
Is there a release notes update needed for this change?:
Yes, included.
Additional documentation:
None.