Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make Alternative Title repeatable #9440

Merged
merged 26 commits into from
Sep 22, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f2946cb
Merge pull request #64 from IQSS/develop
lubitchv Jun 6, 2022
ec31196
Merge pull request #67 from IQSS/develop
lubitchv Dec 20, 2022
c5f44d6
Merge pull request #68 from IQSS/develop
lubitchv Jan 6, 2023
a1f2c78
Merge branch 'IQSS:develop' into develop
lubitchv Mar 8, 2023
af793ec
Merge branch 'IQSS:develop' into develop
lubitchv Mar 10, 2023
9bdcba6
Alternative Title
lubitchv Mar 13, 2023
7b8281c
citation
lubitchv Mar 13, 2023
e5a356a
release notes
lubitchv Mar 13, 2023
5d81fc7
import
lubitchv Mar 14, 2023
991c5f9
prodPlac
lubitchv Mar 15, 2023
89dbc7a
restore back
lubitchv Mar 16, 2023
538b815
Merge branch 'IQSS:develop' into 9428-alternative-title
lubitchv Mar 16, 2023
dbd97ff
develop + title
lubitchv May 11, 2023
be1e639
Merge branch 'IQSS:develop' into 9428-alternative-title
lubitchv May 11, 2023
3bb7cbf
add prodPlac
lubitchv May 11, 2023
b9720c8
Update src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUti…
lubitchv May 15, 2023
5684140
Update src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAir…
lubitchv May 15, 2023
64f4f1f
Update src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAir…
lubitchv May 15, 2023
37a372a
Update src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAir…
lubitchv May 15, 2023
2ef0e5f
test
lubitchv May 15, 2023
b55573e
Merge branch '9428-alternative-title' of github.com:lubitchv/datavers…
lubitchv May 15, 2023
4b4c915
docs
lubitchv May 15, 2023
58964db
Alternative titles
lubitchv Aug 21, 2023
10f3e1e
Merge branch 'IQSS:develop' into 9428-alternative-title
lubitchv Aug 23, 2023
a90aa22
alt title multiple
lubitchv Aug 25, 2023
85e6661
Merge branch '9428-alternative-title' of github.com:lubitchv/datavers…
lubitchv Sep 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion conf/solr/8.11.1/schema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@
<!-- SCHEMA-FIELDS::BEGIN -->
<field name="accessToSources" type="text_en" multiValued="false" stored="true" indexed="true"/>
<field name="actionsToMinimizeLoss" type="text_en" multiValued="false" stored="true" indexed="true"/>
<field name="alternativeTitle" type="text_en" multiValued="false" stored="true" indexed="true"/>
<field name="alternativeTitle" type="text_en" multiValued="true" stored="true" indexed="true"/>
<field name="alternativeURL" type="text_en" multiValued="false" stored="true" indexed="true"/>
<field name="astroFacility" type="text_en" multiValued="true" stored="true" indexed="true"/>
<field name="astroInstrument" type="text_en" multiValued="true" stored="true" indexed="true"/>
Expand Down
6 changes: 6 additions & 0 deletions doc/release-notes/9428-alternative-title.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Alternative Title is made repeatable.
- One will need to update database with updated citation block.
`curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/citation.tsv`
- One will also need to update solr schema:
Change in "alternativeTitle" field multiValued="true" in `/usr/local/solr/solr-8.11.1/server/solr/collection1/conf/schema.xml`
Reload solr schema: `curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"`
2 changes: 1 addition & 1 deletion scripts/api/data/metadatablocks/citation.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#datasetField name title description watermark fieldType displayOrder displayFormat advancedSearchField allowControlledVocabulary allowmultiples facetable displayoncreate required parent metadatablock_id termURI
title Title The main title of the Dataset text 0 TRUE FALSE FALSE FALSE TRUE TRUE citation http://purl.org/dc/terms/title
subtitle Subtitle A secondary title that amplifies or states certain limitations on the main title text 1 FALSE FALSE FALSE FALSE FALSE FALSE citation
alternativeTitle Alternative Title Either 1) a title commonly used to refer to the Dataset or 2) an abbreviation of the main title text 2 FALSE FALSE FALSE FALSE FALSE FALSE citation http://purl.org/dc/terms/alternative
alternativeTitle Alternative Title Either 1) a title commonly used to refer to the Dataset or 2) an abbreviation of the main title text 2 FALSE FALSE TRUE FALSE FALSE FALSE citation http://purl.org/dc/terms/alternative
alternativeURL Alternative URL Another URL where one can view or access the data in the Dataset, e.g. a project or personal webpage https:// url 3 <a href="#VALUE" target="_blank">#VALUE</a> FALSE FALSE FALSE FALSE FALSE FALSE citation https://schema.org/distribution
otherId Other Identifier Another unique identifier for the Dataset (e.g. producer's or another repository's identifier) none 4 : FALSE FALSE TRUE FALSE FALSE FALSE citation
otherIdAgency Agency The name of the agency that generated the other identifier text 5 #VALUE FALSE FALSE FALSE FALSE FALSE FALSE otherId citation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1339,6 +1339,7 @@ private void processProdStmt(XMLStreamReader xmlr, MetadataBlockDTO citation) th
List<HashSet<FieldDTO>> producers = new ArrayList<>();
List<HashSet<FieldDTO>> grants = new ArrayList<>();
List<HashSet<FieldDTO>> software = new ArrayList<>();
List<String> prodPlac = new ArrayList<>();

for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) {
if (event == XMLStreamConstants.START_ELEMENT) {
Expand All @@ -1354,9 +1355,7 @@ private void processProdStmt(XMLStreamReader xmlr, MetadataBlockDTO citation) th
} else if (xmlr.getLocalName().equals("prodDate")) {
citation.getFields().add(FieldDTO.createPrimitiveFieldDTO("productionDate", parseDate(xmlr, "prodDate")));
} else if (xmlr.getLocalName().equals("prodPlac")) {
List<String> prodPlac = new ArrayList<>();
prodPlac.add(parseText(xmlr, "prodPlac"));
citation.getFields().add(FieldDTO.createMultiplePrimitiveFieldDTO(DatasetFieldConstant.productionPlace, prodPlac));
prodPlac.add(parseText(xmlr));
} else if (xmlr.getLocalName().equals("software")) {
HashSet<FieldDTO> set = new HashSet<>();
addToSet(set,"softwareVersion", xmlr.getAttributeValue(null, "version"));
Expand Down Expand Up @@ -1389,6 +1388,9 @@ private void processProdStmt(XMLStreamReader xmlr, MetadataBlockDTO citation) th
if (producers.size()>0) {
citation.getFields().add(FieldDTO.createMultipleCompoundFieldDTO("producer", producers));
}
if (prodPlac.size() > 0) {
citation.getFields().add(FieldDTO.createMultiplePrimitiveFieldDTO(DatasetFieldConstant.productionPlace, prodPlac));
}
return;
}
}
Expand All @@ -1398,6 +1400,7 @@ private void processProdStmt(XMLStreamReader xmlr, MetadataBlockDTO citation) th
private void processTitlStmt(XMLStreamReader xmlr, DatasetDTO datasetDTO) throws XMLStreamException, ImportException {
MetadataBlockDTO citation = datasetDTO.getDatasetVersion().getMetadataBlocks().get("citation");
List<HashSet<FieldDTO>> otherIds = new ArrayList<>();
List<String> altTitles = new ArrayList<>();

for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) {
if (event == XMLStreamConstants.START_ELEMENT) {
Expand All @@ -1408,8 +1411,7 @@ private void processTitlStmt(XMLStreamReader xmlr, DatasetDTO datasetDTO) throws
FieldDTO field = FieldDTO.createPrimitiveFieldDTO("subtitle", parseText(xmlr));
citation.getFields().add(field);
} else if (xmlr.getLocalName().equals("altTitl")) {
FieldDTO field = FieldDTO.createPrimitiveFieldDTO("alternativeTitle", parseText(xmlr));
citation.getFields().add(field);
altTitles.add(parseText(xmlr));
} else if (xmlr.getLocalName().equals("IDNo")) {
if ( AGENCY_HANDLE.equals( xmlr.getAttributeValue(null, "agency") ) || AGENCY_DOI.equals( xmlr.getAttributeValue(null, "agency") ) ) {
importGenericService.reassignIdentifierAsGlobalId(parseText(xmlr), datasetDTO);
Expand Down Expand Up @@ -1437,6 +1439,10 @@ private void processTitlStmt(XMLStreamReader xmlr, DatasetDTO datasetDTO) throws
if (otherIds.size()>0) {
citation.addField(FieldDTO.createMultipleCompoundFieldDTO("otherId", otherIds));
}
if (!altTitles.isEmpty()) {
FieldDTO field = FieldDTO.createMultiplePrimitiveFieldDTO(DatasetFieldConstant.alternativeTitle, altTitles);
citation.getFields().add(field);
}
return;
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,11 @@ private static void createStdyDscr(XMLStreamWriter xmlw, DatasetDTO datasetDto)

writeFullElement(xmlw, "titl", dto2Primitive(version, DatasetFieldConstant.title), datasetDto.getMetadataLanguage());
writeFullElement(xmlw, "subTitl", dto2Primitive(version, DatasetFieldConstant.subTitle));
writeFullElement(xmlw, "altTitl", dto2Primitive(version, DatasetFieldConstant.alternativeTitle));
//writeFullElement(xmlw, "altTitl", dto2Primitive(version, DatasetFieldConstant.alternativeTitle));
lubitchv marked this conversation as resolved.
Show resolved Hide resolved
FieldDTO altField = dto2FieldDTO( version, DatasetFieldConstant.alternativeTitle, "citation" );
if (altField != null) {
writeMultipleElement(xmlw, "altTitl", altField, datasetDto.getMetadataLanguage());
}

xmlw.writeStartElement("IDNo");
writeAttribute(xmlw, "agency", persistentAgency);
Expand Down Expand Up @@ -1002,7 +1006,11 @@ private static void writeProducersElement(XMLStreamWriter xmlw, DatasetVersionDT
// productionPlace was made multiple as of 5.14:
// (a quick backward compatibility check was added to dto2PrimitiveList(),
// see the method for details)
writeFullElementList(xmlw, "prodPlac", dto2PrimitiveList(version, DatasetFieldConstant.productionPlace));

FieldDTO prodPlac = dto2FieldDTO( version, DatasetFieldConstant.productionPlace, "citation" );
if (prodPlac != null) {
writeMultipleElement(xmlw, "prodPlac", prodPlac, null);
}
writeSoftwareElement(xmlw, version);

writeGrantElement(xmlw, version);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
package edu.harvard.iq.dataverse.export.openaire;

import java.io.OutputStream;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.*;
import java.util.logging.Logger;

import javax.json.JsonObject;
Expand Down Expand Up @@ -326,12 +322,36 @@ public static void writeTitlesElement(XMLStreamWriter xmlw, DatasetVersionDTO da
String subtitle = dto2Primitive(datasetVersionDTO, DatasetFieldConstant.subTitle);
title_check = writeTitleElement(xmlw, "Subtitle", subtitle, title_check, language);

String alternativeTitle = dto2Primitive(datasetVersionDTO, DatasetFieldConstant.alternativeTitle);
title_check = writeTitleElement(xmlw, "AlternativeTitle", alternativeTitle, title_check, language);

//String alternativeTitle = dto2Primitive(datasetVersionDTO, DatasetFieldConstant.alternativeTitle);
//title_check = writeTitleElement(xmlw, "AlternativeTitle", alternativeTitle, title_check, language);
title_check = writeMultipleTitleElement(xmlw, "AlternativeTitle", datasetVersionDTO, "citation", title_check, language);
writeEndTag(xmlw, title_check);
}

private static boolean writeMultipleTitleElement(XMLStreamWriter xmlw, String titleType, DatasetVersionDTO datasetVersionDTO, String metadataBlockName, boolean title_check, String language) throws XMLStreamException {
MetadataBlockDTO block = datasetVersionDTO.getMetadataBlocks().get(metadataBlockName);
if (block != null) {
logger.info("Block is not empty");
lubitchv marked this conversation as resolved.
Show resolved Hide resolved
List<FieldDTO> fieldsBlock = block.getFields();
if (fieldsBlock != null) {
for (FieldDTO fieldDTO : fieldsBlock) {
logger.info(titleType + " " + fieldDTO.getTypeName());
lubitchv marked this conversation as resolved.
Show resolved Hide resolved
if (titleType.toLowerCase().equals(fieldDTO.getTypeName().toLowerCase())) {
logger.info("Found Alt title");
lubitchv marked this conversation as resolved.
Show resolved Hide resolved
List<String> fields = fieldDTO.getMultiplePrimitive();
for (String value : fields) {
if (!writeTitleElement(xmlw, titleType, value, title_check, language))
title_check = false;
}
break;
}
}
}
}

return title_check;
}

/**
* 3, Title (with optional type sub-properties) (M)
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,9 @@
},
{
"typeName": "alternativeTitle",
"multiple": false,
"multiple": true,
"typeClass": "primitive",
"value": "Alternative Title"
"value": ["Alternative Title"]
},
{
"typeName": "alternativeURL",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@
},
{
"typeName": "alternativeTitle",
"multiple": false,
"multiple": true,
"typeClass": "primitive",
"value": "Alternative Title"
"value": ["Alternative Title"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like API users will encounter backward incompatibility. Is that right? If so, can this please be added to the release note snippet?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, repeatable altTitl is valid DDI. See https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/field_level_documentation_files/schemas/codebook_xsd/elements/titlStmt.html#a19 It has maxOccurs="unbounded"

I updated dataset-finch xml and json and added multiple alternative titles such that validity test can test it.

I also added comment about incompatibility into release note.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a second value here, just like in dataset-finch1.json -

["Alternative Title1", "Alternative Title2"]

and that will need a corresponding change in src/test/java/edu/harvard/iq/dataverse/export/ddi/exportfull.xml that DdiExportUtilTest compares against.
(I'm putting together a list of all the changes that need to be made when fields are made repeatable, for other developers)
Thank you!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@landreev I added ["Alternative Title1", "Alternative Title2"] into dataset-create-new-all-ddi-fields.json and added another title in corresponding exportfull.xml

},
{
"typeName": "otherId",
Expand Down