Make Production Place repeatable and facetable, make Series repeatable #7810

philippconzett · 2021-04-19T13:37:05Z

What this PR does / why we need it:
Data contained in a dataset may have been produced at multiple places. Making the field Production Place repeatable will make it possible to reflect this fact in the dataset metadata. Making the field facetable will allow us to customize Dataverse collections more appropriately.

Which issue(s) this PR closes:
None, to my knowledge.

Merge from original repo

Turned allowmultiples facetable for Citation Metadata field Production Place from FALSE into TRUE as we'd like to be able to add multiple production places and to be able to filter dataset based on this metadata field.

Turning Citation Metadata section Series from allowmultiples=FALSE to allowmultiples=TRUE.

scolapasta · 2021-04-22T16:05:57Z

@philippconzett just to be sure this is also changing series, is that intended?

philippconzett · 2021-04-23T04:32:33Z

@scolapasta Yes, that's right. I tried to split this into two PRs, but I don't know how to do that :-/

philippconzett · 2021-11-02T14:32:30Z

@djbrooke Could we get this PR merged before the next release?

djbrooke · 2021-11-02T14:36:43Z

Hi @philippconzett - there's the change on line 69, to series, that I thought you did not want to include in this PR:

https://github.com/IQSS/dataverse/pull/7810/files#diff-9a87ef9131a9320671e94ee2b06991ee64d8ae37ddad2bff052215c55e2ddb10R69

Let me know if you'd like me to work on this. I can make changes to the PR so that only your desired changes are included.

philippconzett · 2021-11-03T06:30:18Z

Hi @djbrooke. Actually, I wanted Series to be made repeatable. I probably should have created this as a separate PR, but I didn't know how to remove it from this one. If it's OK, please merge the change together with the Production Place change. Thanks!

jggautier · 2021-11-03T12:04:01Z

Hi @philippconzett, about Production Place, in your datasets' metadata, is each Production Place associated with the metadata entered in Producer fields? I'm wondering if it would make sense to make Production Place a child field of the Producer fields.

I know this isn't how the two elements, Production Place and Producer, are organized in DDI Codebook 2 and I think that's why in the Citation metadata block they're treated as two separate fields. I'm thinking if reorganizing the fields makes sense, we can later ask the DDI folks to incorporate that change in a new version of their schema.

djbrooke · 2021-11-03T14:31:26Z

Thanks @philippconzett, I've updated the title to reflect that you want both of these changes.

Thanks @jggautier for jumping in!

philippconzett · 2021-11-04T05:24:09Z

@djbrooke Thanks for changing the title.
@jggautier Thanks for raising the issue. I'm not sure whether Production Place should be a sub-field of Producer. Production Place has the following description: "The location where the data collection and any other related materials were produced." I think for many scientific fields where data are collected and sometimes also processed offsite/outside campus, Production Place is not identical with the location of the research organization. Also, one dataset may have been produced at different locations. So, I think we should keep Production Place and Producer separate, but still make Production Place repeatable. Thanks.

jggautier · 2021-11-04T14:23:49Z

Thanks @philippconzett. That makes sense. This makes me think that "location" in the description doesn't always mean a geographical location. For some datasets, people may need to include the name of an organization.

@djbrooke you mentioned yesterday that you might have questions?

djbrooke · 2021-11-05T13:40:07Z

@jggautier thanks, no questions at the moment! I'll update from dev and add a release note about updating citation.csv.

djbrooke · 2021-11-05T15:37:34Z

OK - all set. I will keep this assigned to me because I'll need to update from develop once we cut 5.8.

djbrooke · 2021-11-18T17:20:15Z

Thanks @kcondon

@philippconzett we'll take care of the remaining release notes updates on this side. Thanks!

pdurbin

The "series" subfields should be updated in schema.xml. We plan to give more guidance about this in #8248.

pdurbin · 2021-11-19T20:46:11Z

conf/solr/8.8.1/schema.xml

@@ -359,7 +359,7 @@
    <field name="responseRate" type="text_en" multiValued="false" stored="true" indexed="true"/>
    <field name="samplingErrorEstimates" type="text_en" multiValued="false" stored="true" indexed="true"/>
    <field name="samplingProcedure" type="text_en" multiValued="false" stored="true" indexed="true"/>
-    <field name="series" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="series" type="text_en" multiValued="true" stored="true" indexed="true"/>


I think we want the subfields of "series" to be multivalued as well. Please check out this diff:

- <field name="series" type="text_en" multiValued="false" stored="true" indexed="true"/> - <field name="seriesInformation" type="text_en" multiValued="false" stored="true" indexed="true"/> - <field name="seriesName" type="text_en" multiValued="false" stored="true" indexed="true"/> + <field name="series" type="text_en" multiValued="true" stored="true" indexed="true"/> + <field name="seriesInformation" type="text_en" multiValued="true" stored="true" indexed="true"/> + <field name="seriesName" type="text_en" multiValued="true" stored="true" indexed="true"/>

I added the subfields of series in 4509dea.

Or wouldn't it make more sense to change Series to a complex field, such as Author? I guess this would mean:

series
allowmultiples: TRUE

seriesName
allowmultiples: FALSE

seriesInformation
allowmultiples: FALSE

pdurbin · 2021-11-19T21:01:51Z

When I load the TSV in this pull request and do the manual process I used to do I get a bigger "diff" below than I expect.

Here's the resulting schema.xml (renamed so I can upload it): schema.xml.txt

The diff:

diff --git a/conf/solr/8.8.1/schema.xml b/conf/solr/8.8.1/schema.xml
index 1718962bfb..0a4e6f3d96 100644
--- a/conf/solr/8.8.1/schema.xml
+++ b/conf/solr/8.8.1/schema.xml
@@ -342,7 +342,7 @@
     <field name="producerName" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="producerURL" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="productionDate" type="text_en" multiValued="false" stored="true" indexed="true"/>
-    <field name="productionPlace" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="productionPlace" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="publication" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="publicationCitation" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="publicationIDNumber" type="text_en" multiValued="true" stored="true" indexed="true"/>
@@ -359,9 +359,9 @@
     <field name="responseRate" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="samplingErrorEstimates" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="samplingProcedure" type="text_en" multiValued="false" stored="true" indexed="true"/>
-    <field name="series" type="text_en" multiValued="false" stored="true" indexed="true"/>
-    <field name="seriesInformation" type="text_en" multiValued="false" stored="true" indexed="true"/>
-    <field name="seriesName" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="series" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="seriesInformation" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="seriesName" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="socialScienceNotes" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="socialScienceNotesSubject" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="socialScienceNotesText" type="text_en" multiValued="false" stored="true" indexed="true"/>
@@ -376,13 +376,13 @@
     <field name="studyAssayOrganism" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyAssayOtherMeasurmentType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyAssayOtherOrganism" type="text_en" multiValued="true" stored="true" indexed="true"/>
-    <field name="studyAssayPlatform" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyAssayOtherPlatform" type="text_en" multiValued="true" stored="true" indexed="true"/>
-    <field name="studyAssayTechnologyType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyAssayOtherTechnologyType" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="studyAssayPlatform" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="studyAssayTechnologyType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyDesignType" type="text_en" multiValued="true" stored="true" indexed="true"/>
-    <field name="studyOtherDesignType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyFactorType" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="studyOtherDesignType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="studyOtherFactorType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="subject" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="subtitle" type="text_en" multiValued="false" stored="true" indexed="true"/>
@@ -613,13 +613,13 @@
     <copyField source="studyAssayOrganism" dest="_text_" maxChars="3000"/>
     <copyField source="studyAssayOtherMeasurmentType" dest="_text_" maxChars="3000"/>
     <copyField source="studyAssayOtherOrganism" dest="_text_" maxChars="3000"/>
-    <copyField source="studyAssayPlatform" dest="_text_" maxChars="3000"/>
     <copyField source="studyAssayOtherPlatform" dest="_text_" maxChars="3000"/>
-    <copyField source="studyAssayTechnologyType" dest="_text_" maxChars="3000"/>
     <copyField source="studyAssayOtherTechnologyType" dest="_text_" maxChars="3000"/>
+    <copyField source="studyAssayPlatform" dest="_text_" maxChars="3000"/>
+    <copyField source="studyAssayTechnologyType" dest="_text_" maxChars="3000"/>
     <copyField source="studyDesignType" dest="_text_" maxChars="3000"/>
-    <copyField source="studyOtherDesignType" dest="_text_" maxChars="3000"/>
     <copyField source="studyFactorType" dest="_text_" maxChars="3000"/>
+    <copyField source="studyOtherDesignType" dest="_text_" maxChars="3000"/>
     <copyField source="studyOtherFactorType" dest="_text_" maxChars="3000"/>
     <copyField source="subject" dest="_text_" maxChars="3000"/>
     <copyField source="subtitle" dest="_text_" maxChars="3000"/>

Updating from the API output picked up the missing subfields of "series" which is a compound field.

pdurbin · 2021-11-22T16:15:14Z

In 4509dea I updated schema.xml because the subfields of "series" had not been made multivalued. In #8248 we plan to explain the recommended way for developers to update schema.xml when making changes to metadata blocks we ship with the app, such as citation. In short, you should use the output from http://localhost:8080/api/admin/index/solr/schema

I've been poking a Solr and search-related stuff on this pull request. Somewhat surprisingly, it seems like a full reindex may not be required. I was able to create a dataset under "develop" and still have it's newly multivalued fields show up as facets once I deployed the new code.

Once I had the new code deployed, however, I noticed some bugs. Here's what I posted in Slack:

DDI export fails (and maybe other exports?) if you specify multiple production places.

“More than one series title found” is logged over and more if you specify more than series. It seems to be used in the citation.

Basically, there is hard-coded logic around these fields in the Java code. Changing the TSV is not enough.

A rule of thumb might be… if you’re messing with citation fields, go in DatasetFieldConstant and do a “find usages” for the fields you’re changing and try to figure out the impact.

djbrooke · 2021-11-22T17:38:17Z

Hi @philippconzett, as you can see in the note from @pdurbin, during review and testing we realized that these changes will take more work to implement than just updates to the .tsv and accompanying release notes and Solr schema changes. I'll need to look at our other work in progress to see if this is something that we can prioritize, but I don't believe we'll be able to work on the necessary code changes here before the end of the year.

pdurbin · 2022-03-16T20:42:17Z

I just wanted to mention that we recently gained some experience changing allowMultiples from false to true with "Collection Mode" (collectionMode) in PR #8473. One thing to look out for is to make sure DDI import continues to work. See PR #8491 and PR #8483.

jggautier · 2022-03-17T17:39:39Z

This applies the change at the installation level, so it's not possible for managers of collections within a Dataverse installation to control the number of values allowed for a field. Are there collection managers within Dataverse installations who want to make sure that depositors can enter only one Production Place?

Should the same question be investigated for other fields we consider making repeatable, including Collection Mode (although that field's already been made repeatable), before solutions are proposed and implemented?

pdurbin · 2022-10-05T01:12:00Z

@philippconzett heads up that there are merge conflicts in this PR. Do you need a hand resolving them?

pdurbin · 2023-01-04T15:39:59Z

@philippconzett and I just had a call about this PR and we're closing it in favor of these:

philippconzett added 3 commits April 16, 2021 16:06

Merge pull request #1 from IQSS/develop

11a3e94

Merge from original repo

Update citation.tsv

d8ffe4c

Turned allowmultiples facetable for Citation Metadata field Production Place from FALSE into TRUE as we'd like to be able to add multiple production places and to be able to filter dataset based on this metadata field.

Update citation.tsv

f3334f5

Turning Citation Metadata section Series from allowmultiples=FALSE to allowmultiples=TRUE.

djbrooke assigned philippconzett Apr 22, 2021

Merge branch 'IQSS:develop' into develop

8c21211

djbrooke mentioned this pull request Jul 26, 2021

8023 doc adminguide add #8024

Merged

djbrooke self-assigned this Nov 2, 2021

djbrooke changed the title ~~Make Production Place repeatable and facetable~~ Make Production Place repeatable and facetable, make series repeatable Nov 3, 2021

djbrooke changed the title ~~Make Production Place repeatable and facetable, make series repeatable~~ Make Production Place repeatable and facetable, make Series repeatable Nov 3, 2021

djbrooke added 2 commits November 5, 2021 09:40

Merge branch 'develop' into pr/7810

7f61654

adding release note

6060d5f

djbrooke unassigned philippconzett Nov 5, 2021

djbrooke added 2 commits November 5, 2021 14:02

Merge branch 'develop' into pr/7810

35ff4a5

fixing error and rerunning tests

badf7d6

djbrooke removed their assignment Nov 5, 2021

update release note

d83ba71

djbrooke self-assigned this Nov 17, 2021

kcondon removed their assignment Nov 18, 2021

updating release note, updating schema

5c8ff66

djbrooke removed their assignment Nov 18, 2021

djbrooke mentioned this pull request Nov 19, 2021

Better Instructions for Updating Metadata Blocks, how to handle Solr updates in PRs #8248

Open

pdurbin self-assigned this Nov 19, 2021

pdurbin requested changes Nov 19, 2021

View reviewed changes

pdurbin removed their assignment Nov 19, 2021

pdurbin assigned djbrooke Nov 19, 2021

update schema.xml from API output IQSS#7810

4509dea

Updating from the API output picked up the missing subfields of "series" which is a compound field.

pdurbin self-assigned this Nov 22, 2021

djbrooke unassigned pdurbin Nov 22, 2021

pdurbin assigned philippconzett and unassigned djbrooke Oct 5, 2022

mreekie added the bk2211 label Nov 1, 2022

pdurbin closed this Jan 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Production Place repeatable and facetable, make Series repeatable #7810

Make Production Place repeatable and facetable, make Series repeatable #7810

philippconzett commented Apr 19, 2021

scolapasta commented Apr 22, 2021

philippconzett commented Apr 23, 2021

philippconzett commented Nov 2, 2021

djbrooke commented Nov 2, 2021

philippconzett commented Nov 3, 2021

jggautier commented Nov 3, 2021 •

edited

Loading

djbrooke commented Nov 3, 2021

philippconzett commented Nov 4, 2021

jggautier commented Nov 4, 2021 •

edited

Loading

djbrooke commented Nov 5, 2021

djbrooke commented Nov 5, 2021

djbrooke commented Nov 18, 2021

pdurbin left a comment

pdurbin Nov 19, 2021

pdurbin Nov 22, 2021

philippconzett Nov 23, 2021

pdurbin commented Nov 19, 2021

pdurbin commented Nov 22, 2021

djbrooke commented Nov 22, 2021

pdurbin commented Mar 16, 2022

jggautier commented Mar 17, 2022 •

edited

Loading

pdurbin commented Oct 5, 2022

pdurbin commented Jan 4, 2023 •

edited

Loading

Make Production Place repeatable and facetable, make Series repeatable #7810

Make Production Place repeatable and facetable, make Series repeatable #7810

Conversation

philippconzett commented Apr 19, 2021

scolapasta commented Apr 22, 2021

philippconzett commented Apr 23, 2021

philippconzett commented Nov 2, 2021

djbrooke commented Nov 2, 2021

philippconzett commented Nov 3, 2021

jggautier commented Nov 3, 2021 • edited Loading

djbrooke commented Nov 3, 2021

philippconzett commented Nov 4, 2021

jggautier commented Nov 4, 2021 • edited Loading

djbrooke commented Nov 5, 2021

djbrooke commented Nov 5, 2021

djbrooke commented Nov 18, 2021

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Nov 19, 2021

Choose a reason for hiding this comment

pdurbin Nov 22, 2021

Choose a reason for hiding this comment

philippconzett Nov 23, 2021

Choose a reason for hiding this comment

pdurbin commented Nov 19, 2021

pdurbin commented Nov 22, 2021

djbrooke commented Nov 22, 2021

pdurbin commented Mar 16, 2022

jggautier commented Mar 17, 2022 • edited Loading

pdurbin commented Oct 5, 2022

pdurbin commented Jan 4, 2023 • edited Loading

jggautier commented Nov 3, 2021 •

edited

Loading

jggautier commented Nov 4, 2021 •

edited

Loading

jggautier commented Mar 17, 2022 •

edited

Loading

pdurbin commented Jan 4, 2023 •

edited

Loading