Merge branch 'develop' into 4709-postgresql_96 IQSS#4709

doc/sphinx-guides/source/developers/tips.rst

@@ -102,7 +102,7 @@ Deploying With ``asadmin``
 
 Sometimes you want to deploy code without using Netbeans or from the command line on a server you have ssh'ed into.
 
-For the ``asadmin`` commands below, we assume you have already changed directories to ``/usr/local/glassfish4`` or wherever you have installed Glassfish.
+For the ``asadmin`` commands below, we assume you have already changed directories to ``/usr/local/glassfish4/glassfish/bin`` or wherever you have installed Glassfish.
 
 There are four steps to this process:
 

doc/sphinx-guides/source/installation/config.rst

@@ -124,7 +124,7 @@ Out of the box, Dataverse is configured for DOIs. Here are the configuration opt
 - :ref:`:DoiProvider <:DoiProvider>`
 - :ref:`:Protocol <:Protocol>`
 - :ref:`:Authority <:Authority>`
-- :ref:`:DoiSeparator <:DoiSeparator>`
+- :ref:`:Shoulder <:Shoulder>`
 - :ref:`:IdentifierGenerationStyle <:IdentifierGenerationStyle>` (optional)
 - :ref:`:DataFilePIDFormat <:DataFilePIDFormat>` (optional)
 
@@ -552,7 +552,7 @@ See also these related database settings below:
 - :ref:`:DoiProvider`
 - :ref:`:Protocol`  
 - :ref:`:Authority`
-- :ref:`:DoiSeparator`
+- :ref:`:Shoulder`
 
 .. _doi.username:
 
@@ -717,7 +717,7 @@ As of this writing "EZID" and "DataCite" are the only valid options. DoiProvider
 
 ``curl -X PUT -d EZID http://localhost:8080/api/admin/settings/:DoiProvider``
 
-This setting relates to the ``:Protocol``, ``:Authority``, ``:DoiSeparator``, and ``:IdentifierGenerationStyle`` database settings below as well as the following JVM options:
+This setting relates to the ``:Protocol``, ``:Authority``, ``:Shoulder``, and ``:IdentifierGenerationStyle`` database settings below as well as the following JVM options:
 
 - :ref:`doi.baseurlstring`
 - :ref:`doi.username`
@@ -741,25 +741,23 @@ Use the authority assigned to you by your DoiProvider or HandleProvider.
 
 ``curl -X PUT -d 10.xxxx http://localhost:8080/api/admin/settings/:Authority``
 
-.. _:DoiSeparator:
+.. _:Shoulder:
 
-:DoiSeparator
-+++++++++++++
-
-It is recommended that you keep this as a slash ("/").
+:Shoulder
+++++++++++++
 
-``curl -X PUT -d "/" http://localhost:8080/api/admin/settings/:DoiSeparator``
+Out of the box, the DOI shoulder is set to "FK2/" but this is for testing only! When you apply for your DOI namespace, you may have requested a shoulder. The following is only an example and a trailing slash is optional.
 
-**Note:** The name DoiSeparator is a misnomer. This setting is used by some **handles**-specific code too. It *must* be set to '/' when using handles.
+``curl -X PUT -d "MyShoulder/" http://localhost:8080/api/admin/settings/:Shoulder``
 
 .. _:IdentifierGenerationStyle:
 
 :IdentifierGenerationStyle
 ++++++++++++++++++++++++++
 
-By default, Dataverse generates a random 6 character string to use as the identifier
+By default, Dataverse generates a random 6 character string, pre-pended by the Shoulder if set, to use as the identifier
 for a Dataset. Set this to ``sequentialNumber`` to use sequential numeric values 
-instead. (the assumed default setting is ``randomString``). 
+instead (again pre-pended by the Shoulder if set). (the assumed default setting is ``randomString``). 
 In addition to this setting, a database sequence must be created in the database. 
 We provide the script below (downloadable :download:`here </_static/util/createsequence.sql>`).
 You may need to make some changes to suit your system setup, see the comments for more information: 
@@ -785,7 +783,7 @@ This setting controls the way that the "identifier" component of a file's persis
 
 By default the identifier for a file is dependent on its parent dataset. For example, if the identifier of a dataset is "TJCLKP", the identifier for a file within that dataset will consist of the parent dataset's identifier followed by a slash ("/"), followed by a random 6 character string, yielding "TJCLKP/MLGWJO". Identifiers in this format are what you should expect if you leave ``:DataFilePIDFormat`` undefined or set it to ``DEPENDENT`` and have not changed the ``:IdentifierGenerationStyle`` setting from its default.
 
-Alternatively, the indentifier for File PIDs can be configured to be independent of Dataset PIDs using the setting "``INDEPENDENT``". In this case, file PIDs will not contain the PIDs of their parent datasets, and their PIDs will be generated the exact same way that datasets' PIDs are, based on the ``:IdentifierGenerationStyle`` setting described above (random 6 character strings or sequential numbers).
+Alternatively, the identifier for File PIDs can be configured to be independent of Dataset PIDs using the setting "``INDEPENDENT``". In this case, file PIDs will not contain the PIDs of their parent datasets, and their PIDs will be generated the exact same way that datasets' PIDs are, based on the ``:IdentifierGenerationStyle`` setting described above (random 6 character strings or sequential numbers, pre-pended by any shoulder).
 
 The chart below shows examples from each possible combination of parameters from the two settings. ``:IdentifierGenerationStyle`` can be either ``randomString`` (the default) or ``sequentialNumber`` and ``:DataFilePIDFormat`` can be either ``DEPENDENT`` (the default) or ``INDEPENDENT``. In the examples below the "identifier" for the dataset is "TJCLKP" for "randomString" and "100001" for "sequentialNumber".
 

scripts/api/setup-all.sh

@@ -49,9 +49,9 @@ curl -X PUT -d yes "$SERVER/admin/settings/:AllowSignUp"
 curl -X PUT -d /dataverseuser.xhtml?editMode=CREATE "$SERVER/admin/settings/:SignUpUrl"
 
 curl -X PUT -d doi "$SERVER/admin/settings/:Protocol"
-curl -X PUT -d 10.5072/FK2 "$SERVER/admin/settings/:Authority"
+curl -X PUT -d 10.5072 "$SERVER/admin/settings/:Authority"
+curl -X PUT -d "FK2/" "$SERVER/admin/settings/:Shoulder"
 curl -X PUT -d EZID "$SERVER/admin/settings/:DoiProvider"
-curl -X PUT -d / "$SERVER/admin/settings/:DoiSeparator"
 curl -X PUT -d burrito $SERVER/admin/settings/BuiltinUsers.KEY
 curl -X PUT -d localhost-only $SERVER/admin/settings/:BlockedApiPolicy
 echo

scripts/api/setup-optional-harvard.sh

@@ -2,6 +2,9 @@
 SERVER=http://localhost:8080/api
 
 echo "Setting up Harvard-specific settings"
+# :Authority and :Shoulder are commented out so this script can be used on test servers
+#curl -X PUT -d 10.7910 "$SERVER/admin/settings/:Authority"
+#curl -X PUT -d "DVN/" "$SERVER/admin/settings/:Shoulder"
 echo "- Application Status header"
 curl -s -X PUT -d 'Upgrade in progress...' $SERVER/admin/settings/:StatusMessageHeader
 echo "- Application Status message"

scripts/database/upgrades/upgrade_v4.8.6_to_v4.9.0.sql

@@ -1,56 +1,71 @@
-ALTER TABLE externaltool ADD COLUMN type character varying(255);
-ALTER TABLE externaltool ALTER COLUMN type SET NOT NULL;
--- Previously, the only explore tool was TwoRavens. We now persist the name of the tool.
-UPDATE guestbookresponse SET downloadtype = 'TwoRavens' WHERE downloadtype = 'Explore';
-ALTER TABLE filemetadata ADD COLUMN prov_freeform text;
--- ALTER TABLE datafile ADD COLUMN prov_cplid int;
-ALTER TABLE datafile ADD COLUMN prov_entityname text;
-
--- Moves DOI fields from Dataset to DVObject
--- so that Identifiers may be added to DataFiles
-
-ALTER TABLE dvobject ADD COLUMN
-  authority character varying(255),
-  ADD COLUMN doiseparator character varying(255),
-  ADD COLUMN globalidcreatetime timestamp without time zone,
-  ADD COLUMN identifierRegistered boolean,
-  ADD COLUMN identifier character varying(255),
-  ADD COLUMN protocol character varying(255);
-
-
-UPDATE dvobject
-SET authority=(SELECT dataset.authority
-FROM dataset
-WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
-
-UPDATE dvobject
-SET doiseparator=(SELECT dataset.doiseparator
-FROM dataset
-WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
-
-UPDATE dvobject
-SET globalidcreatetime=(SELECT dataset.globalidcreatetime
-FROM dataset
-WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
-
-UPDATE dvobject
-SET identifierRegistered= true where globalidcreatetime is not null;
-
-UPDATE dvobject
-SET identifier=(SELECT dataset.identifier
-FROM dataset
-WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
-
-UPDATE dvobject
-SET protocol=(SELECT dataset.protocol
-FROM dataset
-WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
-
-ALTER TABLE dataset ALTER identifier DROP NOT NULL;
-
-ALTER TABLE dataset DROP COLUMN authority;
-ALTER TABLE dataset DROP COLUMN doiseparator;
-ALTER TABLE dataset DROP COLUMN globalidcreatetime;
-ALTER TABLE dataset DROP COLUMN identifier;
-ALTER TABLE dataset DROP COLUMN protocol;
-
+ALTER TABLE externaltool ADD COLUMN type character varying(255);
+ALTER TABLE externaltool ALTER COLUMN type SET NOT NULL;
+-- Previously, the only explore tool was TwoRavens. We now persist the name of the tool.
+UPDATE guestbookresponse SET downloadtype = 'TwoRavens' WHERE downloadtype = 'Explore';
+ALTER TABLE filemetadata ADD COLUMN prov_freeform text;
+-- ALTER TABLE datafile ADD COLUMN prov_cplid int;
+ALTER TABLE datafile ADD COLUMN prov_entityname text;
+
+-- Moves DOI fields from Dataset to DVObject
+-- so that Identifiers may be added to DataFiles
+
+ALTER TABLE dvobject ADD COLUMN
+ authority character varying(255),
+ ADD COLUMN globalidcreatetime timestamp without time zone,
+ ADD COLUMN doiseparator character varying(255),
+ ADD COLUMN identifierRegistered boolean,
+ ADD COLUMN identifier character varying(255),
+ ADD COLUMN protocol character varying(255);
+
+--Migrate data from Dataset to DvObject
+UPDATE dvobject
+SET authority=(SELECT dataset.authority
+FROM dataset
+WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
+
+UPDATE dvobject
+SET globalidcreatetime=(SELECT dataset.globalidcreatetime
+FROM dataset
+WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
+
+UPDATE dvobject
+SET doiseparator=(SELECT dataset.doiseparator
+FROM dataset
+WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
+
+UPDATE dvobject
+SET identifierRegistered= true where globalidcreatetime is not null;
+
+UPDATE dvobject
+SET identifier=(SELECT dataset.identifier
+FROM dataset
+WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset') where dvobject.dtype='Dataset';
+
+UPDATE dvobject
+SET protocol=(SELECT dataset.protocol
+FROM dataset
+WHERE dataset.id=dvobject.id AND dvobject.dtype='Dataset' ) where dvobject.dtype='Dataset';
+
+--Once in DvObject re-parse identifier and authority
+UPDATE dvobject SET identifier=substring(authority, strpos(authority,'/')+1) || doiseparator || identifier WHERE strpos(authority,'/')>0 ;
+UPDATE dvobject SET authority=substring(authority from 0 for strpos(authority,'/')) WHERE strpos(authority,'/')>0;
+
+ALTER TABLE dataset ALTER identifier DROP NOT NULL;
+
+ALTER TABLE dataset DROP COLUMN authority;
+ALTER TABLE dataset DROP COLUMN doiseparator;
+ALTER TABLE dataset DROP COLUMN globalidcreatetime;
+ALTER TABLE dataset DROP COLUMN identifier;
+ALTER TABLE dataset DROP COLUMN protocol;
+
+ALTER TABLE dvobject DROP COLUMN doiseparator;
+
+--Add new setting into content for shoulder
+INSERT INTO setting(name, content)
+VALUES (':Shoulder', (SELECT substring(content, strpos(content,'/')+1) || '/' from setting  where name = ':Authority'));
+
+ --strip shoulder from authority setting
+ UPDATE setting
+ SET content=(SELECT substring(content from 0 for strpos(content,'/'))
+ FROM setting
+ WHERE name=':Authority' and strpos(content,'/')>0) where name=':Authority';

scripts/deploy/phoenix.dataverse.org/post

@@ -5,6 +5,7 @@ cd ../..
 psql -U dvnapp dvndb -f scripts/database/reference_data.sql
 psql -U dvnapp dvndb -f doc/sphinx-guides/source/_static/util/pg8-createsequence-prep.sql
 psql -U dvnapp dvndb -f doc/sphinx-guides/source/_static/util/createsequence.sql
+curl http://localhost:8080/api/admin/settings/:DoiProvider -X PUT -d DataCite
 scripts/search/tests/publish-dataverse-root
 git checkout scripts/api/data/dv-root.json
 scripts/search/tests/grant-authusers-add-on-root

scripts/search/assumptions

@@ -16,7 +16,7 @@ export FIRST_SPRUCE_DOI=`curl --insecure -s -u spruce:spruce https://localhost:8
 
 export FIRST_SPRUCE_DATASET_ID=`curl -s "http://localhost:8080/api/dataverses/spruce/contents?key=$SPRUCEKEY" | jq '.data[0].id'`
 
-export FIRST_SPRUCE_DOI=doi:10.5072/FK2/`curl -s "http://localhost:8080/api/datasets/$FIRST_SPRUCE_DATASET_ID?key=$SPRUCEKEY" | jq .data.identifier | tr -d \"`
+export FIRST_SPRUCE_DOI=doi:10.5072/`curl -s "http://localhost:8080/api/datasets/$FIRST_SPRUCE_DATASET_ID?key=$SPRUCEKEY" | jq .data.identifier | tr -d \"`
 
 export FIRST_SPRUCE_FILE=`scripts/api/data-deposit/show-statement $FIRST_SPRUCE_DOI 2>/dev/null | xmlstarlet sel -t -v '//_:feed/_:entry/_:id' 2>/dev/null | cut -d '/' -f11`
 

scripts/search/tests/create-all-and-test

@@ -6,11 +6,13 @@ scripts/search/create-bird-dvs1 > /tmp/bird-dvs1
 scripts/search/populate-tree-dvs1
 scripts/search/create-tree-dvs1 > /tmp/tree-dvs1
 echo "Creating some datasets"
-curl -s --insecure --data-binary @scripts/search/tests/data/dataset-trees1.xml -H 'Content-Type: application/atom+xml' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/spruce | xmllint -format - >/dev/null
-curl -s -X POST -H "Content-type:application/json" -d @scripts/search/tests/data/dataset-finch1.json "http://localhost:8080/api/dataverses/finches/datasets/?key=$FINCHKEY" >/dev/null
+curl -s --insecure --data-binary @scripts/search/tests/data/dataset-trees1.xml -H 'Content-Type: application/atom+xml' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/spruce
+echo
+curl -s -X POST -H "Content-type:application/json" -d @scripts/search/tests/data/dataset-finch1.json "http://localhost:8080/api/dataverses/finches/datasets/?key=$FINCHKEY"
 echo "Uploading a file via the SWORD API"
 . scripts/search/assumptions
-curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H 'Content-Disposition: filename=trees.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI >/dev/null
+curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H 'Content-Disposition: filename=trees.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI
+echo
 echo "Uploading a file via the native API"
 # echo $FIRST_FINCH_DOI # FIXME: Why is this empty?
 STATUS_CODE_FROM_UPLOADING_FILE_VIA_NATIVE=$(curl -H "X-Dataverse-key:$FINCHKEY" --insecure --write-out %{http_code} --silent --output /dev/null -X POST -F "file=@scripts/search/data/replace_test/growing_file/2016-01/data.tsv" -F 'jsonData={"description":"My description.","categories":["Data"]}' "http://localhost:8080/api/v1/datasets/$FIRST_FINCH_DATASET_ID/add")

src/main/java/Bundle.properties

@@ -63,6 +63,9 @@ selectedFiles=Selected Files
 htmlAllowedTitle=Allowed HTML Tags
 htmlAllowedMsg=This field supports only certain <span class="text-info popoverHTML">HTML tags</span>.
 htmlAllowedTags=<a>, <b>, <blockquote>, <br>, <code>, <del>, <dd>, <dl>, <dt>, <em>, <hr>, <h1>-<h3>, <i>, <img>, <kbd>, <li>, <ol>, <p>, <pre>, <s>, <sup>, <sub>, <strong>, <strike>, <ul>
+toggleNavigation=Toggle navigation
+defaultBody=Default Body
+filter=Filter
 
 # dataverse_header.xhtml
 header.status.header=Status
@@ -278,6 +281,9 @@ auth.providers.persistentUserIdName.github=ID
 auth.providers.persistentUserIdTooltip.orcid=ORCID provides a persistent digital identifier that distinguishes you from other researchers.
 auth.providers.persistentUserIdTooltip.github=GitHub assigns a unique number to every user.
 auth.providers.orcid.insufficientScope=Dataverse was not granted the permission to read user data from ORCID.
+auth.providers.orcid.helpmessage1=ORCID is an open, non-profit, community-based effort to provide a registry of unique researcher identifiers and a transparent method of linking research activities and outputs to these identifiers. ORCID is unique in its ability to reach across disciplines, research sectors, and national boundaries and its cooperation with other identifier systems. Find out more at <a href="https://orcid.org/about" target="_blank">orcid.org/about</a>.
+auth.providers.orcid.helpmessage2=This repository uses your ORCID for authentication (so you don't need another username/password combination).  Having your ORCID associated with your datasets also makes it easier for people to find the datasets you have published.  
+
 # Friendly AuthenticationProvider names
 authenticationProvider.name.builtin=Dataverse
 authenticationProvider.name.null=(provider is unknown)
@@ -850,6 +856,7 @@ dataverse.permissions.title=Permissions
 dataverse.permissions.dataset.title=Dataset Permissions
 dataverse.permissions.access.accessBtn=Edit Access
 dataverse.permissions.usersOrGroups=Users/Groups
+dataverse.permissions.requests=Requests
 dataverse.permissions.usersOrGroups.assignBtn=Assign Roles to Users/Groups
 dataverse.permissions.usersOrGroups.createGroupBtn=Create Group
 dataverse.permissions.usersOrGroups.description=All the users and groups that have access to your dataverse.
@@ -1487,6 +1494,7 @@ file.dataFilesTab.dataAccess.verify.label=Verify Data
 file.dataFilesTab.dataAccess.local.tooltip=If this data is locally available to you, this is its file path.
 file.dataFilesTab.dataAccess.download.tooltip=Download this data from your preferred mirror by running this command.
 file.dataFilesTab.dataAccess.verify.tooltip=This command runs a checksum to verify the integrity of the data you have downloaded.
+file.dataFilesTab.button.direct=Direct
 
 file.dataFilesTab.versions=Versions
 file.dataFilesTab.versions.headers.dataset=Dataset

src/main/java/edu/harvard/iq/dataverse/AbstractIdServiceBean.java

@@ -26,9 +26,9 @@ public abstract class AbstractIdServiceBean implements IdServiceBean {
     SystemConfig systemConfig;
 
     @Override
-    public String getIdentifierForLookup(String protocol, String authority, String separator, String identifier) {
+    public String getIdentifierForLookup(String protocol, String authority, String identifier) {
         logger.log(Level.FINE,"getIdentifierForLookup");
-        return protocol + ":" + authority + separator + identifier;
+        return protocol + ":" + authority + "/" + identifier;
     }
 
 
@@ -96,7 +96,6 @@ public DvObject generateIdentifier(DvObject dvObject) {
 
         String protocol = dvObject.getProtocol() == null ? settingsService.getValueForKey(SettingsServiceBean.Key.Protocol) : dvObject.getProtocol();
         String authority = dvObject.getAuthority() == null ? settingsService.getValueForKey(SettingsServiceBean.Key.Authority) : dvObject.getAuthority();
-        String doiSeparator = dvObject.getDoiSeparator() == null ? settingsService.getValueForKey(SettingsServiceBean.Key.DoiSeparator) : dvObject.getDoiSeparator();
         IdServiceBean idServiceBean = IdServiceBean.getBean(protocol, commandEngine.getContext());
         if (dvObject.isInstanceofDataset()) {
             dvObject.setIdentifier(datasetService.generateDatasetIdentifier((Dataset) dvObject, idServiceBean));
@@ -109,9 +108,6 @@ public DvObject generateIdentifier(DvObject dvObject) {
         if (dvObject.getAuthority() == null) {
             dvObject.setAuthority(authority);
         }
-        if (dvObject.getDoiSeparator() == null) {
-            dvObject.setDoiSeparator(doiSeparator);
-        }
         return dvObject;
     }
 }