diff --git a/doc/release-notes/7000-mpconfig-support.md b/doc/release-notes/7000-mpconfig-support.md new file mode 100644 index 00000000000..01f21caf37a --- /dev/null +++ b/doc/release-notes/7000-mpconfig-support.md @@ -0,0 +1,8 @@ +# Broader MicroProfile Config Support for Developers + +As of this release, many [JVM options](https://guides.dataverse.org/en/latest/installation/config.html#jvm-options) +can be set using any [MicroProfile Config Source](https://docs.payara.fish/community/docs/Technical%20Documentation/MicroProfile/Config/Overview.html#config-sources). + +Currently this change is only relevant to developers but as settings are migrated to the new "lookup" pattern documented in the [Consuming Configuration](https://guides.dataverse.org/en/latest/developers/configuration.html) section of the Developer Guide, anyone installing the Dataverse software will have much greater flexibility when configuring those settings, especially within containers. These changes will be announced in future releases. + +Please note that an upgrade to Payara 5.2021.8 or higher is required to make use of this. Payara 5.2021.5 threw exceptions, as explained in PR #8823. diff --git a/doc/release-notes/8535-metadata-types-static-facet.md b/doc/release-notes/8535-metadata-types-static-facet.md new file mode 100644 index 00000000000..023000c0977 --- /dev/null +++ b/doc/release-notes/8535-metadata-types-static-facet.md @@ -0,0 +1,6 @@ +## Adding new static search facet: Metadata Types +A new static search facet has been added to the search side panel. This new facet is called "Metadata Types" and is driven from metadata blocks. When a metadata field value is inserted into a dataset, an entry for the metadata block it belongs to is added to this new facet. + +This new facet needs to be configured for it to appear on the search side panel. The configuration assigns to a dataverse what metadata blocks to show. The configuration is inherited by child dataverses. + +To configure the new facet, use the Metadata Block Facet API: \ No newline at end of file diff --git a/doc/release-notes/8611-DataCommons-related-notes.md b/doc/release-notes/8611-DataCommons-related-notes.md new file mode 100644 index 00000000000..af222db5b9f --- /dev/null +++ b/doc/release-notes/8611-DataCommons-related-notes.md @@ -0,0 +1,81 @@ +# Dataverse Software 5.12 + +This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. + +## Release Highlights + +### Harvard Data Commons Additions + +As reported at the 2022 Dataverse Community Meeting, the [Harvard Data Commons](https://sites.harvard.edu/harvard-data-commons/) project has supported a wide range of additions to the Dataverse software that improve support for Big Data, Workflows, Archiving, and interaction with other repositories. In many cases, these additions build upon features developed within the Dataverse community by Borealis, DANS, QDR, TDL, and others. Highlights from this work include: + +- Initial support for Globus file transfer to upload to and download from a Dataverse managed S3 store. The current implementation disables file restriction and embargo on Globus-enabled stores. +- Initial support for Remote File Storage. This capability, enabled via a new RemoteOverlay store type, allows a file stored in a remote system to be added to a dataset (currently only via API) with download requests redirected to the remote system. Use cases include referencing public files hosted on external web servers as well as support for controlled access managed by Dataverse (e.g. via restricted and embargoed status) and/or by the remote store. +- Initial support for computational workflows, including a new metadata block and detected filetypes. +- Support for archiving to any S3 store using Dataverse's RDA-conformant BagIT file format (a BagPack). +- Improved error handling and performance in archival bag creation and new options such as only supporting archiving of one dataset version. +- Additions/corrections to the OAI-ORE metadata format (which is included in archival bags) such as referencing the name/mimetype/size/checksum/download URL of the original file for ingested files, the inclusion of metadata about the parent collection(s) of an archived dataset version, and use of the URL form of PIDs. +- Display of archival status within the dataset page versions table, richer status options including success, pending, and failure states, with a complete API for managing archival status. +- Support for batch archiving via API as an alternative to the current options of configuring archiving upon publication or archiving each dataset version manually. +- Initial support for sending and receiving Linked Data Notification messages indicating relationships between a dataset and external resources (e.g. papers or other dataset) that can be used to trigger additional actions, such as the creation of a back-link to provide, for example, bi-directional linking between a published paper and a Dataverse dataset. +- A new capability to provide custom per field instructions in dataset templates + +## Major Use Cases and Infrastructure Enhancements + +Changes and fixes in this release include: + +- Administrators can configure an S3 store used in Dataverse to support users uploading/downloading files via Globus File Transfer. (PR #8891) +- Administrators can configure a RemoteOverlay store to allow files that remain hosted by a remote system to be added to a dataset. (PR #7325) +- Administrators can configure the Dataverse software to send archival Bag copies of published dataset versions to any S3-compatible service. (PR #8751) +- Users can see information about a dataset's parent collection(s) in the OAI-ORE metadata export. (PR #8770) +- Users and administrators can now use the OAI-ORE metadata export to retrieve and assess the fixity of the original file (for ingested tabular files) via the included checksum. (PR #8901) +- Archiving via RDA-conformant Bags is more robust and is more configurable. (PR #8773, #8747, #8699, #8609, #8606, #8610) +- Users and administrators can see the archival status of the versions of the datasets they manage in the dataset page version table. (PR #8748, #8696) +- Administrators can configure messaging between their Dataverse installation and other repositories that may hold related resources or services interested in activity within that installation. (PR #8775) +- Collection managers can create templates that include custom instructions on how to fill out specific metadata fields. + +## Notes for Dataverse Installation Administrators + +### Enabling Experimental Capabilities + +Several of the capabilities introduced in v5.12 are "experimental" in the sense that further changes and enhancements to these capabilities should be expected and that these changes may involve additional work, for those who use the initial implementations, when upgrading to newer versions of the Dataverse software. Administrators wishing to use them are encouraged to stay in touch, e.g. via the Dataverse Community Slack space, to understand the limits of current capabilities and to plan for future upgrades. + +## New JVM Options and DB Settings + +The following DB settings have been added: + +- `:LDNMessageHosts` +- `:GlobusBasicToken` +- `:GlobusEndpoint` +- `:GlobusStores` +- `:GlobusAppUrl` +- `:GlobusPollingInterval` +- `:GlobusSingleFileTransfer` +- `:S3ArchiverConfig` +- `:S3ArchiverProfile` +- `:DRSArchiverConfig` + +See the [Database Settings](https://guides.dataverse.org/en/5.12/installation/config.html#database-settings) section of the Guides for more information. + +## Notes for Developers and Integrators + +See the "Backward Incompatibilities" section below. + +## Backward Incompatibilities + +### OAI-ORE and Archiving Changes + +The Admin API call to manually sumbit a dataset version for archiving has changed to require POST instead of GET and to have a name making it clearer that archiving is being done for a given dataset version: /api/admin/submitDatasetVersionToArchive. + +Earlier versions of the archival bags included the ingested (tab-separated-value) version of tabular files while providing the checksum of the original file (Issue #8449). This release fixes that by including the original file and its metadata in the archival bag. This means that archival bags created prior to this version do not include a way to validate ingested files. Further, it is likely that capabilities in development (i.e. as part of the [Dataverse Uploader](https://github/org/GlobalDataverseCommunityConsortium/dataverse-uploader) to allow re-creation of a dataset version from an archival bag will only be fully compatible with archival bags generated by a Dataverse instance at a release > v5.12. (Specifically, at a minimum, since only the ingested file is included in earlier archival bags, an upload via DVUploader would not result in the same original file/ingested version as in the original dataset.) Administrators should be aware that re-creating archival bags, i.e. via the new batch archiving API, may be advisable now and will be recommended at some point in the future (i.e. there will be a point where we will start versioning archival bags and will start maintaining backward compatibility for older versions as part of transitioning this from being an experimental capability). + +## Complete List of Changes + +## Installation + +If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.12/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.12/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already. + +## Upgrade Instructions + +8\. Re-export metadata files (OAI_ORE is affected by the PRs in these release notes). Optionally, for those using the Dataverse software's BagIt-based archiving, re-archive dataset versions archived using prior versions of the Dataverse software. This will be recommended/required in a future release. + +9\. Standard instructions for reinstalling the citation metadatablock. There are no new fields so Solr changes/reindex aren't needed. This PR just adds an option to the list of publicationIdTypes diff --git a/doc/release-notes/8639-computational-workflow.md b/doc/release-notes/8639-computational-workflow.md index d1f014e4af3..efd5b26e538 100644 --- a/doc/release-notes/8639-computational-workflow.md +++ b/doc/release-notes/8639-computational-workflow.md @@ -1,6 +1,8 @@ +NOTE: These "workflow" changes should be folded into "Harvard Data Commons Additions" in 8611-DataCommons-related-notes.md + ## Adding Computational Workflow Metadata The new Computational Workflow metadata block will allow depositors to effectively tag datasets as computational workflows. To add the new metadata block, follow the instructions in the user guide: -The location of the new metadata block tsv file is: `dataverse/scripts/api/data/metadatablocks/computational_workflow.tsv` \ No newline at end of file +The location of the new metadata block tsv file is: `dataverse/scripts/api/data/metadatablocks/computational_workflow.tsv` diff --git a/doc/release-notes/8715-importddi-termofuse.md b/doc/release-notes/8715-importddi-termofuse.md new file mode 100644 index 00000000000..3c6479b8bf9 --- /dev/null +++ b/doc/release-notes/8715-importddi-termofuse.md @@ -0,0 +1 @@ +Terms of Use is now imported when using DDI format through harvesting or the native API. (Issue #8715, PR #8743) diff --git a/doc/release-notes/8759-add-computational-worflow-file-types.md b/doc/release-notes/8759-add-computational-worflow-file-types.md index fa2fd3d001c..d2db860fe5f 100644 --- a/doc/release-notes/8759-add-computational-worflow-file-types.md +++ b/doc/release-notes/8759-add-computational-worflow-file-types.md @@ -1,3 +1,5 @@ +NOTE: These "workflow" changes should be folded into "Harvard Data Commons Additions" in 8611-DataCommons-related-notes.md + The following file extensions are now detected: wdl=text/x-workflow-description-language diff --git a/doc/release-notes/8868-fix-json-import.md b/doc/release-notes/8868-fix-json-import.md new file mode 100644 index 00000000000..de0366e395e --- /dev/null +++ b/doc/release-notes/8868-fix-json-import.md @@ -0,0 +1,7 @@ +Under "bug fixes": + +Small bugs have been fixed in the dataset export in the JSON and DDI formats; eliminating the export of "undefined" as a metadata language in the former, and a duplicate keyword tag in the latter. + +Run ReExportall to update Exports + +Following the directions in the [Admin Guide](http://guides.dataverse.org/en/5.12/admin/metadataexport.html#batch-exports-through-the-api) diff --git a/doc/release-notes/8882-shib-affiliation.md b/doc/release-notes/8882-shib-affiliation.md new file mode 100644 index 00000000000..97d27aa22cc --- /dev/null +++ b/doc/release-notes/8882-shib-affiliation.md @@ -0,0 +1,4 @@ +## New DB Settings +The following DB settings have been added: +- `:ShibAffiliationOrder` - Select the first or last entry in an Affiliation array +- `:ShibAffiliationSeparator` (default: ";") - Set the separator for the Affiliation array diff --git a/doc/sphinx-guides/source/_static/api/dataverse-facets.json b/doc/sphinx-guides/source/_static/api/dataverse-facets.json new file mode 100644 index 00000000000..20a8412440d --- /dev/null +++ b/doc/sphinx-guides/source/_static/api/dataverse-facets.json @@ -0,0 +1 @@ +["authorName", "authorAffiliation"] \ No newline at end of file diff --git a/doc/sphinx-guides/source/_static/api/ddi_dataset.xml b/doc/sphinx-guides/source/_static/api/ddi_dataset.xml index 79e0581131e..05eaadc3458 100644 --- a/doc/sphinx-guides/source/_static/api/ddi_dataset.xml +++ b/doc/sphinx-guides/source/_static/api/ddi_dataset.xml @@ -142,6 +142,7 @@ Terms of Access + Terms of Use Data Access Place Original Archive diff --git a/doc/sphinx-guides/source/_static/api/metadata-block-facets.json b/doc/sphinx-guides/source/_static/api/metadata-block-facets.json new file mode 100644 index 00000000000..bc497846592 --- /dev/null +++ b/doc/sphinx-guides/source/_static/api/metadata-block-facets.json @@ -0,0 +1 @@ +["socialscience", "geospatial"] \ No newline at end of file diff --git a/doc/sphinx-guides/source/api/index.rst b/doc/sphinx-guides/source/api/index.rst index 9fc58ef4e5a..c9e79098546 100755 --- a/doc/sphinx-guides/source/api/index.rst +++ b/doc/sphinx-guides/source/api/index.rst @@ -21,5 +21,6 @@ API Guide client-libraries external-tools curation-labels + linkeddatanotification apps faq diff --git a/doc/sphinx-guides/source/api/linkeddatanotification.rst b/doc/sphinx-guides/source/api/linkeddatanotification.rst new file mode 100644 index 00000000000..d55dc4da084 --- /dev/null +++ b/doc/sphinx-guides/source/api/linkeddatanotification.rst @@ -0,0 +1,65 @@ +Linked Data Notification API +============================ + +Dataverse has a limited, experimental API implementing a Linked Data Notification inbox allowing it to receive messages indicating a link between an external resource and a Dataverse dataset. +The motivating use case is to support a use case where Dataverse administrators may wish to create back-links to the remote resource (e.g. as a Related Publication, Related Material, etc.). + +Upon receipt of a relevant message, Dataverse will create Announcement Received notifications for superusers, who can edit the dataset involved. (In the motivating use case, these users may then add an appropriate relationship and use the Update Curent Version publishing option to add it to the most recently published version of the dataset.) + +The ``:LDNMessageHosts`` setting is a comma-separated whitelist of hosts from which Dataverse will accept and process messages. By default, no hosts are allowed. ``*`` can be used in testing to indicate all hosts are allowed. + +Messages can be sent via POST, using the application/ld+json ContentType: + +.. code-block:: bash + + export SERVER_URL=https://demo.dataverse.org + + curl -X POST -H 'ContentType:application/ld+json' $SERVER_URL/api/inbox --upload-file message.jsonld + +The supported message format is described by `our preliminary specification `_. The format is expected to change in the near future to match the standard for relationship announcements being developed as part of `the COAR Notify Project `_. + +An example message is shown below. It indicates that a resource with the name "An Interesting Title" exists and "IsSupplementedBy" the dataset with DOI https://doi.org/10.5072/FK2/GGCCDL. If this dataset is managed in the receiving Dataverse, a notification will be sent to user with the relevant permissions (as described above). + +.. code:: json + + { + "@context": [ + "https://www.w3.org/ns/activitystreams", + "https://purl.org/coar/notify" + ], + "id": "urn:uuid:94ecae35-dcfd-4182-8550-22c7164fe23f", + "actor": { + "id": "https://research-organisation.org/dspace", + "name": "DSpace Repository", + "type": "Service" + }, + "context": { + "IsSupplementedBy": + { + "id": "http://dev-hdc3b.lib.harvard.edu/dataset.xhtml?persistentId=doi:10.5072/FK2/GGCCDL", + "ietf:cite-as": "https://doi.org/10.5072/FK2/GGCCDL", + "type": "sorg:Dataset" + } + }, + "object": { + "id": "https://research-organisation.org/dspace/item/35759679-5df3-4633-b7e5-4cf24b4d0614", + "ietf:cite-as": "https://research-organisation.org/authority/resolve/35759679-5df3-4633-b7e5-4cf24b4d0614", + "sorg:name": "An Interesting Title", + "type": "sorg:ScholarlyArticle" + }, + "origin": { + "id": "https://research-organisation.org/dspace", + "inbox": "https://research-organisation.org/dspace/inbox/", + "type": "Service" + }, + "target": { + "id": "https://research-organisation.org/dataverse", + "inbox": "https://research-organisation.org/dataverse/inbox/", + "type": "Service" + }, + "type": [ + "Announce", + "coar-notify:ReleaseAction" + ] + } + diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index 75eb7b5424e..339a291bf4d 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -219,15 +219,82 @@ Assign search facets for a given Dataverse collection identified by ``id``: export SERVER_URL=https://demo.dataverse.org export ID=root - curl -H X-Dataverse-key:$API_TOKEN" -X POST $SERVER_URL/api/dataverses/$ID/facets --upload-file facets.json + curl -H X-Dataverse-key:$API_TOKEN" -X POST $SERVER_URL/api/dataverses/$ID/facets --upload-file dataverse-facets.json The fully expanded example above (without environment variables) looks like this: .. code-block:: bash - curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST https://demo.dataverse.org/api/dataverses/root/facets --upload-file facets.json + curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST https://demo.dataverse.org/api/dataverses/root/facets --upload-file dataverse-facets.json -Where ``facets.json`` contains a JSON encoded list of metadata keys (e.g. ``["authorName","authorAffiliation"]``). +Where :download:`dataverse-facets.json <../_static/api/dataverse-facets.json>` contains a JSON encoded list of metadata keys (e.g. ``["authorName","authorAffiliation"]``). + +List Metadata Block Facets Configured for a Dataverse Collection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +|CORS| List the metadata block facet configuration with all the metadata block configured for a given Dataverse collection ``id``: + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export ID=root + + curl -H X-Dataverse-key:$API_TOKEN $SERVER_URL/api/dataverses/$ID/metadatablockfacets + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/dataverses/root/metadatablockfacets + +Set Metadata Block Facets for a Dataverse Collection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sets the metadata blocks that will appear in the ``Dataset Features`` facet category for a given Dataverse collection identified by ``id``. + +In order to set or clear the metadata blocks for a collection, you must first :ref:`set the metadata block facet root to true`. + +To clear the metadata blocks set by a parent collection, submit an empty array (e.g. ``[]``): + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export ID=root + + curl -H X-Dataverse-key:$API_TOKEN" -X POST -H "Content-type:application/json" $SERVER_URL/api/dataverses/$ID/metadatablockfacets --upload-file metadata-block-facets.json + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/metadatablockfacets --upload-file metadata-block-facets.json + +Where :download:`metadata-block-facets.json <../_static/api/metadata-block-facets.json>` contains a JSON encoded list of metadata block names (e.g. ``["socialscience","geospatial"]``). This endpoint supports an empty list (e.g. ``[]``) + +.. _metadata-block-facet-root-api: + +Configure a Dataverse Collection to Inherit Its Metadata Block Facets from Its Parent +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set whether the Dataverse collection is a metadata block facet root, or does it uses its parent metadata block facets. Possible values are ``true`` and ``false`` (both are valid JSON expressions). + +When updating the root to false, it will clear any metadata block facets from the collection. When updating to true, it will copy the metadata block facets from the parent collection: + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export ID=root + + curl -H X-Dataverse-key:$API_TOKEN -X POST -H "Content-type:application/json" $SERVER_URL/api/dataverses/$ID/metadatablockfacets/isRoot -d 'true' + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/metadatablockfacets/isRoot -d 'true' Create a New Role in a Dataverse Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1411,7 +1478,37 @@ In practice, you only need one the ``dataset_id`` or the ``persistentId``. The e print '-' * 40 print r.json() print r.status_code + +.. _add-remote-file-api: +Add a Remote File to a Dataset +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If your Dataverse installation has been configured to support :ref:`trusted-remote-storage` +you can add files from remote URLs to datasets. These remote files appear in your Dataverse +installation as if they were ordinary files but are stored remotely. + +The location of the remote file is specified in the ``storageIdentifier`` field in JSON you supply. +The base URL of the file is contained in the "store" (e.g. "trsa" in the example below) and is followed by the +path to the file (e.g. "themes/custom..."). + +In the JSON example below, all fields are required except for ``description``. Other optional fields are shown under :ref:`add-file-api`. + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_ID=doi:10.5072/FK2/J8SJZB + export JSON_DATA='{"description":"A remote image.","storageIdentifier":"trsa://themes/custom/qdr/images/CoreTrustSeal-logo-transparent.png","checksumType":"MD5","md5Hash":"509ef88afa907eaf2c17c1c8d8fde77e","label":"testlogo.png","fileName":"testlogo.png","mimeType":"image/png"}' + + curl -H "X-Dataverse-key: $API_TOKEN" -X POST "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID" -F "jsonData=$JSON_DATA" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST https://demo.dataverse.org/api/datasets/:persistentId/add?persistentId=doi:10.5072/FK2/J8SJZB -F 'jsonData={"description":"A remote image.","storageIdentifier":"trsa://themes/custom/qdr/images/CoreTrustSeal-logo-transparent.png","checksumType":"MD5","md5Hash":"509ef88afa907eaf2c17c1c8d8fde77e","label":"testlogo.png","fileName":"testlogo.png","mimeType":"image/png"}' + Report the data (file) size of a Dataset ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/developers/big-data-support.rst b/doc/sphinx-guides/source/developers/big-data-support.rst index 641307372ee..0782fd239a1 100644 --- a/doc/sphinx-guides/source/developers/big-data-support.rst +++ b/doc/sphinx-guides/source/developers/big-data-support.rst @@ -1,19 +1,19 @@ Big Data Support ================ -Big data support is highly experimental. Eventually this content will move to the Installation Guide. +Big data support includes some highly experimental options. Eventually more of this content will move to the Installation Guide. .. contents:: |toctitle| :local: -Various components need to be installed and/or configured for big data support. +Various components will need to be installed and/or configured for big data support via the methods described below. S3 Direct Upload and Download ----------------------------- A lightweight option for supporting file sizes beyond a few gigabytes - a size that can cause performance issues when uploaded through a Dataverse installation itself - is to configure an S3 store to provide direct upload and download via 'pre-signed URLs'. When these options are configured, file uploads and downloads are made directly to and from a configured S3 store using secure (https) connections that enforce a Dataverse installation's access controls. (The upload and download URLs are signed with a unique key that only allows access for a short time period and a Dataverse installation will only generate such a URL if the user has permission to upload/download the specific file in question.) -This option can handle files >40GB and could be appropriate for files up to a TB. Other options can scale farther, but this option has the advantages that it is simple to configure and does not require any user training - uploads and downloads are done via the same interface as normal uploads to a Dataverse installation. +This option can handle files >300GB and could be appropriate for files up to a TB or larger. Other options can scale farther, but this option has the advantages that it is simple to configure and does not require any user training - uploads and downloads are done via the same interface as normal uploads to a Dataverse installation. To configure these options, an administrator must set two JVM options for the Dataverse installation using the same process as for other configuration options: @@ -32,7 +32,7 @@ For AWS, the minimum allowed part size is 5*1024*1024 bytes and the maximum is 5 It is also possible to set file upload size limits per store. See the :MaxFileUploadSizeInBytes setting described in the :doc:`/installation/config` guide. -At present, one potential drawback for direct-upload is that files are only partially 'ingested', tabular and FITS files are processed, but zip files are not unzipped, and the file contents are not inspected to evaluate their mimetype. This could be appropriate for large files, or it may be useful to completely turn off ingest processing for performance reasons (ingest processing requires a copy of the file to be retrieved by the Dataverse installation from the S3 store). A store using direct upload can be configured to disable all ingest processing for files above a given size limit: +At present, one potential drawback for direct-upload is that files are only partially 'ingested' - tabular and FITS files are processed, but zip files are not unzipped, and the file contents are not inspected to evaluate their mimetype. This could be appropriate for large files, or it may be useful to completely turn off ingest processing for performance reasons (ingest processing requires a copy of the file to be retrieved by the Dataverse installation from the S3 store). A store using direct upload can be configured to disable all ingest processing for files above a given size limit: ``./asadmin create-jvm-options "-Ddataverse.files..ingestsizelimit="`` @@ -61,6 +61,93 @@ Alternatively, you can enable CORS using the AWS S3 web interface, using json-en Since the direct upload mechanism creates the final file rather than an intermediate temporary file, user actions, such as neither saving or canceling an upload session before closing the browser page, can leave an abandoned file in the store. The direct upload mechanism attempts to use S3 Tags to aid in identifying/removing such files. Upon upload, files are given a "dv-state":"temp" tag which is removed when the dataset changes are saved and the new file(s) are added in the Dataverse installation. Note that not all S3 implementations support Tags: Minio does not. WIth such stores, direct upload works, but Tags are not used. +Trusted Remote Storage with the ``remote`` Store Type +----------------------------------------------------- + +For very large, and/or very sensitive data, it may not make sense to transfer or copy files to Dataverse at all. The experimental ``remote`` store type in the Dataverse software now supports this use case. + +With this storage option Dataverse stores a URL reference for the file rather than transferring the file bytes to a store managed directly by Dataverse. Basic configuration for a remote store is described at :ref:`file-storage` in the Configuration Guide. + +Once the store is configured, it can be assigned to a collection or individual datasets as with other stores. In a dataset using this store, users can reference remote files which will then appear the same basic way as other datafiles. + +Currently, remote files can only be added via the API. Users can also upload smaller files via the UI or API which will be stored in the configured base store. + +If the store has been configured with a remote-store-name or remote-store-url, the dataset file table will include this information for remote files. These provide a visual indicator that the files are not managed directly by Dataverse and are stored/managed by a remote trusted store. + +Rather than sending the file bytes, metadata for the remote file is added using the "jsonData" parameter. +jsonData normally includes information such as a file description, tags, provenance, whether the file is restricted, etc. For remote references, the jsonData object must also include values for: + +* "storageIdentifier" - String, as specified in prior calls +* "fileName" - String +* "mimeType" - String +* fixity/checksum: either: + + * "md5Hash" - String with MD5 hash value, or + * "checksum" - Json Object with "@type" field specifying the algorithm used and "@value" field with the value from that algorithm, both Strings + +The allowed checksum algorithms are defined by the edu.harvard.iq.dataverse.DataFile.CheckSumType class and currently include MD5, SHA-1, SHA-256, and SHA-512 + +(The remote store leverages the same JSON upload syntax as the last step in direct upload to S3 described in the :ref:`Adding the Uploaded file to the Dataset ` section of the :doc:`/developers/s3-direct-upload-api`.) + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_IDENTIFIER=doi:10.5072/FK27U7YBV + export JSON_DATA="{'description':'My description.','directoryLabel':'data/subdir1','categories':['Data'], 'restrict':'false', 'storageIdentifier':'trs://images/dataverse_project_logo.svg', 'fileName':'dataverse_logo.svg', 'mimeType':'image/svg+xml', 'checksum': {'@type': 'SHA-1', '@value': '123456'}}" + + curl -X POST -H "X-Dataverse-key: $API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_IDENTIFIER" -F "jsonData=$JSON_DATA" + +The variant allowing multiple files to be added once that is discussed in the :doc:`/developers/s3-direct-upload-api` document can also be used. + +Considerations: + +* Remote stores are configured with a base-url which limits what files can be referenced, i.e. the absolute URL for the file is /. +* The current store will not prevent you from providing a relative URL that results in a 404 when resolved. (I.e. if you make a typo). You should check to make sure the file exists at the location you specify - by trying to download in Dataverse, by checking to see that Dataverse was able to get the file size (which it does by doing a HEAD call to that location), or just manually trying the URL in your browser. +* Admins are trusting the organization managing the site/service at base-url to maintain the referenced files for as long as the Dataverse instance needs them. Formal agreements are recommended for production +* For large files, direct-download should always be used with a remote store. (Otherwise the Dataverse will be involved in the download.) +* For simple websites, a remote store should be marked public which will turn off restriction and embargo functionality in Dataverse (since Dataverse cannot restrict access to the file on the remote website) +* Remote stores can be configured with a secret-key. This key will be used to sign URLs when Dataverse retrieves the file content or redirects a user for download. If remote service is able to validate the signature and reject invalid requests, the remote store mechanism can be used to manage restricted and embargoes files, access requests in Dataverse, etc. Dataverse contains Java code that validates these signatures which could be used, for example, to create a validation proxy in front of a web server to allow Dataverse to manage access. The secret-key is a shared secret between Dataverse and the remote service and is not shared with/is not accessible by users or those with access to user's machines. +* Sophisticated remote services may wish to register file URLs that do not directly reference the file contents (bytes) but instead direct the user to a website where further information about the remote service's download process can be found. +* Due to the current design, ingest cannot be done on remote files and administrators should disable ingest when using a remote store. This can be done by setting the ingest size limit for the store to 0 and/or using the recently added option to not perform tabular ingest on upload. +* Dataverse will normally try to access the file contents itself, i.e. for ingest (in future versions), full-text indexing, thumbnail creation, etc. This processing may not be desirable for large/sensitive data, and, for the case where the URL does not reference the file itself, would not be possible. At present, administrators should configure the relevant size limits to avoid such actions. +* The current implementation of remote stores is experimental in the sense that future work to enhance it is planned. This work may result in changes to how the store works and lead to additional work when upgrading for sites that start using this mechanism now. + +To configure the options mentioned above, an administrator must set two JVM options for the Dataverse installation using the same process as for other configuration options: + +``./asadmin create-jvm-options "-Ddataverse.files..download-redirect=true"`` +``./asadmin create-jvm-options "-Ddataverse.files..secret-key=somelongrandomalphanumerickeythelongerthebetter123456"`` +``./asadmin create-jvm-options "-Ddataverse.files..public=true"`` +``./asadmin create-jvm-options "-Ddataverse.files..ingestsizelimit="`` + +.. _globus-support: + +Globus File Transfer +-------------------- + +Note: Globus file transfer is still experimental but feedback is welcome! See :ref:`support`. + +Users can transfer files via `Globus `_ into and out of datasets when their Dataverse installation is configured to use a Globus accessible S3 store and a community-developed `dataverse-globus `_ "transfer" app has been properly installed and configured. + +Due to differences in the access control models of a Dataverse installation and Globus, enabling the Globus capability on a store will disable the ability to restrict and embargo files in that store. + +As Globus aficionados know, Globus endpoints can be in a variety of places, from data centers to personal computers. This means that from within the Dataverse software, a Globus transfer can feel like an upload or a download (with Globus Personal Connect running on your laptop, for example) or it can feel like a true transfer from one server to another (from a cluster in a data center into a Dataverse dataset or vice versa). + +Globus transfer uses a very efficient transfer mechanism and has additional features that make it suitable for large files and large numbers of files: + +* robust file transfer capable of restarting after network or endpoint failures +* third-party transfer, which enables a user accessing a Dataverse installation in their desktop browser to initiate transfer of their files from a remote endpoint (i.e. on a local high-performance computing cluster), directly to an S3 store managed by the Dataverse installation + +Globus transfer requires use of the Globus S3 connector which requires a paid Globus subscription at the host institution. Users will need a Globus account which could be obtained via their institution or directly from Globus (at no cost). + +The setup required to enable Globus is described in the `Community Dataverse-Globus Setup and Configuration document `_ and the references therein. + +As described in that document, Globus transfers can be initiated by choosing the Globus option in the dataset upload panel. (Globus, which does asynchronous transfers, is not available during dataset creation.) Analogously, "Globus Transfer" is one of the download options in the "Access Dataset" menu and optionally the file landing page download menu (if/when supported in the dataverse-globus app). + +An overview of the control and data transfer interactions between components was presented at the 2022 Dataverse Community Meeting and can be viewed in the `Integrations and Tools Session Video `_ around the 1 hr 28 min mark. + +See also :ref:`Globus settings <:GlobusBasicToken>`. + Data Capture Module (DCM) ------------------------- diff --git a/doc/sphinx-guides/source/developers/configuration.rst b/doc/sphinx-guides/source/developers/configuration.rst index 0eac7de3134..fb15fea7900 100644 --- a/doc/sphinx-guides/source/developers/configuration.rst +++ b/doc/sphinx-guides/source/developers/configuration.rst @@ -18,12 +18,14 @@ authentication providers, harvesters and others. Simple Configuration Options ---------------------------- -Developers have accessed the simple properties via +Developers can access simple properties via: -1. ``System.getProperty(...)`` for JVM system property settings -2. ``SettingsServiceBean.get(...)`` for database settings and +1. ``JvmSettings..lookup(...)`` for JVM system property settings. +2. ``SettingsServiceBean.get(...)`` for database settings. 3. ``SystemConfig.xxx()`` for specially treated settings, maybe mixed from 1 and 2 and other sources. -4. ``SettingsWrapper`` must be used to obtain settings from 2 and 3 in frontend JSF (xhtml) pages. Please see the note on how to :ref:`avoid common efficiency issues with JSF render logic expressions `. +4. ``SettingsWrapper`` for use in frontend JSF (xhtml) pages to obtain settings from 2 and 3. Using the wrapper is a must for performance as explained in :ref:`avoid common efficiency issues with JSF render logic expressions + `. +5. ``System.getProperty()`` only for very special use cases not covered by ``JvmSettings``. As of Dataverse Software 5.3, we start to streamline our efforts into using a more consistent approach, also bringing joy and happiness to all the system administrators out there. This will be done by adopting the use of @@ -49,6 +51,7 @@ Developers benefit from: - Config API is also pushing for validation of configuration, as it's typesafe and converters for non-standard types can be added within our codebase. - Defaults in code or bundled in ``META-INF/microprofile-config.properties`` allow for optional values without much hassle. +- A single place to lookup any existing JVM setting in code, easier to keep in sync with the documentation. System administrators benefit from: @@ -57,9 +60,9 @@ System administrators benefit from: - Running a Dataverse installation in containers gets much easier when configuration can be provisioned in a streamlined fashion, mitigating the need for scripting glue and distinguishing between setting types. - Classic installations have a profit, too: we can enable using a single config file, e.g. living in - ``/etc/dataverse/config.properties``. + ``/etc/dataverse/config.properties`` by adding our own, hot-reload config source. - Features for monitoring resources and others are easier to use with this streamlined configuration, as we can - avoid people having to deal with ``asadmin`` commands and change a setting comfortably instead. + avoid people having to deal with ``asadmin`` commands and change a setting with comfort instead. Adopting MicroProfile Config API --------------------------------- @@ -68,33 +71,41 @@ This technology is introduced on a step-by-step basis. There will not be a big s Instead, we will provide backward compatibility by deprecating renamed or moved config options, while still supporting the old way of setting them. -- Introducing a new setting or moving and old one should result in a key ``dataverse..``. - That way we enable sys admins to recognize the meaning of an option and avoid name conflicts. +- Introducing a new setting or moving an old one should result in a scoped key + ``dataverse..``. That way we enable sys admins to recognize the meaning of an option + and avoid name conflicts. Starting with ``dataverse`` makes it perfectly clear that this is a setting meant for this application, which is important when using environment variables, system properties or other MPCONFIG sources. -- Replace ``System.getProperty()`` calls with either injected configs or retrieve programmatically if more complex - handling is necessary. If you rename the property, you should provide an alias. See below. -- Database settings need to be refactored in multiple steps. First you need to change the code retrieving it to use - MicroProfile Config API instead (just like above). Then you should provide an alias to retain backward compatibility. - See below. +- Replace ``System.getProperty()`` calls with ``JvmSettings..lookup(...)``, adding the setting there first. + This might be paired with renaming and providing backward-compatible aliases. +- Database settings need to be refactored in multiple steps and it is not yet clear how this will be done. + Many Database settings are of very static nature and might be moved to JVM settings (in backward compatible ways). -Moving or Replacing a JVM Setting -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Adding a JVM Setting +^^^^^^^^^^^^^^^^^^^^ -When moving an old key to a new (especially when doing so with a former JVM system property setting), you should -add an alias to ``src/main/resources/META-INF/microprofile-aliases.properties`` to enable backward compatibility. -The format is always like ``dataverse..newname...=old.property.name``. +Whenever a new option gets added or an existing configuration gets migrated to +``edu.harvard.iq.dataverse.settings.JvmSettings``, you will attach the setting to an existing scope or create new +sub-scopes first. -Details can be found in ``edu.harvard.iq.dataverse.settings.source.AliasConfigSource`` +- Scopes and settings are organised in a tree-like structure within a single enum ``JvmSettings``. +- The root scope is "dataverse". +- All sub-scopes are below that. +- Scopes are separated by dots (periods). +- A scope may be a placeholder, filled with a variable during lookup. (Named object mapping.) -Aliasing Database Setting -^^^^^^^^^^^^^^^^^^^^^^^^^ +Any consumer of the setting can choose to use one of the fluent ``lookup()`` methods, which hides away alias handling, +conversion etc from consuming code. See also the detailed Javadoc for these methods. -When moving a database setting (``:ExampleSetting``), configure an alias -``dataverse.my.example.setting=dataverse.settings.fromdb.ExampleSetting`` in -``src/main/resources/META-INF/microprofile-aliases.properties``. This will enable backward compatibility. +Moving or Replacing a JVM Setting +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When moving an old key to a new (especially when doing so with a former JVM system property setting), you should +add an alias to the ``JvmSettings`` definition to enable backward compatibility. Old names given there are capable of +being used with patterned lookups. -A database setting with an i18n attribute using *lang* will have available language codes appended to the name. -Example: ``dataverse.settings.fromdb.ExampleI18nSetting.en``, ``dataverse.settings.fromdb.ExampleI18nSetting.de`` +Another option is to add the alias in ``src/main/resources/META-INF/microprofile-aliases.properties``. The format is +always like ``dataverse..newname...=old.property.name``. Note this doesn't provide support for patterned +aliases. -More details in ``edu.harvard.iq.dataverse.settings.source.DbSettingConfigSource`` +Details can be found in ``edu.harvard.iq.dataverse.settings.source.AliasConfigSource`` diff --git a/doc/sphinx-guides/source/developers/s3-direct-upload-api.rst b/doc/sphinx-guides/source/developers/s3-direct-upload-api.rst index d1a71c313ca..3dc73ce6a0c 100644 --- a/doc/sphinx-guides/source/developers/s3-direct-upload-api.rst +++ b/doc/sphinx-guides/source/developers/s3-direct-upload-api.rst @@ -88,6 +88,8 @@ If the client is unable to complete the multipart upload, it should call the abo curl -X DELETE "$SERVER_URL/api/datasets/mpload?..." +.. _direct-add-to-dataset-api: + Adding the Uploaded file to the Dataset --------------------------------------- @@ -117,7 +119,7 @@ Note that this API call can be used independently of the others, e.g. supporting With current S3 stores the object identifier must be in the correct bucket for the store, include the PID authority/identifier of the parent dataset, and be guaranteed unique, and the supplied storage identifer must be prefaced with the store identifier used in the Dataverse installation, as with the internally generated examples above. To add multiple Uploaded Files to the Dataset -------------------------------------------------- +--------------------------------------------- Once the files exists in the s3 bucket, a final API call is needed to add all the files to the Dataset. In this API call, additional metadata is added using the "jsonData" parameter. jsonData normally includes information such as a file description, tags, provenance, whether the file is restricted, etc. For direct uploads, the jsonData object must also include values for: diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 7bde4055e33..4b3d5fd0a55 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -79,6 +79,22 @@ greatly extended parameterized testing. Some guidance how to write those: - https://blog.codefx.org/libraries/junit-5-parameterized-tests/ - See also some examples in our codebase. +JUnit 5 Test Helper Extensions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Our codebase provides little helpers to ease dealing with state during tests. +Some tests might need to change something which should be restored after the test ran. + +For unit tests, the most interesting part is to set a JVM setting just for the current test. +Please use the ``@JvmSetting(key = JvmSettings.XXX, value = "")`` annotation on a test method or +a test class to set and clear the property automatically. + +To set arbitrary system properties for the current test, a similar extension +``@SystemProperty(key = "", value = "")`` has been added. + +Both extensions will ensure the global state of system properties is non-interfering for +test executions. Tests using these extensions will be executed in serial. + Observing Changes to Code Coverage ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/sphinx-guides/source/developers/workflows.rst b/doc/sphinx-guides/source/developers/workflows.rst index df63bf239fe..38ca6f4e141 100644 --- a/doc/sphinx-guides/source/developers/workflows.rst +++ b/doc/sphinx-guides/source/developers/workflows.rst @@ -201,3 +201,31 @@ Note - the example step includes two settings required for any archiver, three ( } } + +ldnannounce ++++++++++++ + +An experimental step that sends a Linked Data Notification (LDN) message to a specific LDN Inbox announcing the publication/availability of a dataset meeting certain criteria. + +The two parameters are +* ``:LDNAnnounceRequiredFields`` - a list of metadata fields that must exist to trigger the message. Currently, the message also includes the values for these fields but future versions may only send the dataset's persistent identifier (making the receiver responsible for making a call-back to get any metadata). +* ``:LDNTarget`` - a JSON object containing an ``inbox`` key whose value is the URL of the target LDN inbox to which messages should be sent, e.g. ``{"id": "https://dashv7-dev.lib.harvard.edu","inbox": "https://dashv7-api-dev.lib.harvard.edu/server/ldn/inbox","type": "Service"}`` ). + +The supported message format is desribed by `our preliminary specification `_. The format is expected to change in the near future to match the standard for relationship announcements being developed as part of `the COAR Notify Project `_. + + +.. code:: json + + + { + "provider":":internal", + "stepType":"ldnannounce", + "parameters": { + "stepName":"LDN Announce" + }, + "requiredSettings": { + ":LDNAnnounceRequiredFields": "string", + ":LDNTarget": "string" + } + } + diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 04c17298a97..0b6e4e7cf4e 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -385,13 +385,17 @@ Logging & Slow Performance -File Storage: Using a Local Filesystem and/or Swift and/or object stores ------------------------------------------------------------------------- +.. _file-storage: + +File Storage: Using a Local Filesystem and/or Swift and/or Object Stores and/or Trusted Remote Stores +----------------------------------------------------------------------------------------------------- By default, a Dataverse installation stores all data files (files uploaded by end users) on the filesystem at ``/usr/local/payara5/glassfish/domains/domain1/files``. This path can vary based on answers you gave to the installer (see the :ref:`dataverse-installer` section of the Installation Guide) or afterward by reconfiguring the ``dataverse.files.\.directory`` JVM option described below. A Dataverse installation can alternately store files in a Swift or S3-compatible object store, and can now be configured to support multiple stores at once. With a multi-store configuration, the location for new files can be controlled on a per-Dataverse collection basis. +A Dataverse installation may also be configured to reference some files (e.g. large and/or sensitive data) stored in a web-accessible trusted remote store. + The following sections describe how to set up various types of stores and how to configure for multiple stores. Multi-store Basics @@ -810,6 +814,40 @@ Migrating from Local Storage to S3 Is currently documented on the :doc:`/developers/deployment` page. +.. _trusted-remote-storage: + +Trusted Remote Storage +++++++++++++++++++++++ + +In addition to having the type "remote" and requiring a label, Trusted Remote Stores are defined in terms of a baseURL - all files managed by this store must be at a path starting with this URL, and a baseStore - a file, s3, or swift store that can be used to store additional ancillary dataset files (e.g. metadata exports, thumbnails, auxiliary files, etc.). +These and other available options are described in the table below. + +Trusted remote stores can range from being a static trusted website to a sophisticated service managing access requests and logging activity +and/or managing access to a secure enclave. See :doc:`/developers/big-data-support` for additional information on how to use a trusted remote store. For specific remote stores, consult their documentation when configuring the remote store in your Dataverse installation. + +Note that in the current implementation, activites where Dataverse needs access to data bytes, e.g. to create thumbnails or validate hash values at publication will fail if a remote store does not allow Dataverse access. Implementers of such trusted remote stores should consider using Dataverse's settings to disable ingest, validation of files at publication, etc. as needed. + +Once you have configured a trusted remote store, you can point your users to the :ref:`add-remote-file-api` section of the API Guide. + +.. table:: + :align: left + + =========================================== ================== ========================================================================== =================== + JVM Option Value Description Default value + =========================================== ================== ========================================================================== =================== + dataverse.files..type ``remote`` **Required** to mark this storage as remote. (none) + dataverse.files..label **Required** label to be shown in the UI for this storage. (none) + dataverse.files..base-url **Required** All files must have URLs of the form /* . (none) + dataverse.files..base-store **Optional** The id of a base store (of type file, s3, or swift). (the default store) + dataverse.files..download-redirect ``true``/``false`` Enable direct download (should usually be true). ``false`` + dataverse.files..secret-key A key used to sign download requests sent to the remote store. Optional. (none) + dataverse.files..url-expiration-minutes If direct downloads and using signing: time until links expire. Optional. 60 + dataverse.files..remote-store-name A short name used in the UI to indicate where a file is located. Optional. (none) + dataverse.files..remote-store-url A url to an info page about the remote store used in the UI. Optional. (none) + + =========================================== ================== ========================================================================== =================== + + .. _Branding Your Installation: @@ -1228,7 +1266,7 @@ These archival Bags include all of the files and metadata in a given dataset ver The Dataverse Software offers an internal archive workflow which may be configured as a PostPublication workflow via an admin API call to manually submit previously published Datasets and prior versions to a configured archive such as Chronopolis. The workflow creates a `JSON-LD `_ serialized `OAI-ORE `_ map file, which is also available as a metadata export format in the Dataverse Software web interface. -At present, archiving classes include the DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchive, and S3SubmitToArchiveCommand , which all extend the AbstractSubmitToArchiveCommand and use the configurable mechanisms discussed below. +At present, archiving classes include the DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchive, and S3SubmitToArchiveCommand , which all extend the AbstractSubmitToArchiveCommand and use the configurable mechanisms discussed below. (A DRSSubmitToArchiveCommand, which works with Harvard's DRS also exists and, while specific to DRS, is a useful example of how Archivers can support single-version-only semantics and support archiving only from specified collections (with collection specific parameters)). All current options support the archival status APIs and the same status is available in the dataset page version table (for contributors/those who could view the unpublished dataset, with more detail available to superusers). @@ -1332,7 +1370,7 @@ The S3 Archiver defines one custom setting, a required :S3ArchiverConfig. It can The credentials for your S3 account, can be stored in a profile in a standard credentials file (e.g. ~/.aws/credentials) referenced via "profile" key in the :S3ArchiverConfig setting (will default to the default entry), or can via MicroProfile settings as described for S3 stores (dataverse.s3archiver.access-key and dataverse.s3archiver.secret-key) -The :S3ArchiverConfig setting is a json object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", "payload-signing", and "chunked-encoding". +The :S3ArchiverConfig setting is a JSON object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", "payload-signing", and "chunked-encoding". \:S3ArchiverConfig - minimally includes the name of the bucket to use. For example: @@ -2514,6 +2552,30 @@ In the case you get garbled characters in Shibboleth-supplied fields (e.g. given If you managed to get correct accented characters from shibboleth while this setting is _false_, please contact us with your application server and Shibboleth configuration! +:ShibAffiliationOrder ++++++++++++++++++++++ + +Will select the last or first value of an array in affiliation, the array separator can be set using ``:ShibAffiliationSeparator`` . + +To select the last value : + +``curl -X PUT -d "lastAffiliation" http://localhost:8080/api/admin/settings/:ShibAffiliationOrder`` + +To select the first value : + +``curl -X PUT -d "firstAffiliation" http://localhost:8080/api/admin/settings/:ShibAffiliationOrder`` + + +:ShibAffiliationSeparator ++++++++++++++++++++++++++ + +Set the separator to be used for ``:ShibAffiliationOrder``. +Default separator : ";" + +To change the separator : + +``curl -X PUT -d ";" http://localhost:8080/api/admin/settings/:ShibAffiliationSeparator`` + .. _:ComputeBaseUrl: :ComputeBaseUrl @@ -3000,3 +3062,51 @@ For configuration details, see :ref:`mute-notifications`. Overrides the default empty list of never muted notifications. Never muted notifications cannot be muted by the users. Always muted notifications are grayed out and are not adjustable by the user. For configuration details, see :ref:`mute-notifications`. + +:LDNMessageHosts +++++++++++++++++ + +The comma-separated list of hosts allowed to send Dataverse Linked Data Notification messages. See :doc:`/api/linkeddatanotification` for details. ``*`` allows messages from anywhere (not recommended for production). By default, messages are not accepted from anywhere. + + +:LDN_TARGET ++++++++++++ + +The URL of an LDN Inbox to which the LDN Announce workflow step will send messages. See :doc:`/developers/workflows` for details. + +:LDNAnnounceRequiredFields +++++++++++++++++++++++++++ + +The list of parent dataset field names for which the LDN Announce workflow step should send messages. See :doc:`/developers/workflows` for details. + +.. _:GlobusBasicToken: + +:GlobusBasicToken ++++++++++++++++++ + +GlobusBasicToken encodes credentials for Globus integration. See :ref:`globus-support` for details. + +:GlobusEndpoint ++++++++++++++++ + +GlobusEndpoint is Globus endpoint id used with Globus integration. See :ref:`globus-support` for details. + +:GlobusStores ++++++++++++++ + +A comma-separated list of the S3 stores that are configured to support Globus integration. See :ref:`globus-support` for details. + +:GlobusAppURL ++++++++++++++ + +The URL where the `dataverse-globus `_ "transfer" app has been deployed to support Globus integration. See :ref:`globus-support` for details. + +:GlobusPollingInterval +++++++++++++++++++++++ + +The interval in seconds between Dataverse calls to Globus to check on upload progress. Defaults to 50 seconds. See :ref:`globus-support` for details. + +:GlobusSingleFileTransfer ++++++++++++++++++++++++++ + +A true/false option to add a Globus transfer option to the file download menu which is not yet fully supported in the dataverse-globus app. See :ref:`globus-support` for details. diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst index e33c774a33a..c491659cd56 100644 --- a/doc/sphinx-guides/source/installation/prep.rst +++ b/doc/sphinx-guides/source/installation/prep.rst @@ -27,6 +27,7 @@ Advanced Installation There are some community-lead projects to use configuration management tools such as Ansible and Puppet to automate the installation and configuration of the Dataverse Software, but support for these solutions is limited to what the Dataverse Community can offer as described in each project's webpage: - https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible +- https://gitlab.com/lip-computing/dataverse - https://github.com/IQSS/dataverse-puppet (Please note that the "dataverse-ansible" repo is used in a script that allows the Dataverse Software to be installed on Amazon Web Services (AWS) from arbitrary GitHub branches as described in the :doc:`/developers/deployment` section of the Developer Guide.) diff --git a/doc/sphinx-guides/source/user/dataverse-management.rst b/doc/sphinx-guides/source/user/dataverse-management.rst index efe98e8327c..ed90497da8c 100755 --- a/doc/sphinx-guides/source/user/dataverse-management.rst +++ b/doc/sphinx-guides/source/user/dataverse-management.rst @@ -44,7 +44,7 @@ To edit your Dataverse collection, navigate to your Dataverse collection's landi - :ref:`Theme `: upload a logo for your Dataverse collection, add a link to your department or personal website, add a custom footer image, and select colors for your Dataverse collection in order to brand it - :ref:`Widgets `: get code to add to your website to have your Dataverse collection display on it - :ref:`Permissions `: give other users permissions to your Dataverse collection, i.e.-can edit datasets, and see which users already have which permissions for your Dataverse collection -- :ref:`Dataset Templates `: these are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in +- :ref:`Dataset Templates `: these are useful when you want to provide custom instructions on how to fill out fields or have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in - :ref:`Dataset Guestbooks `: allows you to collect data about who is downloading the files from your datasets - :ref:`Featured Dataverse collections `: if you have one or more Dataverse collection, you can use this option to show them at the top of your Dataverse collection page to help others easily find interesting or important Dataverse collections - **Delete Dataverse**: you are able to delete your Dataverse collection as long as it is not published and does not have any draft datasets @@ -52,7 +52,7 @@ To edit your Dataverse collection, navigate to your Dataverse collection's landi .. _general-information: General Information ---------------------- +------------------- The General Information page is how you edit the information you filled in while creating your Dataverse collection. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the Dataverse collection, change which metadata fields are hidden, required, or optional, and update the facets you would like displayed for browsing the Dataverse collection. If you plan on using templates, you need to select the metadata fields on the General Information page. @@ -60,8 +60,8 @@ Tip: The metadata fields you select as required will appear on the Create Datase .. _theme: -Theme ---------- +Theme +----- The Theme features provides you with a way to customize the look of your Dataverse collection. You can: @@ -77,7 +77,7 @@ Supported image types for logo images and footer images are JPEG, TIFF, or PNG a .. _dataverse-widgets: Widgets --------------- +------- The Widgets feature provides you with code for you to put on your personal website to have your Dataverse collection displayed there. There are two types of Widgets for a Dataverse collection, a Dataverse collection Search Box widget and a Dataverse collection Listing widget. Once a Dataverse collection has been published, from the Widgets tab on the Dataverse collection's Theme + Widgets page, it is possible to copy the code snippets for the widget(s) you would like to add to your website. If you need to adjust the height of the widget on your website, you may do so by editing the `heightPx=500` parameter in the code snippet. @@ -94,7 +94,7 @@ The Dataverse Collection Listing Widget provides a listing of all your Dataverse .. _openscholar-dataverse-level: Adding Widgets to an OpenScholar Website -****************************************** +**************************************** #. Log in to your OpenScholar website #. Either build a new page or navigate to the page you would like to use to show the Dataverse collection widgets. #. Click on the Settings Cog and select Layout @@ -102,8 +102,8 @@ Adding Widgets to an OpenScholar Website .. _dataverse-permissions: -Roles & Permissions ---------------------- +Roles & Permissions +------------------- Dataverse installation user accounts can be granted roles that define which actions they are allowed to take on specific Dataverse collections, datasets, and/or files. Each role comes with a set of permissions, which define the specific actions that users may take. Roles and permissions may also be granted to groups. Groups can be defined as a collection of Dataverse installation user accounts, a collection of IP addresses (e.g. all users of a library's computers), or a collection of all users who log in using a particular institutional login (e.g. everyone who logs in with a particular university's account credentials). @@ -127,7 +127,7 @@ When you access a Dataverse collection's permissions page, you will see three se Please note that even on a newly created Dataverse collection, you may see user and groups have already been granted role(s) if your installation has ``:InheritParentRoleAssignments`` set. For more on this setting, see the :doc:`/installation/config` section of the Installation Guide. Setting Access Configurations -******************************* +***************************** Under the Permissions tab, you can click the "Edit Access" button to open a box where you can add to your Dataverse collection and what permissions are granted to those who add to your Dataverse collection. @@ -140,7 +140,7 @@ The second question on this page allows you to choose the role (and thus the per Both of these settings can be changed at any time. Assigning Roles to Users and Groups -************************************* +*********************************** Under the Users/Groups tab, you can add, edit, or remove the roles granted to users and groups on your Dataverse collection. A role is a set of permissions granted to a user or group when they're using your Dataverse collection. For example, giving your research assistant the "Contributor" role would give them the following self-explanatory permissions on your Dataverse collection and all datasets within your Dataverse collection: "ViewUnpublishedDataset", "DownloadFile", "EditDataset", and "DeleteDatasetDraft". They would, however, lack the "PublishDataset" permission, and thus would be unable to publish datasets on your Dataverse collection. If you wanted to give them that permission, you would give them a role with that permission, like the Curator role. Users and groups can hold multiple roles at the same time if needed. Roles can be removed at any time. All roles and their associated permissions are listed under the "Roles" tab of the same page. @@ -155,15 +155,16 @@ Note: If you need to assign a role to ALL user accounts in a Dataverse installat .. _dataset-templates: Dataset Templates -------------------- +----------------- -Templates are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in, or if you want to use a custom set of Terms of Use and Access for multiple datasets in a Dataverse collection. In Dataverse Software 4.0+, templates are created at the Dataverse collection level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already. +Templates are useful when you want to provide custom instructions on how to fill out a field, have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in, or if you want to use a custom set of Terms of Use and Access for multiple datasets in a Dataverse collection. In Dataverse Software 4.0+, templates are created at the Dataverse collection level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already. How do you create a template? #. Navigate to your Dataverse collection, click on the Edit Dataverse button and select Dataset Templates. #. Once you have clicked on Dataset Templates, you will be brought to the Dataset Templates page. On this page, you can 1) decide to use the dataset templates from your parent Dataverse collection 2) create a new dataset template or 3) do both. #. Click on the Create Dataset Template to get started. You will see that the template is the same as the create dataset page with an additional field at the top of the page to add a name for the template. +#. To add custom instructions, click on ''(None - click to add)'' and enter the instructions you wish users to see. If you wish to edit existing instructions, click on them to make the text editable. #. After adding information into the metadata fields you have information for and clicking Save and Add Terms, you will be brought to the page where you can add custom Terms of Use and Access. If you do not need custom Terms of Use and Access, click the Save Dataset Template, and only the metadata fields will be saved. #. After clicking Save Dataset Template, you will be brought back to the Manage Dataset Templates page and should see your template listed there now with the make default, edit, view, or delete options. #. A Dataverse collection does not have to have a default template and users can select which template they would like to use while on the Create Dataset page. @@ -174,7 +175,7 @@ How do you create a template? .. _dataset-guestbooks: Dataset Guestbooks ------------------------------ +------------------ Guestbooks allow you to collect data about who is downloading the files from your datasets. You can decide to collect account information (username, given name & last name, affiliation, etc.) as well as create custom questions (e.g., What do you plan to use this data for?). You are also able to download the data collected from the enabled guestbooks as CSV files to store and use outside of the Dataverse installation. @@ -227,7 +228,7 @@ Similarly to dataset linking, Dataverse collection linking allows a Dataverse co If you need to have a Dataverse collection linked to your Dataverse collection, please contact the support team for the Dataverse installation you are using. Publish Your Dataverse Collection -================================================================= +================================= Once your Dataverse collection is ready to go public, go to your Dataverse collection page, click on the "Publish" button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish, since once a Dataverse collection diff --git a/local_lib/com/apicatalog/titanium-json-ld/1.3.0-SNAPSHOT/titanium-json-ld-1.3.0-SNAPSHOT.jar b/local_lib/com/apicatalog/titanium-json-ld/1.3.0-SNAPSHOT/titanium-json-ld-1.3.0-SNAPSHOT.jar new file mode 100644 index 00000000000..ee499ae4b76 Binary files /dev/null and b/local_lib/com/apicatalog/titanium-json-ld/1.3.0-SNAPSHOT/titanium-json-ld-1.3.0-SNAPSHOT.jar differ diff --git a/modules/dataverse-parent/pom.xml b/modules/dataverse-parent/pom.xml index 22ea30795ba..14b84f80279 100644 --- a/modules/dataverse-parent/pom.xml +++ b/modules/dataverse-parent/pom.xml @@ -164,7 +164,7 @@ 1.15.0 - 0.4.1 + 2.10.1 4.13.1 5.7.0 diff --git a/pom.xml b/pom.xml index eab64e522a5..585b7ee30a9 100644 --- a/pom.xml +++ b/pom.xml @@ -52,7 +52,7 @@ --> - + @@ -112,7 +112,7 @@ com.apicatalog titanium-json-ld - 0.8.6 + 1.3.0-SNAPSHOT com.google.code.gson @@ -357,7 +357,7 @@ commons-codec commons-codec - 1.9 + 1.15 @@ -516,7 +516,19 @@ google-cloud-storage - + + + + com.auth0 + java-jwt + 3.19.1 + + + + io.github.erdtman + java-json-canonicalization + 1.1 + @@ -601,9 +613,9 @@ test - org.microbean - microbean-microprofile-config - ${microbean-mpconfig.version} + io.smallrye.config + smallrye-config + ${smallrye-mpconfig.version} test @@ -641,10 +653,17 @@ **/*.xml **/firstNames/*.* **/*.xsl - **/*.properties **/services/* + + src/main/resources + + true + + **/*.properties + + diff --git a/scripts/api/data/metadatablocks/citation.tsv b/scripts/api/data/metadatablocks/citation.tsv index 94aa509334f..29d121aae16 100644 --- a/scripts/api/data/metadatablocks/citation.tsv +++ b/scripts/api/data/metadatablocks/citation.tsv @@ -111,6 +111,7 @@ publicationIDType upc 14 publicationIDType url 15 publicationIDType urn 16 + publicationIDType DASH-NRS 17 contributorType Data Collector 0 contributorType Data Curator 1 contributorType Data Manager 2 diff --git a/scripts/api/data/workflows/internal-ldnannounce-workflow.json b/scripts/api/data/workflows/internal-ldnannounce-workflow.json new file mode 100644 index 00000000000..9cf058b68a1 --- /dev/null +++ b/scripts/api/data/workflows/internal-ldnannounce-workflow.json @@ -0,0 +1,16 @@ +{ + "name": "LDN Announce workflow", + "steps": [ + { + "provider":":internal", + "stepType":"ldnannounce", + "parameters": { + "stepName":"LDN Announce" + }, + "requiredSettings": { + ":LDNAnnounceRequiredFields": "string", + ":LDNTarget": "string" + } + } + ] +} diff --git a/src/main/java/edu/harvard/iq/dataverse/DataFile.java b/src/main/java/edu/harvard/iq/dataverse/DataFile.java index b21ab5fb7ba..cb43dff0e20 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataFile.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataFile.java @@ -605,7 +605,11 @@ public void setFilesize(long filesize) { * @return */ public String getFriendlySize() { - return FileSizeChecker.bytesToHumanReadable(filesize); + if (filesize != null) { + return FileSizeChecker.bytesToHumanReadable(filesize); + } else { + return BundleUtil.getStringFromBundle("file.sizeNotAvailable"); + } } public boolean isRestricted() { diff --git a/src/main/java/edu/harvard/iq/dataverse/Dataset.java b/src/main/java/edu/harvard/iq/dataverse/Dataset.java index c60ea7020bd..a4f82d41bac 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Dataset.java +++ b/src/main/java/edu/harvard/iq/dataverse/Dataset.java @@ -33,8 +33,8 @@ import javax.persistence.Table; import javax.persistence.Temporal; import javax.persistence.TemporalType; -import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.StringUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; /** * @@ -152,6 +152,19 @@ public void setCitationDateDatasetFieldType(DatasetFieldType citationDateDataset this.citationDateDatasetFieldType = citationDateDatasetFieldType; } + + @ManyToOne + @JoinColumn(name="template_id",nullable = true) + private Template template; + + public Template getTemplate() { + return template; + } + + public void setTemplate(Template template) { + this.template = template; + } + public Dataset() { DatasetVersion datasetVersion = new DatasetVersion(); datasetVersion.setDataset(this); @@ -743,6 +756,11 @@ public void setHarvestIdentifier(String harvestIdentifier) { this.harvestIdentifier = harvestIdentifier; } + public String getLocalURL() { + //Assumes GlobalId != null + return SystemConfig.getDataverseSiteUrlStatic() + "/dataset.xhtml?persistentId=" + this.getGlobalId().asString(); + } + public String getRemoteArchiveURL() { if (isHarvested()) { if (HarvestingClient.HARVEST_STYLE_DATAVERSE.equals(this.getHarvestedFrom().getHarvestStyle())) { diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java index d0ba86ab68e..7b857545c20 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java @@ -77,7 +77,10 @@ public enum Reason { /** DCM (rsync) upload in progress */ DcmUpload, - + + /** Globus upload in progress */ + GlobusUpload, + /** Tasks handled by FinalizeDatasetPublicationCommand: Registering PIDs for DS and DFs and/or file validation */ finalizePublication, diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java index 2236bdc24ba..0a8db69bf5b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java @@ -113,6 +113,7 @@ import edu.harvard.iq.dataverse.engine.command.impl.SubmitDatasetForReviewCommand; import edu.harvard.iq.dataverse.externaltools.ExternalTool; import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.export.SchemaDotOrgExporter; import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean; @@ -251,6 +252,8 @@ public enum DisplayMode { LicenseServiceBean licenseServiceBean; @Inject DataFileCategoryServiceBean dataFileCategoryService; + @Inject + GlobusServiceBean globusService; private Dataset dataset = new Dataset(); @@ -334,7 +337,7 @@ public void setSelectedHostDataverse(Dataverse selectedHostDataverse) { private Boolean hasRsyncScript = false; private Boolean hasTabular = false; - + /** * If the dataset version has at least one tabular file. The "hasTabular" @@ -1191,7 +1194,7 @@ public String getComputeUrl(FileMetadata metadata) { } catch (IOException e) { logger.info("DatasetPage: Failed to get storageIO"); } - if (settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)) { + if (isHasPublicStore()) { return settingsWrapper.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getPersistentId() + "=" + swiftObject.getSwiftFileName(); } @@ -1768,6 +1771,7 @@ public void handleChangeButton() { workingVersion.initDefaultValues(licenseServiceBean.getDefault()); updateDatasetFieldInputLevels(); } + dataset.setTemplate(selectedTemplate); /* Issue 8646: necessary for the access popup which is shared by the dataset page and the file page */ @@ -1827,15 +1831,21 @@ public void updateOwnerDataverse() { // initiate from scratch: (isolate the creation of a new dataset in its own method?) init(true); - // rebuild the bred crumbs display: + // rebuild the bread crumbs display: dataverseHeaderFragment.initBreadcrumbs(dataset); } } public boolean rsyncUploadSupported() { - return settingsWrapper.isRsyncUpload() && DatasetUtil.isAppropriateStorageDriver(dataset); + return settingsWrapper.isRsyncUpload() && DatasetUtil.isRsyncAppropriateStorageDriver(dataset); + } + + public boolean globusUploadSupported() { + return settingsWrapper.isGlobusUpload() && settingsWrapper.isGlobusEnabledStorageDriver(dataset.getEffectiveStorageDriverId()); } + + private String init(boolean initFull) { @@ -2005,10 +2015,10 @@ private String init(boolean initFull) { } } catch (RuntimeException ex) { logger.warning("Problem getting rsync script(RuntimeException): " + ex.getLocalizedMessage()); - FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", ex.getLocalizedMessage())); + FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", ex.getLocalizedMessage())); } catch (CommandException cex) { logger.warning("Problem getting rsync script (Command Exception): " + cex.getLocalizedMessage()); - FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", cex.getLocalizedMessage())); + FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", cex.getLocalizedMessage())); } } @@ -2055,6 +2065,8 @@ private String init(boolean initFull) { selectedTemplate = testT; } } + //Initalize with the default if there is one + dataset.setTemplate(selectedTemplate); workingVersion = dataset.getEditVersion(selectedTemplate, null); updateDatasetFieldInputLevels(); } else { @@ -2062,7 +2074,7 @@ private String init(boolean initFull) { updateDatasetFieldInputLevels(); } - if (settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){ + if (isHasPublicStore()){ JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("dataset.message.label.fileAccess"), BundleUtil.getStringFromBundle("dataset.message.publicInstall")); } @@ -2175,6 +2187,10 @@ private void displayLockInfo(Dataset dataset) { BundleUtil.getStringFromBundle("file.rsyncUpload.inProgressMessage.details")); lockedDueToDcmUpload = true; } + if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.globusUpload.inProgressMessage.summary"), + BundleUtil.getStringFromBundle("file.globusUpload.inProgressMessage.details")); + } //This is a hack to remove dataset locks for File PID registration if //the dataset is released //in testing we had cases where datasets with 1000 files were remaining locked after being published successfully @@ -2896,7 +2912,7 @@ public String editFileMetadata(){ public String deleteDatasetVersion() { DeleteDatasetVersionCommand cmd; - + Map deleteStorageLocations = datafileService.getPhysicalFilesToDelete(dataset.getLatestVersion()); boolean deleteCommandSuccess = false; try { @@ -2908,7 +2924,7 @@ public String deleteDatasetVersion() { JH.addMessage(FacesMessage.SEVERITY_FATAL, BundleUtil.getStringFromBundle("dataset.message.deleteFailure")); logger.severe(ex.getMessage()); } - + if (deleteCommandSuccess && !deleteStorageLocations.isEmpty()) { datafileService.finalizeFileDeletes(deleteStorageLocations); } @@ -3572,6 +3588,7 @@ public String save() { if (editMode == EditMode.CREATE) { //Lock the metadataLanguage once created dataset.setMetadataLanguage(getEffectiveMetadataLanguage()); + //ToDo - could drop use of selectedTemplate and just use the persistent dataset.getTemplate() if ( selectedTemplate != null ) { if ( isSessionUserAuthenticated() ) { cmd = new CreateNewDatasetCommand(dataset, dvRequestService.getDataverseRequest(), false, selectedTemplate); @@ -5022,7 +5039,7 @@ public boolean isFileAccessRequestMultiButtonRequired(){ } for (FileMetadata fmd : workingVersion.getFileMetadatas()){ //Change here so that if all restricted files have pending requests there's no Request Button - if ((!this.fileDownloadHelper.canDownloadFile(fmd) && (fmd.getDataFile().getFileAccessRequesters() == null + if ((!this.fileDownloadHelper.canDownloadFile(fmd) && (fmd.getDataFile().getFileAccessRequesters() == null || ( fmd.getDataFile().getFileAccessRequesters() != null && !fmd.getDataFile().getFileAccessRequesters().contains((AuthenticatedUser)session.getUser()))))){ return true; @@ -5750,7 +5767,7 @@ public boolean isFileDeleted (DataFile dataFile) { return dataFile.getDeleted(); } - + public String getEffectiveMetadataLanguage() { return getEffectiveMetadataLanguage(false); } @@ -5761,16 +5778,16 @@ public String getEffectiveMetadataLanguage(boolean ofParent) { } return mdLang; } - + public String getLocaleDisplayName(String code) { String displayName = settingsWrapper.getBaseMetadataLanguageMap(false).get(code); if(displayName==null && !code.equals(DvObjectContainer.UNDEFINED_METADATA_LANGUAGE_CODE)) { //Default (for cases such as :when a Dataset has a metadatalanguage code but :MetadataLanguages is no longer defined). - displayName = new Locale(code).getDisplayName(); + displayName = new Locale(code).getDisplayName(); } - return displayName; + return displayName; } - + public Set> getMetadataLanguages() { return settingsWrapper.getBaseMetadataLanguageMap(false).entrySet(); } @@ -5782,7 +5799,7 @@ public List getVocabScripts() { public String getFieldLanguage(String languages) { return fieldService.getFieldLanguage(languages,session.getLocaleCode()); } - + public void setExternalStatus(String status) { try { dataset = commandEngine.submit(new SetCurationStatusCommand(dvRequestService.getDataverseRequest(), dataset, status)); @@ -6013,7 +6030,7 @@ public void validateTerms(FacesContext context, UIComponent component, Object va } } } - + public boolean downloadingRestrictedFiles() { if (fileMetadataForAction != null) { return fileMetadataForAction.isRestricted(); @@ -6025,4 +6042,24 @@ public boolean downloadingRestrictedFiles() { } return false; } + + + //Determines whether this Dataset uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(dataset.getEffectiveStorageDriverId())); + } + + public void startGlobusTransfer() { + ApiToken apiToken = null; + User user = session.getUser(); + if (user instanceof AuthenticatedUser) { + apiToken = authService.findApiTokenByUser((AuthenticatedUser) user); + } else if (user instanceof PrivateUrlUser) { + PrivateUrlUser privateUrlUser = (PrivateUrlUser) user; + PrivateUrl privUrl = privateUrlService.getPrivateUrlFromDatasetId(privateUrlUser.getDatasetId()); + apiToken = new ApiToken(); + apiToken.setTokenString(privUrl.getToken()); + } + PrimeFaces.current().executeScript(globusService.getGlobusDownloadScript(dataset, apiToken)); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java index ce8420b91e5..e1ca5c19a90 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java @@ -16,23 +16,17 @@ import edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand; import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetStorageSizeCommand; import edu.harvard.iq.dataverse.export.ExportService; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean; import edu.harvard.iq.dataverse.search.IndexServiceBean; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.SystemConfig; import edu.harvard.iq.dataverse.workflows.WorkflowComment; -import java.io.File; -import java.io.IOException; -import java.io.InputStream; + +import java.io.*; import java.text.SimpleDateFormat; -import java.util.ArrayList; -import java.util.Date; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; +import java.util.*; import java.util.logging.FileHandler; import java.util.logging.Level; import java.util.logging.Logger; @@ -96,6 +90,12 @@ public class DatasetServiceBean implements java.io.Serializable { @EJB SystemConfig systemConfig; + @EJB + GlobusServiceBean globusServiceBean; + + @EJB + UserNotificationServiceBean userNotificationService; + private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); @PersistenceContext(unitName = "VDCNet-ejbPU") @@ -1130,4 +1130,5 @@ public void deleteHarvestedDataset(Dataset dataset, DataverseRequest request, Lo hdLogger.warning("Failed to destroy the dataset"); } } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/Dataverse.java b/src/main/java/edu/harvard/iq/dataverse/Dataverse.java index db5f9d172cd..bc8716b6129 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Dataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/Dataverse.java @@ -324,8 +324,31 @@ public boolean isHarvested() { return harvestingClient != null; } */ - - + private boolean metadataBlockFacetRoot; + + public boolean isMetadataBlockFacetRoot() { + return metadataBlockFacetRoot; + } + + public void setMetadataBlockFacetRoot(boolean metadataBlockFacetRoot) { + this.metadataBlockFacetRoot = metadataBlockFacetRoot; + } + + @OneToMany(mappedBy = "dataverse",cascade={ CascadeType.REMOVE, CascadeType.MERGE,CascadeType.PERSIST }, orphanRemoval=true) + private List metadataBlockFacets = new ArrayList<>(); + + public List getMetadataBlockFacets() { + if (isMetadataBlockFacetRoot() || getOwner() == null) { + return metadataBlockFacets; + } else { + return getOwner().getMetadataBlockFacets(); + } + } + + public void setMetadataBlockFacets(List metadataBlockFacets) { + this.metadataBlockFacets = metadataBlockFacets; + } + public List getParentGuestbooks() { List retList = new ArrayList<>(); Dataverse testDV = this; diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacet.java b/src/main/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacet.java new file mode 100644 index 00000000000..a2659b81974 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacet.java @@ -0,0 +1,82 @@ +package edu.harvard.iq.dataverse; + +import javax.persistence.Entity; +import javax.persistence.GeneratedValue; +import javax.persistence.GenerationType; +import javax.persistence.Id; +import javax.persistence.Index; +import javax.persistence.JoinColumn; +import javax.persistence.ManyToOne; +import javax.persistence.Table; +import java.io.Serializable; +import java.util.Objects; + +/** + * + * @author adaybujeda + */ +@Entity +@Table(indexes = {@Index(columnList="dataverse_id") + , @Index(columnList="metadatablock_id")}) +public class DataverseMetadataBlockFacet implements Serializable { + private static final long serialVersionUID = 1L; + + @Id + @GeneratedValue(strategy = GenerationType.IDENTITY) + private Long id; + + @ManyToOne + @JoinColumn(name = "dataverse_id") + private Dataverse dataverse; + + @ManyToOne + @JoinColumn(name = "metadatablock_id") + private MetadataBlock metadataBlock; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Dataverse getDataverse() { + return dataverse; + } + + public void setDataverse(Dataverse dataverse) { + this.dataverse = dataverse; + } + + public MetadataBlock getMetadataBlock() { + return metadataBlock; + } + + public void setMetadataBlock(MetadataBlock metadataBlock) { + this.metadataBlock = metadataBlock; + } + + @Override + public int hashCode() { + int hash = 0; + hash += (this.id != null ? this.id.hashCode() : 0); + return hash; + } + + @Override + public boolean equals(Object object) { + if (!(object instanceof DataverseMetadataBlockFacet)) { + return false; + } + DataverseMetadataBlockFacet other = (DataverseMetadataBlockFacet) object; + return !(!Objects.equals(this.id, other.id) && (this.id == null || !this.id.equals(other.id))); + } + + @Override + public String toString() { + return String.format("edu.harvard.iq.dataverse.DataverseMetadataBlockFacet[ id=%s ]", id); + } + +} + diff --git a/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java b/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java index 746efded48b..6ff01ef3ea8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java +++ b/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java @@ -15,7 +15,6 @@ public abstract class DvObjectContainer extends DvObject { - //Default to "file" is for tests only public static final String UNDEFINED_METADATA_LANGUAGE_CODE = "undefined"; //Used in dataverse.xhtml as a non-null selection option value (indicating inheriting the default) @@ -93,6 +92,9 @@ public void setMetadataLanguage(String ml) { } } + public static boolean isMetadataLanguageSet(String mdLang) { + return mdLang!=null && !mdLang.equals(UNDEFINED_METADATA_LANGUAGE_CODE); + } /* Dataverse collections can be configured to allow use of Curation labels and have this inheritable value to decide which set of labels to use. diff --git a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java index 37ce04ed5c4..6cf294ffd6d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java @@ -650,8 +650,8 @@ public String init() { setUpRsync(); } - if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){ - JH.addMessage(FacesMessage.SEVERITY_WARN, getBundleString("dataset.message.publicInstall")); + if (isHasPublicStore()){ + JH.addMessage(FacesMessage.SEVERITY_WARN, getBundleString("dataset.message.label.fileAccess"), getBundleString("dataset.message.publicInstall")); } return null; @@ -1933,7 +1933,7 @@ private void handleReplaceFileUpload(String fullStorageLocation, fileReplacePageHelper.resetReplaceFileHelper(); saveEnabled = false; - String storageIdentifier = DataAccess.getStorarageIdFromLocation(fullStorageLocation); + String storageIdentifier = DataAccess.getStorageIdFromLocation(fullStorageLocation); if (fileReplacePageHelper.handleNativeFileUpload(null, storageIdentifier, fileName, contentType, checkSumValue, checkSumType)) { saveEnabled = true; @@ -2078,8 +2078,12 @@ public void handleExternalUpload() { if (!checksumTypeString.isBlank()) { checksumType = ChecksumType.fromString(checksumTypeString); } + + //Should only be one colon with curent design int lastColon = fullStorageIdentifier.lastIndexOf(':'); - String storageLocation = fullStorageIdentifier.substring(0, lastColon) + "/" + dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/" + fullStorageIdentifier.substring(lastColon + 1); + String storageLocation = fullStorageIdentifier.substring(0,lastColon) + "/" + dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/" + fullStorageIdentifier.substring(lastColon+1); + storageLocation = DataAccess.expandStorageIdentifierIfNeeded(storageLocation); + if (uploadInProgress.isFalse()) { uploadInProgress.setValue(true); } @@ -3044,16 +3048,24 @@ public void saveAdvancedOptions() { } public boolean rsyncUploadSupported() { - // ToDo - rsync was written before multiple store support and currently is hardcoded to use the "s3" store. + // ToDo - rsync was written before multiple store support and currently is hardcoded to use the DataAccess.S3 store. // When those restrictions are lifted/rsync can be configured per store, the test in the // Dataset Util method should be updated - if (settingsWrapper.isRsyncUpload() && !DatasetUtil.isAppropriateStorageDriver(dataset)) { + if (settingsWrapper.isRsyncUpload() && !DatasetUtil.isRsyncAppropriateStorageDriver(dataset)) { //dataset.file.upload.setUp.rsync.failed.detail FacesMessage message = new FacesMessage(FacesMessage.SEVERITY_ERROR, BundleUtil.getStringFromBundle("dataset.file.upload.setUp.rsync.failed"), BundleUtil.getStringFromBundle("dataset.file.upload.setUp.rsync.failed.detail")); FacesContext.getCurrentInstance().addMessage(null, message); } - return settingsWrapper.isRsyncUpload() && DatasetUtil.isAppropriateStorageDriver(dataset); + return settingsWrapper.isRsyncUpload() && DatasetUtil.isRsyncAppropriateStorageDriver(dataset); + } + + // Globus must be one of the upload methods listed in the :UploadMethods setting + // and the dataset's store must be in the list allowed by the GlobusStores + // setting + public boolean globusUploadSupported() { + return settingsWrapper.isGlobusUpload() + && settingsWrapper.isGlobusEnabledStorageDriver(dataset.getEffectiveStorageDriverId()); } private void populateFileMetadatas() { @@ -3089,4 +3101,9 @@ public boolean isFileAccessRequest() { public void setFileAccessRequest(boolean fileAccessRequest) { this.fileAccessRequest = fileAccessRequest; } + + //Determines whether this Dataset uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(dataset.getEffectiveStorageDriverId())); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java index 8110b1e0189..65e6b259bf4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java @@ -560,12 +560,12 @@ public void addFileToCustomZipJob(String key, DataFile dataFile, Timestamp times public String getDirectStorageLocatrion(String storageLocation) { String storageDriverId; - int separatorIndex = storageLocation.indexOf("://"); + int separatorIndex = storageLocation.indexOf(DataAccess.SEPARATOR); if ( separatorIndex > 0 ) { storageDriverId = storageLocation.substring(0,separatorIndex); String storageType = DataAccess.getDriverType(storageDriverId); - if ("file".equals(storageType) || "s3".equals(storageType)) { + if (DataAccess.FILE.equals(storageType) || DataAccess.S3.equals(storageType)) { return storageType.concat(storageLocation.substring(separatorIndex)); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/FilePage.java b/src/main/java/edu/harvard/iq/dataverse/FilePage.java index 3fa6d4fdfff..7f2c6dfca5c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FilePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/FilePage.java @@ -13,6 +13,7 @@ import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.dataaccess.StorageIO; import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; @@ -843,7 +844,7 @@ public String getComputeUrl() throws IOException { if (swiftObject != null) { swiftObject.open(); //generate a temp url for a file - if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)) { + if (isHasPublicStore()) { return settingsService.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getFile().getOwner().getGlobalIdString() + "=" + swiftObject.getSwiftFileName(); } return settingsService.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getFile().getOwner().getGlobalIdString() + "=" + swiftObject.getSwiftFileName() + "&temp_url_sig=" + swiftObject.getTempUrlSignature() + "&temp_url_expires=" + swiftObject.getTempUrlExpiry(); @@ -935,8 +936,8 @@ public String getPublicDownloadUrl() { try { SwiftAccessIO swiftIO = (SwiftAccessIO) storageIO; swiftIO.open(); - //if its a public install, lets just give users the permanent URL! - if (systemConfig.isPublicInstall()){ + //if its a public store, lets just give users the permanent URL! + if (isHasPublicStore()){ fileDownloadUrl = swiftIO.getRemoteUrl(); } else { //TODO: if a user has access to this file, they should be given the swift url @@ -1165,5 +1166,10 @@ public String getEmbargoPhrase() { public String getIngestMessage() { return BundleUtil.getStringFromBundle("file.ingestFailed.message", Arrays.asList(settingsWrapper.getGuidesBaseUrl(), settingsWrapper.getGuidesVersion())); } + + //Determines whether this File uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(DataAccess.getStorageDriverFromIdentifier(file.getStorageIdentifier()))); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java index f39fb8b0a32..2bfd342d899 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java @@ -16,6 +16,8 @@ import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.MailUtil; import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.json.JsonUtil; + import java.io.UnsupportedEncodingException; import java.text.MessageFormat; import java.util.ArrayList; @@ -169,7 +171,7 @@ public boolean sendSystemEmail(String to, String subject, String messageText, bo return sent; } - private InternetAddress getSystemAddress() { + public InternetAddress getSystemAddress() { String systemEmail = settingsService.getValueForKey(Key.SystemEmail); return MailUtil.parseSystemAddress(systemEmail); } @@ -568,6 +570,49 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio logger.fine("fileImportMsg: " + fileImportMsg); return messageText += fileImportMsg; + case GLOBUSUPLOADCOMPLETED: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String uploadCompletedMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.upload.completed", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return uploadCompletedMessage; + + case GLOBUSDOWNLOADCOMPLETED: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String downloadCompletedMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.download.completed", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return downloadCompletedMessage; + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String uploadCompletedWithErrorsMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.upload.completedWithErrors", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return uploadCompletedWithErrorsMessage; + + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String downloadCompletedWithErrorsMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.download.completedWithErrors", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return downloadCompletedWithErrorsMessage; + case CHECKSUMIMPORT: version = (DatasetVersion) targetObject; String checksumImportMsg = BundleUtil.getStringFromBundle("notification.import.checksum", Arrays.asList( @@ -608,6 +653,26 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio )); return ingestedCompletedWithErrorsMessage; + case DATASETMENTIONED: + String additionalInfo = userNotification.getAdditionalInfo(); + dataset = (Dataset) targetObject; + javax.json.JsonObject citingResource = null; + citingResource = JsonUtil.getJsonObject(additionalInfo); + + + pattern = BundleUtil.getStringFromBundle("notification.email.datasetWasMentioned"); + Object[] paramArrayDatasetMentioned = { + userNotification.getUser().getName(), + BrandingUtil.getInstallationBrandName(), + citingResource.getString("@type"), + citingResource.getString("@id"), + citingResource.getString("name"), + citingResource.getString("relationship"), + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalId().toString(), + dataset.getDisplayName()}; + messageText = MessageFormat.format(pattern, paramArrayDatasetMentioned); + return messageText; } return ""; @@ -632,6 +697,7 @@ public Object getObjectOfNotification (UserNotification userNotification){ case GRANTFILEACCESS: case REJECTFILEACCESS: case DATASETCREATED: + case DATASETMENTIONED: return datasetService.find(userNotification.getObjectId()); case CREATEDS: case SUBMITTEDDS: @@ -648,6 +714,11 @@ public Object getObjectOfNotification (UserNotification userNotification){ return datasetService.find(userNotification.getObjectId()); case FILESYSTEMIMPORT: return versionService.find(userNotification.getObjectId()); + case GLOBUSUPLOADCOMPLETED: + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + case GLOBUSDOWNLOADCOMPLETED: + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + return datasetService.find(userNotification.getObjectId()); case CHECKSUMIMPORT: return versionService.find(userNotification.getObjectId()); case APIGENERATED: diff --git a/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java b/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java index 844c0ec5be7..33e75efffb5 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java +++ b/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java @@ -202,10 +202,18 @@ public String toString() { return "edu.harvard.iq.dataverse.MetadataBlock[ id=" + id + " ]"; } - public String getLocaleDisplayName() - { + public String getLocaleDisplayName() { + return getLocaleValue("metadatablock.displayName"); + } + + public String getLocaleDisplayFacet() { + return getLocaleValue("metadatablock.displayFacet"); + } + + // Visible for testing + String getLocaleValue(String metadataBlockKey) { try { - return BundleUtil.getStringFromPropertyFile("metadatablock.displayName", getName()); + return BundleUtil.getStringFromPropertyFile(metadataBlockKey, getName()); } catch (MissingResourceException e) { return displayName; } diff --git a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java index aaf38af1b36..8f7f53de1a2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java @@ -733,6 +733,9 @@ else if (dataset.isLockedFor(DatasetLock.Reason.Workflow)) { else if (dataset.isLockedFor(DatasetLock.Reason.DcmUpload)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); } + else if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); + } else if (dataset.isLockedFor(DatasetLock.Reason.EditInProgress)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); } @@ -768,6 +771,9 @@ else if (dataset.isLockedFor(DatasetLock.Reason.Workflow)) { else if (dataset.isLockedFor(DatasetLock.Reason.DcmUpload)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); } + else if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); + } else if (dataset.isLockedFor(DatasetLock.Reason.EditInProgress)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); } diff --git a/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java b/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java index 9bf155740af..aa40423000d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java +++ b/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java @@ -13,6 +13,7 @@ import edu.harvard.iq.dataverse.util.MailUtil; import edu.harvard.iq.dataverse.util.StringUtil; import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.json.JsonUtil; import edu.harvard.iq.dataverse.UserNotification.Type; import java.time.LocalDate; @@ -92,7 +93,15 @@ public class SettingsWrapper implements java.io.Serializable { private Boolean rsyncUpload = null; - private Boolean rsyncDownload = null; + private Boolean rsyncDownload = null; + + private Boolean globusUpload = null; + private Boolean globusDownload = null; + private Boolean globusFileDownload = null; + + private String globusAppUrl = null; + + private List globusStoreList = null; private Boolean httpUpload = null; @@ -292,6 +301,42 @@ public boolean isRsyncDownload() { } return rsyncDownload; } + + public boolean isGlobusUpload() { + if (globusUpload == null) { + globusUpload = systemConfig.isGlobusUpload(); + } + return globusUpload; + } + + public boolean isGlobusDownload() { + if (globusDownload == null) { + globusDownload = systemConfig.isGlobusDownload(); + } + return globusDownload; + } + + public boolean isGlobusFileDownload() { + if (globusFileDownload == null) { + globusFileDownload = systemConfig.isGlobusFileDownload(); + } + return globusFileDownload; + } + + public boolean isGlobusEnabledStorageDriver(String driverId) { + if (globusStoreList == null) { + globusStoreList = systemConfig.getGlobusStoresList(); + } + return globusStoreList.contains(driverId); + } + + public String getGlobusAppUrl() { + if (globusAppUrl == null) { + globusAppUrl = settingsService.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost"); + } + return globusAppUrl; + + } public boolean isRsyncOnly() { if (rsyncOnly == null) { @@ -646,5 +691,4 @@ public boolean isCustomLicenseAllowed() { } return customLicenseAllowed; } -} - +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/Shib.java b/src/main/java/edu/harvard/iq/dataverse/Shib.java index 324f6e185a6..0f0e20aba94 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Shib.java +++ b/src/main/java/edu/harvard/iq/dataverse/Shib.java @@ -218,7 +218,26 @@ public void init() { ? getValueFromAssertion(shibAffiliationAttribute) : shibService.getAffiliation(shibIdp, shibService.getDevShibAccountType()); + if (affiliation != null) { + String ShibAffiliationSeparator = settingsService.getValueForKey(SettingsServiceBean.Key.ShibAffiliationSeparator); + if (ShibAffiliationSeparator == null) { + ShibAffiliationSeparator = ";"; + } + String ShibAffiliationOrder = settingsService.getValueForKey(SettingsServiceBean.Key.ShibAffiliationOrder); + if (ShibAffiliationOrder != null) { + if (ShibAffiliationOrder.equals("lastAffiliation")) { + affiliation = affiliation.substring(affiliation.lastIndexOf(ShibAffiliationSeparator) + 1); //patch for affiliation array returning last part + } + else if (ShibAffiliationOrder.equals("firstAffiliation")) { + try{ + affiliation = affiliation.substring(0,affiliation.indexOf(ShibAffiliationSeparator)); //patch for affiliation array returning first part + } + catch (Exception e){ + logger.info("Affiliation does not contain \"" + ShibAffiliationSeparator + "\""); + } + } + } affiliationToDisplayAtConfirmation = affiliation; friendlyNameForInstitution = affiliation; } diff --git a/src/main/java/edu/harvard/iq/dataverse/Template.java b/src/main/java/edu/harvard/iq/dataverse/Template.java index 4a2e8272c7d..61f0a78656f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Template.java +++ b/src/main/java/edu/harvard/iq/dataverse/Template.java @@ -1,7 +1,6 @@ package edu.harvard.iq.dataverse; import java.io.Serializable; -import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; @@ -10,6 +9,11 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.stream.Collectors; + +import javax.json.Json; +import javax.json.JsonObjectBuilder; +import javax.json.JsonString; import javax.persistence.CascadeType; import javax.persistence.Column; import javax.persistence.Entity; @@ -28,6 +32,8 @@ import javax.validation.constraints.Size; import edu.harvard.iq.dataverse.util.DateUtil; +import edu.harvard.iq.dataverse.util.json.JsonUtil; + import javax.persistence.NamedQueries; import javax.persistence.NamedQuery; import org.hibernate.validator.constraints.NotBlank; @@ -125,7 +131,13 @@ public void setTermsOfUseAndAccess(TermsOfUseAndAccess termsOfUseAndAccess) { public List getDatasetFields() { return datasetFields; } + + @Column(columnDefinition="TEXT", nullable = true ) + private String instructions; + @Transient + private Map instructionsMap = null; + @Transient private Map> metadataBlocksForView = new HashMap<>(); @Transient @@ -256,26 +268,31 @@ public void setMetadataValueBlocks() { metadataBlocksForView.clear(); metadataBlocksForEdit.clear(); List filledInFields = this.getDatasetFields(); + + Map instructionsMap = getInstructionsMap(); List viewMDB = new ArrayList<>(); - List editMDB=this.getDataverse().getMetadataBlocks(true); + List editMDB=this.getDataverse().getMetadataBlocks(false); + //The metadatablocks in this template include any from the Dataverse it is associated with + //plus any others where the template has a displayable field (i.e. from before a block was dropped in the dataverse/collection) viewMDB.addAll(this.getDataverse().getMetadataBlocks(true)); - for (DatasetField dsfv : filledInFields) { - if (!dsfv.isEmptyForDisplay()) { - MetadataBlock mdbTest = dsfv.getDatasetFieldType().getMetadataBlock(); + for (DatasetField dsf : filledInFields) { + if (!dsf.isEmptyForDisplay()) { + MetadataBlock mdbTest = dsf.getDatasetFieldType().getMetadataBlock(); if (!viewMDB.contains(mdbTest)) { viewMDB.add(mdbTest); } } - } - + } + for (MetadataBlock mdb : viewMDB) { List datasetFieldsForView = new ArrayList<>(); for (DatasetField dsf : this.getDatasetFields()) { if (dsf.getDatasetFieldType().getMetadataBlock().equals(mdb)) { - if (!dsf.isEmpty()) { + //For viewing, show the field if it has a value or custom instructions + if (!dsf.isEmpty() || instructionsMap.containsKey(dsf.getDatasetFieldType().getName())) { datasetFieldsForView.add(dsf); } } @@ -344,6 +361,9 @@ public Template cloneNewTemplate(Template source) { } terms.setTemplate(newTemplate); newTemplate.setTermsOfUseAndAccess(terms); + + newTemplate.getInstructionsMap().putAll(source.getInstructionsMap()); + newTemplate.updateInstructions(); return newTemplate; } @@ -383,6 +403,45 @@ private List getFlatDatasetFields(List dsfList) { return retList; } + //Cache values in map for reading + public Map getInstructionsMap() { + if(instructionsMap==null) + if(instructions != null) { + instructionsMap = JsonUtil.getJsonObject(instructions).entrySet().stream().collect(Collectors.toMap(entry -> entry.getKey(),entry -> ((JsonString)entry.getValue()).getString())); + } else { + instructionsMap = new HashMap(); + } + return instructionsMap; + } + + //Get the cutstom instructions defined for a give fieldType + public String getInstructionsFor(String fieldType) { + return getInstructionsMap().get(fieldType); + } + + /* + //Add/change or remove (null instructionString) instructions for a given fieldType + public void setInstructionsFor(String fieldType, String instructionString) { + if(instructionString==null) { + getInstructionsMap().remove(fieldType); + } else { + getInstructionsMap().put(fieldType, instructionString); + } + updateInstructions(); + } + */ + + //Keep instructions up-to-date on any change + public void updateInstructions() { + JsonObjectBuilder builder = Json.createObjectBuilder(); + getInstructionsMap().forEach((key, value) -> { + if (value != null) + builder.add(key, value); + }); + instructions = JsonUtil.prettyPrint(builder.build()); + } + + @Override public int hashCode() { int hash = 0; diff --git a/src/main/java/edu/harvard/iq/dataverse/TemplatePage.java b/src/main/java/edu/harvard/iq/dataverse/TemplatePage.java index 5f574c07d17..6da0d99da20 100644 --- a/src/main/java/edu/harvard/iq/dataverse/TemplatePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/TemplatePage.java @@ -185,6 +185,8 @@ public String save(String redirectPage) { DatasetFieldUtil.tidyUpFields( template.getDatasetFields(), false ); + template.updateInstructions(); + if (editMode == EditMode.CREATE) { template.setCreateTime(new Timestamp(new Date().getTime())); template.setUsageCount(new Long(0)); @@ -247,5 +249,11 @@ public String deleteTemplate(Long templateId) { } return "/manage-templates.xhtml?dataverseId=" + dataverse.getId() + "&faces-redirect=true"; } + + //Get the cutstom instructions defined for a give fieldType + public String getInstructionsLabelFor(String fieldType) { + String fieldInstructions = template.getInstructionsMap().get(fieldType); + return (fieldInstructions!=null && !fieldInstructions.isBlank()) ? fieldInstructions : BundleUtil.getStringFromBundle("template.instructions.empty.label"); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java index 5714a879527..b68a1b9d13e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java +++ b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java @@ -37,7 +37,9 @@ public enum Type { ASSIGNROLE, REVOKEROLE, CREATEDV, CREATEDS, CREATEACC, SUBMITTEDDS, RETURNEDDS, PUBLISHEDDS, REQUESTFILEACCESS, GRANTFILEACCESS, REJECTFILEACCESS, FILESYSTEMIMPORT, CHECKSUMIMPORT, CHECKSUMFAIL, CONFIRMEMAIL, APIGENERATED, INGESTCOMPLETED, INGESTCOMPLETEDWITHERRORS, - PUBLISHFAILED_PIDREG, WORKFLOW_SUCCESS, WORKFLOW_FAILURE, STATUSUPDATED, DATASETCREATED; + PUBLISHFAILED_PIDREG, WORKFLOW_SUCCESS, WORKFLOW_FAILURE, STATUSUPDATED, DATASETCREATED, DATASETMENTIONED, + GLOBUSUPLOADCOMPLETED, GLOBUSUPLOADCOMPLETEDWITHERRORS, + GLOBUSDOWNLOADCOMPLETED, GLOBUSDOWNLOADCOMPLETEDWITHERRORS; public String getDescription() { return BundleUtil.getStringFromBundle("notification.typeDescription." + this.name()); @@ -88,6 +90,8 @@ public static String toStringValue(Set typesSet) { @Column( nullable = false ) private Type type; private Long objectId; + + private String additionalInfo; @Transient private boolean displayAsRead; @@ -196,4 +200,12 @@ public void setRoleString(String roleString) { public String getLocaleSendDate() { return DateUtil.formatDate(sendDate); } + + public String getAdditionalInfo() { + return additionalInfo; + } + + public void setAdditionalInfo(String additionalInfo) { + this.additionalInfo = additionalInfo; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/UserNotificationServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/UserNotificationServiceBean.java index 6792a7bedc7..947ee3ce989 100644 --- a/src/main/java/edu/harvard/iq/dataverse/UserNotificationServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/UserNotificationServiceBean.java @@ -110,12 +110,16 @@ public void sendNotification(AuthenticatedUser dataverseUser, Timestamp sendDate } public void sendNotification(AuthenticatedUser dataverseUser, Timestamp sendDate, Type type, Long objectId, String comment, AuthenticatedUser requestor, boolean isHtmlContent) { + sendNotification(dataverseUser, sendDate, type, objectId, comment, requestor, isHtmlContent, null); + } + public void sendNotification(AuthenticatedUser dataverseUser, Timestamp sendDate, Type type, Long objectId, String comment, AuthenticatedUser requestor, boolean isHtmlContent, String additionalInfo) { UserNotification userNotification = new UserNotification(); userNotification.setUser(dataverseUser); userNotification.setSendDate(sendDate); userNotification.setType(type); userNotification.setObjectId(objectId); userNotification.setRequestor(requestor); + userNotification.setAdditionalInfo(additionalInfo); if (!isEmailMuted(userNotification) && mailService.sendNotificationEmail(userNotification, comment, requestor, isHtmlContent)) { logger.fine("email was sent"); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java index cfb30cc0753..abeedf23b59 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Access.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Access.java @@ -332,6 +332,11 @@ public Response datafile(@PathParam("fileId") String fileId, @QueryParam("gbrecs dInfo.addServiceAvailable(new OptionalAccessService("preprocessed", "application/json", "format=prep", "Preprocessed data in JSON")); dInfo.addServiceAvailable(new OptionalAccessService("subset", "text/tab-separated-values", "variables=<LIST>", "Column-wise Subsetting")); } + + if(systemConfig.isGlobusFileDownload() && systemConfig.getGlobusStoresList().contains(DataAccess.getStorageDriverFromIdentifier(df.getStorageIdentifier()))) { + dInfo.addServiceAvailable(new OptionalAccessService("GlobusTransfer", df.getContentType(), "format=GlobusTransfer", "Download via Globus")); + } + DownloadInstance downloadInstance = new DownloadInstance(dInfo); downloadInstance.setRequestUriInfo(uriInfo); downloadInstance.setRequestHttpHeaders(headers); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 7941dfd70c8..ddc80aa00b6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -7,6 +7,7 @@ import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.RoleAssignee; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.batch.jobs.importer.ImportMode; @@ -59,6 +60,7 @@ import edu.harvard.iq.dataverse.ingest.IngestServiceBean; import edu.harvard.iq.dataverse.privateurl.PrivateUrl; +import edu.harvard.iq.dataverse.S3PackageImporter; import edu.harvard.iq.dataverse.api.dto.RoleAssignmentDTO; import edu.harvard.iq.dataverse.batch.util.LoggingUtil; import edu.harvard.iq.dataverse.dataaccess.DataAccess; @@ -82,6 +84,7 @@ import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.EjbUtil; import edu.harvard.iq.dataverse.util.FileUtil; +import edu.harvard.iq.dataverse.util.MarkupChecker; import edu.harvard.iq.dataverse.util.SystemConfig; import edu.harvard.iq.dataverse.util.bagit.OREMap; import edu.harvard.iq.dataverse.util.json.JSONLDUtil; @@ -97,6 +100,8 @@ import edu.harvard.iq.dataverse.workflow.WorkflowServiceBean; import edu.harvard.iq.dataverse.workflow.WorkflowContext.TriggerType; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; + import java.io.IOException; import java.io.InputStream; import java.io.StringReader; @@ -106,9 +111,10 @@ import java.text.SimpleDateFormat; import java.time.LocalDate; import java.time.LocalDateTime; +import java.util.*; +import java.util.concurrent.*; import java.time.ZoneId; import java.time.format.DateTimeFormatter; -import java.util.*; import java.util.Map.Entry; import java.util.logging.Level; import java.util.logging.Logger; @@ -116,7 +122,6 @@ import javax.ejb.EJB; import javax.ejb.EJBException; -import javax.faces.context.FacesContext; import javax.inject.Inject; import javax.json.*; import javax.json.stream.JsonParsingException; @@ -134,10 +139,7 @@ import javax.ws.rs.PathParam; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.HttpHeaders; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; +import javax.ws.rs.core.*; import javax.ws.rs.core.Response.Status; import static javax.ws.rs.core.Response.Status.BAD_REQUEST; import javax.ws.rs.core.UriInfo; @@ -162,6 +164,9 @@ public class Datasets extends AbstractApiBean { @EJB DataverseServiceBean dataverseService; + @EJB + GlobusServiceBean globusService; + @EJB UserNotificationServiceBean userNotificationService; @@ -429,7 +434,7 @@ public Response setCitationDate( @PathParam("id") String id, String dsfTypeName) execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), dsfType)); return ok("Citation Date for dataset " + id + " set to: " + (dsfType != null ? dsfType.getDisplayName() : "default")); }); - } + } @DELETE @Path("{id}/citationdate") @@ -438,7 +443,7 @@ public Response useDefaultCitationDate( @PathParam("id") String id) { execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), null)); return ok("Citation Date for dataset " + id + " set to default"); }); - } + } @GET @Path("{id}/versions") @@ -454,9 +459,9 @@ public Response listVersions( @PathParam("id") String id ) { @Path("{id}/versions/{versionId}") public Response getVersion( @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> { - DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); + DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); return (dsv == null || dsv.getId() == null) ? notFound("Dataset version not found") - : ok(json(dsv)); + : ok(json(dsv)); }); } @@ -473,9 +478,9 @@ public Response getVersionFiles( @PathParam("id") String datasetId, @PathParam(" public Response getFileAccessFolderView(@PathParam("id") String datasetId, @QueryParam("version") String versionId, @QueryParam("folder") String folderName, @QueryParam("original") Boolean originals, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { folderName = folderName == null ? "" : folderName; - versionId = versionId == null ? ":latest-published" : versionId; + versionId = versionId == null ? ":latest-published" : versionId; - DatasetVersion version; + DatasetVersion version; try { DataverseRequest req = createDataverseRequest(findUserOrDie()); version = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); @@ -587,7 +592,7 @@ public Response updateDatasetPIDMetadataAll() { } catch (WrappedResponse ex) { Logger.getLogger(Datasets.class.getName()).log(Level.SEVERE, null, ex); } - }); + }); return ok(BundleUtil.getStringFromBundle("datasets.api.updatePIDMetadata.success.for.update.all")); }); } @@ -775,7 +780,7 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav boolean found = false; for (DatasetField dsf : dsv.getDatasetFields()) { if (dsf.getDatasetFieldType().equals(updateField.getDatasetFieldType())) { - if (dsf.getDatasetFieldType().isAllowMultiples()) { + if (dsf.getDatasetFieldType().isAllowMultiples()) { if (updateField.getDatasetFieldType().isControlledVocabulary()) { if (dsf.getDatasetFieldType().isAllowMultiples()) { for (ControlledVocabularyValue cvv : updateField.getControlledVocabularyValues()) { @@ -840,7 +845,7 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav datasetFieldCompoundValueItemsToRemove.forEach((remove) -> { dsf.getDatasetFieldCompoundValues().remove(remove); }); - if (!found) { + if (!found) { logger.log(Level.SEVERE, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + deleteVal + " not found."); return error(Response.Status.BAD_REQUEST, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + deleteVal + " not found."); } @@ -860,12 +865,11 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav logger.log(Level.SEVERE, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + displayValue + " not found." ); return error(Response.Status.BAD_REQUEST, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + displayValue + " not found." ); } - } + } - boolean updateDraft = ds.getLatestVersion().isDraft(); - DatasetVersion managedVersion = updateDraft + DatasetVersion managedVersion = updateDraft ? execCommand(new UpdateDatasetVersionCommand(ds, req)).getEditVersion() : execCommand(new CreateDatasetVersionCommand(req, ds, dsv)); return ok(json(managedVersion)); @@ -884,13 +888,13 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav private String getCompoundDisplayValue (DatasetFieldCompoundValue dscv){ String returnString = ""; - for (DatasetField dsf : dscv.getChildDatasetFields()) { - for (String value : dsf.getValues()) { - if (!(value == null)) { - returnString += (returnString.isEmpty() ? "" : "; ") + value.trim(); - } + for (DatasetField dsf : dscv.getChildDatasetFields()) { + for (String value : dsf.getValues()) { + if (!(value == null)) { + returnString += (returnString.isEmpty() ? "" : "; ") + value.trim(); } } + } return returnString; } @@ -919,13 +923,13 @@ private Response processDatasetUpdate(String jsonBody, String id, DataverseReque DatasetVersion dsv = ds.getEditVersion(); dsv.getTermsOfUseAndAccess().setDatasetVersion(dsv); List fields = new LinkedList<>(); - DatasetField singleField = null; + DatasetField singleField = null; JsonArray fieldsJson = json.getJsonArray("fields"); - if( fieldsJson == null ){ - singleField = jsonParser().parseField(json, Boolean.FALSE); + if (fieldsJson == null) { + singleField = jsonParser().parseField(json, Boolean.FALSE); fields.add(singleField); - } else{ + } else { fields = jsonParser().parseMultipleFields(json); } @@ -1086,15 +1090,15 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S case "major": isMinor = false; break; - case "updatecurrent": - if(user.isSuperuser()) { - updateCurrent=true; - } else { - return error(Response.Status.FORBIDDEN, "Only superusers can update the current version"); - } - break; + case "updatecurrent": + if (user.isSuperuser()) { + updateCurrent = true; + } else { + return error(Response.Status.FORBIDDEN, "Only superusers can update the current version"); + } + break; default: - return error(Response.Status.BAD_REQUEST, "Illegal 'type' parameter value '" + type + "'. It needs to be either 'major', 'minor', or 'updatecurrent'."); + return error(Response.Status.BAD_REQUEST, "Illegal 'type' parameter value '" + type + "'. It needs to be either 'major', 'minor', or 'updatecurrent'."); } Dataset ds = findDatasetOrDie(id); @@ -1114,7 +1118,7 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S * set and if so, if it after the modification time. If the modification time is * set and the index time is null or is before the mod time, the 409/conflict * error is returned. - * + * */ if ((ds.getModificationTime()!=null && (ds.getIndexTime() == null || (ds.getIndexTime().compareTo(ds.getModificationTime()) <= 0))) || (ds.getPermissionModificationTime()!=null && (ds.getPermissionIndexTime() == null || (ds.getPermissionIndexTime().compareTo(ds.getPermissionModificationTime()) <= 0)))) { @@ -1178,10 +1182,10 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S .build(); } } else { - PublishDatasetResult res = execCommand(new PublishDatasetCommand(ds, + PublishDatasetResult res = execCommand(new PublishDatasetCommand(ds, createDataverseRequest(user), - isMinor)); - return res.isWorkflow() ? accepted(json(res.getDataset())) : ok(json(res.getDataset())); + isMinor)); + return res.isWorkflow() ? accepted(json(res.getDataset())) : ok(json(res.getDataset())); } } catch (WrappedResponse ex) { return ex.getResponse(); @@ -1282,7 +1286,7 @@ public Response publishMigratedDataset(String jsonldBody, @PathParam("id") Strin @Path("{id}/move/{targetDataverseAlias}") public Response moveDataset(@PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { try { - User u = findUserOrDie(); + User u = findUserOrDie(); Dataset ds = findDatasetOrDie(id); Dataverse target = dataverseService.findByAlias(targetDataverseAlias); if (target == null) { @@ -1564,21 +1568,21 @@ public Response removeFileEmbargo(@PathParam("id") String id, String jsonBody){ @PUT - @Path("{linkedDatasetId}/link/{linkingDataverseAlias}") - public Response linkDataset(@PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { - try{ - User u = findUserOrDie(); + @Path("{linkedDatasetId}/link/{linkingDataverseAlias}") + public Response linkDataset(@PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { + try { + User u = findUserOrDie(); Dataset linked = findDatasetOrDie(linkedDatasetId); Dataverse linking = findDataverseOrDie(linkingDataverseAlias); if (linked == null){ return error(Response.Status.BAD_REQUEST, "Linked Dataset not found."); - } - if (linking == null){ + } + if (linking == null) { return error(Response.Status.BAD_REQUEST, "Linking Dataverse not found."); - } + } execCommand(new LinkDatasetCommand( createDataverseRequest(u), linking, linked - )); + )); return ok("Dataset " + linked.getId() + " linked successfully to " + linking.getAlias()); } catch (WrappedResponse ex) { return ex.getResponse(); @@ -1592,8 +1596,7 @@ public Response getCustomTermsTab(@PathParam("id") String id, @PathParam("versio User user = session.getUser(); String persistentId; try { - if (getDatasetVersionOrDie(createDataverseRequest(user), versionId, findDatasetOrDie(id), uriInfo, headers) - .getTermsOfUseAndAccess().getLicense() != null) { + if (DatasetUtil.getLicense(getDatasetVersionOrDie(createDataverseRequest(user), versionId, findDatasetOrDie(id), uriInfo, headers)) != null) { return error(Status.NOT_FOUND, "This Dataset has no custom license"); } persistentId = getRequestParameter(":persistentId".substring(1)); @@ -1634,8 +1637,8 @@ public Response getLinks(@PathParam("id") String idSupplied ) { /** * Add a given assignment to a given user or group - * @param ra role assignment DTO - * @param id dataset id + * @param ra role assignment DTO + * @param id dataset id * @param apiKey */ @POST @@ -1647,7 +1650,7 @@ public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") RoleAssignee assignee = findAssignee(ra.getAssignee()); if (assignee == null) { return error(Response.Status.BAD_REQUEST, BundleUtil.getStringFromBundle("datasets.api.grant.role.assignee.not.found.error")); - } + } DataverseRole theRole; Dataverse dv = dataset.getOwner(); @@ -1699,10 +1702,10 @@ public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam( @GET @Path("{identifier}/assignments") public Response getAssignments(@PathParam("identifier") String id) { - return response( req -> - ok( execCommand( - new ListRoleAssignments(req, findDatasetOrDie(id))) - .stream().map(ra->json(ra)).collect(toJsonArray())) ); + return response(req -> + ok(execCommand( + new ListRoleAssignments(req, findDatasetOrDie(id))) + .stream().map(ra -> json(ra)).collect(toJsonArray()))); } @GET @@ -1710,8 +1713,8 @@ public Response getAssignments(@PathParam("identifier") String id) { public Response getPrivateUrlData(@PathParam("id") String idSupplied) { return response( req -> { PrivateUrl privateUrl = execCommand(new GetPrivateUrlCommand(req, findDatasetOrDie(idSupplied))); - return (privateUrl != null) ? ok(json(privateUrl)) - : error(Response.Status.NOT_FOUND, "Private URL not found."); + return (privateUrl != null) ? ok(json(privateUrl)) + : error(Response.Status.NOT_FOUND, "Private URL not found."); }); } @@ -1721,7 +1724,7 @@ public Response createPrivateUrl(@PathParam("id") String idSupplied,@DefaultValu if(anonymizedAccess && settingsSvc.getValueForKey(SettingsServiceBean.Key.AnonymizedFieldTypeNames)==null) { throw new NotAcceptableException("Anonymized Access not enabled"); } - return response( req -> + return response(req -> ok(json(execCommand( new CreatePrivateUrlCommand(req, findDatasetOrDie(idSupplied), anonymizedAccess))))); } @@ -1855,13 +1858,13 @@ public Response getRsync(@PathParam("identifier") String id) { } /** - * This api endpoint triggers the creation of a "package" file in a dataset - * after that package has been moved onto the same filesystem via the Data Capture Module. + * This api endpoint triggers the creation of a "package" file in a dataset + * after that package has been moved onto the same filesystem via the Data Capture Module. * The package is really just a way that Dataverse interprets a folder created by DCM, seeing it as just one file. * The "package" can be downloaded over RSAL. - * + * * This endpoint currently supports both posix file storage and AWS s3 storage in Dataverse, and depending on which one is active acts accordingly. - * + * * The initial design of the DCM/Dataverse interaction was not to use packages, but to allow import of all individual files natively into Dataverse. * But due to the possibly immense number of files (millions) the package approach was taken. * This is relevant because the posix ("file") code contains many remnants of that development work. @@ -1885,7 +1888,7 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String try { Dataset dataset = findDatasetOrDie(id); if ("validation passed".equals(statusMessageFromDcm)) { - logger.log(Level.INFO, "Checksum Validation passed for DCM."); + logger.log(Level.INFO, "Checksum Validation passed for DCM."); String storageDriver = dataset.getDataverseContext().getEffectiveStorageDriverId(); String uploadFolder = jsonFromDcm.getString("uploadFolder"); @@ -1908,7 +1911,7 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String String message = wr.getMessage(); return error(Response.Status.INTERNAL_SERVER_ERROR, "Uploaded files have passed checksum validation but something went wrong while attempting to put the files into Dataverse. Message was '" + message + "'."); } - } else if(storageDriverType.equals("s3")) { + } else if(storageDriverType.equals(DataAccess.S3)) { logger.log(Level.INFO, "S3 storage driver used for DCM (dataset id={0})", dataset.getId()); try { @@ -1947,10 +1950,10 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String JsonObjectBuilder job = Json.createObjectBuilder(); return ok(job); - } catch (IOException e) { + } catch (IOException e) { String message = e.getMessage(); return error(Response.Status.INTERNAL_SERVER_ERROR, "Uploaded files have passed checksum validation but something went wrong while attempting to move the files into Dataverse. Message was '" + message + "'."); - } + } } else { return error(Response.Status.INTERNAL_SERVER_ERROR, "Invalid storage driver in Dataverse, not compatible with dcm"); } @@ -2003,7 +2006,7 @@ public Response returnToAuthor(@PathParam("id") String idSupplied, String jsonBo JsonObject json = Json.createReader(rdr).readObject(); try { Dataset dataset = findDatasetOrDie(idSupplied); - String reasonForReturn = null; + String reasonForReturn = null; reasonForReturn = json.getString("reasonForReturn"); // TODO: Once we add a box for the curator to type into, pass the reason for return to the ReturnDatasetToAuthorCommand and delete this check and call to setReturnReason on the API side. if (reasonForReturn == null || reasonForReturn.isEmpty()) { @@ -2056,7 +2059,7 @@ public Response setCurationStatus(@PathParam("id") String idSupplied, @QueryPara return Response.fromResponse(wr.getResponse()).status(Response.Status.BAD_REQUEST).build(); } } - + @DELETE @Path("{id}/curationStatus") public Response deleteCurationStatus(@PathParam("id") String idSupplied) { @@ -2076,228 +2079,228 @@ public Response deleteCurationStatus(@PathParam("id") String idSupplied) { return Response.fromResponse(wr.getResponse()).status(Response.Status.BAD_REQUEST).build(); } } - -@GET -@Path("{id}/uploadsid") -@Deprecated -public Response getUploadUrl(@PathParam("id") String idSupplied) { - try { - Dataset dataset = findDatasetOrDie(idSupplied); - - boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info("Exception thrown while trying to figure out permissions while getting upload URL for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - if (!canUpdateDataset) { - return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); - } - S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); - if(s3io == null) { - return error(Response.Status.NOT_FOUND,"Direct upload not supported for files in this dataset: " + dataset.getId()); - } - String url = null; - String storageIdentifier = null; - try { - url = s3io.generateTemporaryS3UploadUrl(); - storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); - } catch (IOException io) { - logger.warning(io.getMessage()); - throw new WrappedResponse(io, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); - } - - JsonObjectBuilder response = Json.createObjectBuilder() - .add("url", url) - .add("storageIdentifier", storageIdentifier ); - return ok(response); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} -@GET -@Path("{id}/uploadurls") -public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { - try { - Dataset dataset = findDatasetOrDie(idSupplied); - - boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions while getting upload URLs for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - if (!canUpdateDataset) { - return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); - } - S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); - if (s3io == null) { - return error(Response.Status.NOT_FOUND, - "Direct upload not supported for files in this dataset: " + dataset.getId()); - } - JsonObjectBuilder response = null; - String storageIdentifier = null; - try { - storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); - response = s3io.generateTemporaryS3UploadUrls(dataset.getGlobalId().asString(), storageIdentifier, fileSize); - - } catch (IOException io) { - logger.warning(io.getMessage()); - throw new WrappedResponse(io, - error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); - } - - response.add("storageIdentifier", storageIdentifier); - return ok(response); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + @GET + @Path("{id}/uploadsid") + @Deprecated + public Response getUploadUrl(@PathParam("id") String idSupplied) { + try { + Dataset dataset = findDatasetOrDie(idSupplied); -@DELETE -@Path("mpupload") -public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { - try { - Dataset dataset = datasetSvc.findByGlobalId(idSupplied); - //Allow the API to be used within a session (e.g. for direct upload in the UI) - User user =session.getUser(); - if (!user.isAuthenticated()) { - try { - user = findAuthenticatedUserOrDie(); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions while getting aborting upload for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - } - boolean allowed = false; - if (dataset != null) { - allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } else { - /* - * The only legitimate case where a global id won't correspond to a dataset is - * for uploads during creation. Given that this call will still fail unless all - * three parameters correspond to an active multipart upload, it should be safe - * to allow the attempt for an authenticated user. If there are concerns about - * permissions, one could check with the current design that the user is allowed - * to create datasets in some dataverse that is configured to use the storage - * provider specified in the storageidentifier, but testing for the ability to - * create a dataset in a specific dataverse would requiring changing the design - * somehow (e.g. adding the ownerId to this call). - */ - allowed = true; - } - if (!allowed) { - return error(Response.Status.FORBIDDEN, - "You are not permitted to abort file uploads with the supplied parameters."); - } - try { - S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); - } catch (IOException io) { - logger.warning("Multipart upload abort failed for uploadId: " + uploadId + " storageidentifier=" - + storageidentifier + " dataset Id: " + dataset.getId()); - logger.warning(io.getMessage()); - throw new WrappedResponse(io, - error(Response.Status.INTERNAL_SERVER_ERROR, "Could not abort multipart upload")); - } - return Response.noContent().build(); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + boolean canUpdateDataset = false; + try { + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetVersionCommand.class); + } catch (WrappedResponse ex) { + logger.info("Exception thrown while trying to figure out permissions while getting upload URL for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + if (!canUpdateDataset) { + return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); + } + S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); + if (s3io == null) { + return error(Response.Status.NOT_FOUND, "Direct upload not supported for files in this dataset: " + dataset.getId()); + } + String url = null; + String storageIdentifier = null; + try { + url = s3io.generateTemporaryS3UploadUrl(); + storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); + } catch (IOException io) { + logger.warning(io.getMessage()); + throw new WrappedResponse(io, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); + } -@PUT -@Path("mpupload") -public Response completeMPUpload(String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { - try { - Dataset dataset = datasetSvc.findByGlobalId(idSupplied); - //Allow the API to be used within a session (e.g. for direct upload in the UI) - User user =session.getUser(); - if (!user.isAuthenticated()) { - try { - user=findAuthenticatedUserOrDie(); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions to complete mpupload for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - } - boolean allowed = false; - if (dataset != null) { - allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } else { - /* - * The only legitimate case where a global id won't correspond to a dataset is - * for uploads during creation. Given that this call will still fail unless all - * three parameters correspond to an active multipart upload, it should be safe - * to allow the attempt for an authenticated user. If there are concerns about - * permissions, one could check with the current design that the user is allowed - * to create datasets in some dataverse that is configured to use the storage - * provider specified in the storageidentifier, but testing for the ability to - * create a dataset in a specific dataverse would requiring changing the design - * somehow (e.g. adding the ownerId to this call). - */ - allowed = true; - } - if (!allowed) { - return error(Response.Status.FORBIDDEN, - "You are not permitted to complete file uploads with the supplied parameters."); - } - List eTagList = new ArrayList(); - logger.info("Etags: " + partETagBody); - try { - JsonReader jsonReader = Json.createReader(new StringReader(partETagBody)); - JsonObject object = jsonReader.readObject(); - jsonReader.close(); - for(String partNo : object.keySet()) { - eTagList.add(new PartETag(Integer.parseInt(partNo), object.getString(partNo))); - } - for(PartETag et: eTagList) { - logger.fine("Part: " + et.getPartNumber() + " : " + et.getETag()); - } - } catch (JsonException je) { - logger.info("Unable to parse eTags from: " + partETagBody); - throw new WrappedResponse(je, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); - } - try { - S3AccessIO.completeMultipartUpload(idSupplied, storageidentifier, uploadId, eTagList); - } catch (IOException io) { - logger.warning("Multipart upload completion failed for uploadId: " + uploadId +" storageidentifier=" + storageidentifier + " globalId: " + idSupplied); - logger.warning(io.getMessage()); - try { - S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); - } catch (IOException e) { - logger.severe("Also unable to abort the upload (and release the space on S3 for uploadId: " + uploadId +" storageidentifier=" + storageidentifier + " globalId: " + idSupplied); - logger.severe(io.getMessage()); - } - - throw new WrappedResponse(io, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); - } - return ok("Multipart Upload completed"); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + JsonObjectBuilder response = Json.createObjectBuilder() + .add("url", url) + .add("storageIdentifier", storageIdentifier); + return ok(response); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @GET + @Path("{id}/uploadurls") + public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { + try { + Dataset dataset = findDatasetOrDie(idSupplied); + + boolean canUpdateDataset = false; + try { + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions while getting upload URLs for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + if (!canUpdateDataset) { + return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); + } + S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); + if (s3io == null) { + return error(Response.Status.NOT_FOUND, + "Direct upload not supported for files in this dataset: " + dataset.getId()); + } + JsonObjectBuilder response = null; + String storageIdentifier = null; + try { + storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); + response = s3io.generateTemporaryS3UploadUrls(dataset.getGlobalId().asString(), storageIdentifier, fileSize); + + } catch (IOException io) { + logger.warning(io.getMessage()); + throw new WrappedResponse(io, + error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); + } + + response.add("storageIdentifier", storageIdentifier); + return ok(response); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @DELETE + @Path("mpupload") + public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + try { + Dataset dataset = datasetSvc.findByGlobalId(idSupplied); + //Allow the API to be used within a session (e.g. for direct upload in the UI) + User user = session.getUser(); + if (!user.isAuthenticated()) { + try { + user = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions while getting aborting upload for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + } + boolean allowed = false; + if (dataset != null) { + allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } else { + /* + * The only legitimate case where a global id won't correspond to a dataset is + * for uploads during creation. Given that this call will still fail unless all + * three parameters correspond to an active multipart upload, it should be safe + * to allow the attempt for an authenticated user. If there are concerns about + * permissions, one could check with the current design that the user is allowed + * to create datasets in some dataverse that is configured to use the storage + * provider specified in the storageidentifier, but testing for the ability to + * create a dataset in a specific dataverse would requiring changing the design + * somehow (e.g. adding the ownerId to this call). + */ + allowed = true; + } + if (!allowed) { + return error(Response.Status.FORBIDDEN, + "You are not permitted to abort file uploads with the supplied parameters."); + } + try { + S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); + } catch (IOException io) { + logger.warning("Multipart upload abort failed for uploadId: " + uploadId + " storageidentifier=" + + storageidentifier + " dataset Id: " + dataset.getId()); + logger.warning(io.getMessage()); + throw new WrappedResponse(io, + error(Response.Status.INTERNAL_SERVER_ERROR, "Could not abort multipart upload")); + } + return Response.noContent().build(); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @PUT + @Path("mpupload") + public Response completeMPUpload(String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + try { + Dataset dataset = datasetSvc.findByGlobalId(idSupplied); + //Allow the API to be used within a session (e.g. for direct upload in the UI) + User user = session.getUser(); + if (!user.isAuthenticated()) { + try { + user = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions to complete mpupload for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + } + boolean allowed = false; + if (dataset != null) { + allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } else { + /* + * The only legitimate case where a global id won't correspond to a dataset is + * for uploads during creation. Given that this call will still fail unless all + * three parameters correspond to an active multipart upload, it should be safe + * to allow the attempt for an authenticated user. If there are concerns about + * permissions, one could check with the current design that the user is allowed + * to create datasets in some dataverse that is configured to use the storage + * provider specified in the storageidentifier, but testing for the ability to + * create a dataset in a specific dataverse would requiring changing the design + * somehow (e.g. adding the ownerId to this call). + */ + allowed = true; + } + if (!allowed) { + return error(Response.Status.FORBIDDEN, + "You are not permitted to complete file uploads with the supplied parameters."); + } + List eTagList = new ArrayList(); + logger.info("Etags: " + partETagBody); + try { + JsonReader jsonReader = Json.createReader(new StringReader(partETagBody)); + JsonObject object = jsonReader.readObject(); + jsonReader.close(); + for (String partNo : object.keySet()) { + eTagList.add(new PartETag(Integer.parseInt(partNo), object.getString(partNo))); + } + for (PartETag et : eTagList) { + logger.info("Part: " + et.getPartNumber() + " : " + et.getETag()); + } + } catch (JsonException je) { + logger.info("Unable to parse eTags from: " + partETagBody); + throw new WrappedResponse(je, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); + } + try { + S3AccessIO.completeMultipartUpload(idSupplied, storageidentifier, uploadId, eTagList); + } catch (IOException io) { + logger.warning("Multipart upload completion failed for uploadId: " + uploadId + " storageidentifier=" + storageidentifier + " globalId: " + idSupplied); + logger.warning(io.getMessage()); + try { + S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); + } catch (IOException e) { + logger.severe("Also unable to abort the upload (and release the space on S3 for uploadId: " + uploadId + " storageidentifier=" + storageidentifier + " globalId: " + idSupplied); + logger.severe(io.getMessage()); + } + + throw new WrappedResponse(io, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); + } + return ok("Multipart Upload completed"); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } /** * Add a File to an existing Dataset - * + * * @param idSupplied * @param jsonData * @param fileInputStream * @param contentDispositionHeader * @param formDataBodyPart - * @return + * @return */ @POST @Path("{id}/add") @@ -2322,7 +2325,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } catch (WrappedResponse ex) { return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); + ); } @@ -2335,7 +2338,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, try { dataset = findDatasetOrDie(idSupplied); } catch (WrappedResponse wr) { - return wr.getResponse(); + return wr.getResponse(); } //------------------------------------ @@ -2354,12 +2357,12 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, // (2a) Load up optional params via JSON //--------------------------------------- OptionalFileParams optionalFileParams = null; - msgt("(api) jsonData: " + jsonData); + msgt("(api) jsonData: " + jsonData); try { optionalFileParams = new OptionalFileParams(jsonData); } catch (DataFileTagException ex) { - return error( Response.Status.BAD_REQUEST, ex.getMessage()); + return error(Response.Status.BAD_REQUEST, ex.getMessage()); } catch (ClassCastException | com.google.gson.JsonParseException ex) { return error(Response.Status.BAD_REQUEST, BundleUtil.getStringFromBundle("file.addreplace.error.parsing")); @@ -2371,42 +2374,47 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, String newFilename = null; String newFileContentType = null; String newStorageIdentifier = null; - if (null == contentDispositionHeader) { - if (optionalFileParams.hasStorageIdentifier()) { - newStorageIdentifier = optionalFileParams.getStorageIdentifier(); - // ToDo - check that storageIdentifier is valid - if (optionalFileParams.hasFileName()) { - newFilename = optionalFileParams.getFileName(); - if (optionalFileParams.hasMimetype()) { - newFileContentType = optionalFileParams.getMimeType(); - } - } - } else { - return error(BAD_REQUEST, - "You must upload a file or provide a storageidentifier, filename, and mimetype."); - } - } else { - newFilename = contentDispositionHeader.getFileName(); - // Let's see if the form data part has the mime (content) type specified. - // Note that we don't want to rely on formDataBodyPart.getMediaType() - - // because that defaults to "text/plain" when no "Content-Type:" header is - // present. Instead we'll go through the headers, and see if "Content-Type:" - // is there. If not, we'll default to "application/octet-stream" - the generic - // unknown type. This will prompt the application to run type detection and - // potentially find something more accurate. - //newFileContentType = formDataBodyPart.getMediaType().toString(); - - for (String header : formDataBodyPart.getHeaders().keySet()) { - if (header.equalsIgnoreCase("Content-Type")) { - newFileContentType = formDataBodyPart.getHeaders().get(header).get(0); - } - } - if (newFileContentType == null) { - newFileContentType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT; - } - } + if (null == contentDispositionHeader) { + if (optionalFileParams.hasStorageIdentifier()) { + newStorageIdentifier = optionalFileParams.getStorageIdentifier(); + newStorageIdentifier = DataAccess.expandStorageIdentifierIfNeeded(newStorageIdentifier); + + if(!DataAccess.uploadToDatasetAllowed(dataset, newStorageIdentifier)) { + return error(BAD_REQUEST, + "Dataset store configuration does not allow provided storageIdentifier."); + } + if (optionalFileParams.hasFileName()) { + newFilename = optionalFileParams.getFileName(); + if (optionalFileParams.hasMimetype()) { + newFileContentType = optionalFileParams.getMimeType(); + } + } + } else { + return error(BAD_REQUEST, + "You must upload a file or provide a valid storageidentifier, filename, and mimetype."); + } + } else { + newFilename = contentDispositionHeader.getFileName(); + // Let's see if the form data part has the mime (content) type specified. + // Note that we don't want to rely on formDataBodyPart.getMediaType() - + // because that defaults to "text/plain" when no "Content-Type:" header is + // present. Instead we'll go through the headers, and see if "Content-Type:" + // is there. If not, we'll default to "application/octet-stream" - the generic + // unknown type. This will prompt the application to run type detection and + // potentially find something more accurate. + // newFileContentType = formDataBodyPart.getMediaType().toString(); + + for (String header : formDataBodyPart.getHeaders().keySet()) { + if (header.equalsIgnoreCase("Content-Type")) { + newFileContentType = formDataBodyPart.getHeaders().get(header).get(0); + } + } + if (newFileContentType == null) { + newFileContentType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT; + } + } + - //------------------- // (3) Create the AddReplaceFileHelper object //------------------- @@ -2414,11 +2422,11 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, DataverseRequest dvRequest2 = createDataverseRequest(authUser); AddReplaceFileHelper addFileHelper = new AddReplaceFileHelper(dvRequest2, - ingestService, - datasetService, - fileService, - permissionSvc, - commandEngine, + ingestService, + datasetService, + fileService, + permissionSvc, + commandEngine, systemConfig, licenseSvc); @@ -2427,16 +2435,16 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, // (4) Run "runAddFileByDatasetId" //------------------- addFileHelper.runAddFileByDataset(dataset, - newFilename, - newFileContentType, - newStorageIdentifier, - fileInputStream, - optionalFileParams); + newFilename, + newFileContentType, + newStorageIdentifier, + fileInputStream, + optionalFileParams); - if (addFileHelper.hasError()){ + if (addFileHelper.hasError()) { return error(addFileHelper.getHttpErrorCode(), addFileHelper.getErrorMessagesAsString("\n")); - }else{ + } else { String successMsg = BundleUtil.getStringFromBundle("file.addreplace.success.add"); try { //msgt("as String: " + addFileHelper.getSuccessResult()); @@ -2462,71 +2470,77 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } } - + } // end: addFileToDataset - - private void msg(String m){ + private void msg(String m) { //System.out.println(m); logger.fine(m); } - private void dashes(){ + + private void dashes() { msg("----------------"); } - private void msgt(String m){ - dashes(); msg(m); dashes(); + + private void msgt(String m) { + dashes(); + msg(m); + dashes(); } - - - public static T handleVersion( String versionId, DsVersionHandler hdl ) - throws WrappedResponse { + + + public static T handleVersion(String versionId, DsVersionHandler hdl) + throws WrappedResponse { switch (versionId) { - case ":latest": return hdl.handleLatest(); - case ":draft": return hdl.handleDraft(); - case ":latest-published": return hdl.handleLatestPublished(); + case ":latest": + return hdl.handleLatest(); + case ":draft": + return hdl.handleDraft(); + case ":latest-published": + return hdl.handleLatestPublished(); default: try { String[] versions = versionId.split("\\."); switch (versions.length) { case 1: - return hdl.handleSpecific(Long.parseLong(versions[0]), (long)0.0); + return hdl.handleSpecific(Long.parseLong(versions[0]), (long) 0.0); case 2: - return hdl.handleSpecific( Long.parseLong(versions[0]), Long.parseLong(versions[1]) ); + return hdl.handleSpecific(Long.parseLong(versions[0]), Long.parseLong(versions[1])); default: - throw new WrappedResponse(error( Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); + throw new WrappedResponse(error(Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); } - } catch ( NumberFormatException nfe ) { - throw new WrappedResponse( error( Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'") ); + } catch (NumberFormatException nfe) { + throw new WrappedResponse(error(Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); } } } - - private DatasetVersion getDatasetVersionOrDie( final DataverseRequest req, String versionNumber, final Dataset ds, UriInfo uriInfo, HttpHeaders headers) throws WrappedResponse { - DatasetVersion dsv = execCommand( handleVersion(versionNumber, new DsVersionHandler>(){ - @Override - public Command handleLatest() { - return new GetLatestAccessibleDatasetVersionCommand(req, ds); - } + private DatasetVersion getDatasetVersionOrDie(final DataverseRequest req, String versionNumber, final Dataset ds, UriInfo uriInfo, HttpHeaders headers) throws WrappedResponse { + DatasetVersion dsv = execCommand(handleVersion(versionNumber, new DsVersionHandler>() { - @Override - public Command handleDraft() { - return new GetDraftDatasetVersionCommand(req, ds); - } - - @Override - public Command handleSpecific(long major, long minor) { - return new GetSpecificPublishedDatasetVersionCommand(req, ds, major, minor); - } + @Override + public Command handleLatest() { + return new GetLatestAccessibleDatasetVersionCommand(req, ds); + } - @Override - public Command handleLatestPublished() { - return new GetLatestPublishedDatasetVersionCommand(req, ds); - } - })); - if ( dsv == null || dsv.getId() == null ) { - throw new WrappedResponse( notFound("Dataset version " + versionNumber + " of dataset " + ds.getId() + " not found") ); + @Override + public Command handleDraft() { + return new GetDraftDatasetVersionCommand(req, ds); + } + + @Override + public Command handleSpecific(long major, long minor) { + return new GetSpecificPublishedDatasetVersionCommand(req, ds, major, minor); + } + + @Override + public Command handleLatestPublished() { + return new GetLatestPublishedDatasetVersionCommand(req, ds); + } + })); + if (dsv == null || dsv.getId() == null) { + throw new WrappedResponse(notFound("Dataset version " + versionNumber + " of dataset " + ds.getId() + " not found")); } if (dsv.isReleased()&& uriInfo!=null) { MakeDataCountLoggingServiceBean.MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, ds); @@ -2542,14 +2556,14 @@ public Response getLocksForDataset(@PathParam("identifier") String id, @QueryPar Dataset dataset = null; try { dataset = findDatasetOrDie(id); - Set locks; + Set locks; if (lockType == null) { locks = dataset.getLocks(); } else { // request for a specific type lock: DatasetLock lock = dataset.getLockFor(lockType); - locks = new HashSet<>(); + locks = new HashSet<>(); if (lock != null) { locks.add(lock); } @@ -2559,9 +2573,9 @@ public Response getLocksForDataset(@PathParam("identifier") String id, @QueryPar } catch (WrappedResponse wr) { return wr.getResponse(); - } - } - + } + } + @DELETE @Path("{identifier}/locks") public Response deleteLocks(@PathParam("identifier") String id, @QueryParam("type") DatasetLock.Reason lockType) { @@ -2634,7 +2648,7 @@ public Response lockDataset(@PathParam("identifier") String id, @PathParam("type AuthenticatedUser user = findAuthenticatedUserOrDie(); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "This API end point can be used by superusers only."); - } + } Dataset dataset = findDatasetOrDie(id); DatasetLock lock = dataset.getLockFor(lockType); if (lock != null) { @@ -2727,7 +2741,7 @@ public Response getMakeDataCountCitations(@PathParam("id") String idSupplied) { Dataset dataset = findDatasetOrDie(idSupplied); JsonArrayBuilder datasetsCitations = Json.createArrayBuilder(); List externalCitations = datasetExternalCitationsService.getDatasetExternalCitationsByDataset(dataset); - for (DatasetExternalCitations citation : externalCitations ){ + for (DatasetExternalCitations citation : externalCitations) { JsonObjectBuilder candidateObj = Json.createObjectBuilder(); /** * In the future we can imagine storing and presenting more @@ -2738,9 +2752,9 @@ public Response getMakeDataCountCitations(@PathParam("id") String idSupplied) { */ candidateObj.add("citationUrl", citation.getCitedByUrl()); datasetsCitations.add(candidateObj); - } - return ok(datasetsCitations); - + } + return ok(datasetsCitations); + } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -2756,20 +2770,20 @@ public Response getMakeDataCountMetricCurrentMonth(@PathParam("id") String idSup @GET @Path("{identifier}/storagesize") - public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.storage"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached,GetDatasetStorageSizeCommand.Mode.STORAGE, null))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached, GetDatasetStorageSizeCommand.Mode.STORAGE, null))))); } @GET @Path("{identifier}/versions/{versionId}/downloadsize") - public Response getDownloadSize(@PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + public Response getDownloadSize(@PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.download"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version , findDatasetOrDie(dvIdtf), uriInfo, headers)))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version, findDatasetOrDie(dvIdtf), uriInfo, headers)))))); } @GET @@ -2893,7 +2907,7 @@ public Response getFileStore(@PathParam("identifier") String dvIdtf, } catch (WrappedResponse ex) { return error(Response.Status.NOT_FOUND, "No such dataset"); } - + return response(req -> ok(dataset.getEffectiveStorageDriverId())); } @@ -2912,10 +2926,10 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf, } if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); - } - - Dataset dataset; - + } + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -2930,15 +2944,15 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf, return ok("Storage driver set to: " + store.getKey() + "/" + store.getValue()); } } - return error(Response.Status.BAD_REQUEST, - "No Storage Driver found for : " + storageDriverLabel); + return error(Response.Status.BAD_REQUEST, + "No Storage Driver found for : " + storageDriverLabel); } @DELETE @Path("{identifier}/storageDriver") public Response resetFileStore(@PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -2948,10 +2962,10 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf, } if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); - } - - Dataset dataset; - + } + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -2960,14 +2974,14 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf, dataset.setStorageDriverId(null); datasetService.merge(dataset); - return ok("Storage reset to default: " + DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER); + return ok("Storage reset to default: " + DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER); } @GET @Path("{identifier}/curationLabelSet") public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + try { AuthenticatedUser user = findAuthenticatedUserOrDie(); if (!user.isSuperuser()) { @@ -2976,24 +2990,24 @@ public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, } catch (WrappedResponse wr) { return wr.getResponse(); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { return ex.getResponse(); } - + return response(req -> ok(dataset.getEffectiveCurationLabelSetName())); } - + @PUT @Path("{identifier}/curationLabelSet") public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, @QueryParam("name") String curationLabelSet, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -3004,9 +3018,9 @@ public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -3028,12 +3042,12 @@ public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, return error(Response.Status.BAD_REQUEST, "No Such Curation Label Set"); } - + @DELETE @Path("{identifier}/curationLabelSet") public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -3044,15 +3058,15 @@ public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { return ex.getResponse(); } - + dataset.setCurationLabelSetName(SystemConfig.DEFAULTCURATIONLABELSET); datasetService.merge(dataset); return ok("Curation Label Set reset to default: " + SystemConfig.DEFAULTCURATIONLABELSET); @@ -3061,16 +3075,16 @@ public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, @GET @Path("{identifier}/allowedCurationLabels") public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { AuthenticatedUser user = null; try { user = findAuthenticatedUserOrDie(); } catch (WrappedResponse wr) { return wr.getResponse(); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -3083,7 +3097,7 @@ public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, return error(Response.Status.FORBIDDEN, "You are not permitted to view the allowed curation labels for this dataset."); } } - + @GET @Path("{identifier}/timestamps") @Produces(MediaType.APPLICATION_JSON) @@ -3113,6 +3127,7 @@ public Response getTimestamps(@PathParam("identifier") String id) { if (dataset.getLastExportTime() != null) { timestamps.add("lastMetadataExportTime", formatter.format(dataset.getLastExportTime().toInstant().atZone(ZoneId.systemDefault()))); + } if (dataset.getMostRecentMajorVersionReleaseDate() != null) { @@ -3124,11 +3139,11 @@ public Response getTimestamps(@PathParam("identifier") String id) { timestamps.add("hasStaleIndex", (dataset.getModificationTime() != null && (dataset.getIndexTime() == null || (dataset.getIndexTime().compareTo(dataset.getModificationTime()) <= 0))) ? true - : false); + : false); timestamps.add("hasStalePermissionIndex", (dataset.getPermissionModificationTime() != null && (dataset.getIndexTime() == null || (dataset.getIndexTime().compareTo(dataset.getModificationTime()) <= 0))) ? true - : false); + : false); } // More detail if you can see a draft if (canSeeDraft) { @@ -3157,6 +3172,129 @@ public Response getTimestamps(@PathParam("identifier") String id) { } + @POST + @Path("{id}/addglobusFiles") + @Consumes(MediaType.MULTIPART_FORM_DATA) + public Response addGlobusFilesToDataset(@PathParam("id") String datasetId, + @FormDataParam("jsonData") String jsonData, + @Context UriInfo uriInfo, + @Context HttpHeaders headers + ) throws IOException, ExecutionException, InterruptedException { + + logger.info(" ==== (api addGlobusFilesToDataset) jsonData ====== " + jsonData); + + if (!systemConfig.isHTTPUpload()) { + return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); + } + + // ------------------------------------- + // (1) Get the user from the API key + // ------------------------------------- + AuthenticatedUser authUser; + try { + authUser = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") + ); + } + + // ------------------------------------- + // (2) Get the Dataset Id + // ------------------------------------- + Dataset dataset; + + try { + dataset = findDatasetOrDie(datasetId); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + + //------------------------------------ + // (2b) Make sure dataset does not have package file + // -------------------------------------- + + for (DatasetVersion dv : dataset.getVersions()) { + if (dv.isHasPackageFile()) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.api.alreadyHasPackageFile") + ); + } + } + + + String lockInfoMessage = "Globus Upload API started "; + DatasetLock lock = datasetService.addDatasetLock(dataset.getId(), DatasetLock.Reason.GlobusUpload, + (authUser).getId(), lockInfoMessage); + if (lock != null) { + dataset.addLock(lock); + } else { + logger.log(Level.WARNING, "Failed to lock the dataset (dataset id={0})", dataset.getId()); + } + + + ApiToken token = authSvc.findApiTokenByUser(authUser); + + if(uriInfo != null) { + logger.info(" ==== (api uriInfo.getRequestUri()) jsonData ====== " + uriInfo.getRequestUri().toString()); + } + + + String requestUrl = headers.getRequestHeader("origin").get(0); + + if(requestUrl.contains("localhost")){ + requestUrl = "http://localhost:8080"; + } + + // Async Call + globusService.globusUpload(jsonData, token, dataset, requestUrl, authUser); + + return ok("Async call to Globus Upload started "); + + } + + @POST + @Path("{id}/deleteglobusRule") + @Consumes(MediaType.MULTIPART_FORM_DATA) + public Response deleteglobusRule(@PathParam("id") String datasetId,@FormDataParam("jsonData") String jsonData + ) throws IOException, ExecutionException, InterruptedException { + + + logger.info(" ==== (api deleteglobusRule) jsonData ====== " + jsonData); + + + if (!systemConfig.isHTTPUpload()) { + return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); + } + + // ------------------------------------- + // (1) Get the user from the API key + // ------------------------------------- + User authUser; + try { + authUser = findUserOrDie(); + } catch (WrappedResponse ex) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") + ); + } + + // ------------------------------------- + // (2) Get the Dataset Id + // ------------------------------------- + Dataset dataset; + + try { + dataset = findDatasetOrDie(datasetId); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + + // Async Call + globusService.globusDownload(jsonData, dataset, authUser); + + return ok("Async call to Globus Download started"); + + } + + /** * Add multiple Files to an existing Dataset * @@ -3196,6 +3334,9 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, return wr.getResponse(); } + dataset.getLocks().forEach(dl -> { + logger.info(dl.toString()); + }); //------------------------------------ // (2a) Make sure dataset does not have package file @@ -3225,10 +3366,10 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, return addFileHelper.addFiles(jsonData, dataset, authUser); } - - /** + + /** * API to find curation assignments and statuses - * + * * @return * @throws WrappedResponse */ @@ -3319,17 +3460,20 @@ public Response getDatasetVersionArchivalStatus(@PathParam("id") String datasetI @Consumes(MediaType.APPLICATION_JSON) @Path("/{id}/{version}/archivalStatus") public Response setDatasetVersionArchivalStatus(@PathParam("id") String datasetId, - @PathParam("version") String versionNumber, JsonObject update, @Context UriInfo uriInfo, + @PathParam("version") String versionNumber, String newStatus, @Context UriInfo uriInfo, @Context HttpHeaders headers) { - logger.fine(JsonUtil.prettyPrint(update)); + logger.fine(newStatus); try { AuthenticatedUser au = findAuthenticatedUserOrDie(); if (!au.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - + + //Verify we have valid json after removing any HTML tags (the status gets displayed in the UI, so we want plain text). + JsonObject update= JsonUtil.getJsonObject(MarkupChecker.stripAllTags(newStatus)); + if (update.containsKey(DatasetVersion.ARCHIVAL_STATUS) && update.containsKey(DatasetVersion.ARCHIVAL_STATUS_MESSAGE)) { String status = update.getString(DatasetVersion.ARCHIVAL_STATUS); if (status.equals(DatasetVersion.ARCHIVAL_STATUS_PENDING) || status.equals(DatasetVersion.ARCHIVAL_STATUS_FAILURE) @@ -3360,8 +3504,9 @@ public Response setDatasetVersionArchivalStatus(@PathParam("id") String datasetI } } catch (WrappedResponse wr) { return wr.getResponse(); + } catch (JsonException| IllegalStateException ex) { + return error(Status.BAD_REQUEST, "Unable to parse provided JSON"); } - return error(Status.BAD_REQUEST, "Unacceptable status format"); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java index d15b0f1c48f..90130cb3944 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java @@ -7,17 +7,17 @@ import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.DataverseFacet; import edu.harvard.iq.dataverse.DataverseContact; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; import edu.harvard.iq.dataverse.DataverseServiceBean; import edu.harvard.iq.dataverse.api.datadeposit.SwordServiceBean; +import edu.harvard.iq.dataverse.api.dto.DataverseMetadataBlockFacetDTO; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.DvObject; -import edu.harvard.iq.dataverse.DvObjectContainer; import edu.harvard.iq.dataverse.GlobalId; import edu.harvard.iq.dataverse.GuestbookResponseServiceBean; import edu.harvard.iq.dataverse.GuestbookServiceBean; import edu.harvard.iq.dataverse.MetadataBlock; import edu.harvard.iq.dataverse.RoleAssignment; -import static edu.harvard.iq.dataverse.api.AbstractApiBean.error; import edu.harvard.iq.dataverse.api.dto.ExplicitGroupDTO; import edu.harvard.iq.dataverse.api.dto.RoleAssignmentDTO; import edu.harvard.iq.dataverse.api.dto.RoleDTO; @@ -41,6 +41,7 @@ import edu.harvard.iq.dataverse.engine.command.impl.DeleteDataverseCommand; import edu.harvard.iq.dataverse.engine.command.impl.DeleteDataverseLinkingDataverseCommand; import edu.harvard.iq.dataverse.engine.command.impl.DeleteExplicitGroupCommand; +import edu.harvard.iq.dataverse.engine.command.impl.UpdateMetadataBlockFacetRootCommand; import edu.harvard.iq.dataverse.engine.command.impl.GetDataverseCommand; import edu.harvard.iq.dataverse.engine.command.impl.GetDataverseStorageSizeCommand; import edu.harvard.iq.dataverse.engine.command.impl.GetExplicitGroupCommand; @@ -49,6 +50,7 @@ import edu.harvard.iq.dataverse.engine.command.impl.ListDataverseContentCommand; import edu.harvard.iq.dataverse.engine.command.impl.ListExplicitGroupsCommand; import edu.harvard.iq.dataverse.engine.command.impl.ListFacetsCommand; +import edu.harvard.iq.dataverse.engine.command.impl.ListMetadataBlockFacetsCommand; import edu.harvard.iq.dataverse.engine.command.impl.ListMetadataBlocksCommand; import edu.harvard.iq.dataverse.engine.command.impl.ListRoleAssignments; import edu.harvard.iq.dataverse.engine.command.impl.ListRolesCommand; @@ -62,6 +64,7 @@ import edu.harvard.iq.dataverse.engine.command.impl.UpdateDataverseDefaultContributorRoleCommand; import edu.harvard.iq.dataverse.engine.command.impl.UpdateDataverseMetadataBlocksCommand; import edu.harvard.iq.dataverse.engine.command.impl.UpdateExplicitGroupCommand; +import edu.harvard.iq.dataverse.engine.command.impl.UpdateMetadataBlockFacetsCommand; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.ConstraintViolationUtil; @@ -69,7 +72,6 @@ import static edu.harvard.iq.dataverse.util.StringUtil.nonEmpty; import edu.harvard.iq.dataverse.util.json.JSONLDUtil; -import edu.harvard.iq.dataverse.util.json.JsonLDTerm; import edu.harvard.iq.dataverse.util.json.JsonParseException; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.brief; import java.io.StringReader; @@ -91,7 +93,6 @@ import javax.json.JsonValue; import javax.json.JsonValue.ValueType; import javax.json.stream.JsonParsingException; -import javax.validation.ConstraintViolation; import javax.validation.ConstraintViolationException; import javax.ws.rs.BadRequestException; import javax.ws.rs.Consumes; @@ -114,9 +115,9 @@ import java.text.SimpleDateFormat; import java.util.Arrays; import java.util.Date; -import java.util.HashMap; import java.util.Map; import java.util.Optional; +import java.util.stream.Collectors; import javax.servlet.http.HttpServletResponse; import javax.ws.rs.WebApplicationException; import javax.ws.rs.core.Context; @@ -713,6 +714,78 @@ public Response setFacets(@PathParam("identifier") String dvIdtf, String facetId } } + @GET + @Path("{identifier}/metadatablockfacets") + @Produces(MediaType.APPLICATION_JSON) + public Response listMetadataBlockFacets(@PathParam("identifier") String dvIdtf) { + try { + User u = findUserOrDie(); + DataverseRequest request = createDataverseRequest(u); + Dataverse dataverse = findDataverseOrDie(dvIdtf); + List metadataBlockFacets = Optional.ofNullable(execCommand(new ListMetadataBlockFacetsCommand(request, dataverse))).orElse(Collections.emptyList()); + List metadataBlocksDTOs = metadataBlockFacets.stream() + .map(item -> new DataverseMetadataBlockFacetDTO.MetadataBlockDTO(item.getMetadataBlock().getName(), item.getMetadataBlock().getLocaleDisplayFacet())) + .collect(Collectors.toList()); + DataverseMetadataBlockFacetDTO response = new DataverseMetadataBlockFacetDTO(dataverse.getId(), dataverse.getAlias(), dataverse.isMetadataBlockFacetRoot(), metadataBlocksDTOs); + return Response.ok(response).build(); + } catch (WrappedResponse e) { + return e.getResponse(); + } + } + + @POST + @Path("{identifier}/metadatablockfacets") + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + public Response setMetadataBlockFacets(@PathParam("identifier") String dvIdtf, List metadataBlockNames) { + try { + Dataverse dataverse = findDataverseOrDie(dvIdtf); + + if(!dataverse.isMetadataBlockFacetRoot()) { + return badRequest(String.format("Dataverse: %s must have metadata block facet root set to true", dvIdtf)); + } + + List metadataBlockFacets = new LinkedList<>(); + for(String metadataBlockName: metadataBlockNames) { + MetadataBlock metadataBlock = findMetadataBlock(metadataBlockName); + if (metadataBlock == null) { + return badRequest(String.format("Invalid metadata block name: %s", metadataBlockName)); + } + + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setDataverse(dataverse); + metadataBlockFacet.setMetadataBlock(metadataBlock); + metadataBlockFacets.add(metadataBlockFacet); + } + + execCommand(new UpdateMetadataBlockFacetsCommand(createDataverseRequest(findUserOrDie()), dataverse, metadataBlockFacets)); + return ok(String.format("Metadata block facets updated. DataverseId: %s blocks: %s", dvIdtf, metadataBlockNames)); + + } catch (WrappedResponse ex) { + return ex.getResponse(); + } + } + + @POST + @Path("{identifier}/metadatablockfacets/isRoot") + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + public Response updateMetadataBlockFacetsRoot(@PathParam("identifier") String dvIdtf, String body) { + try { + final boolean blockFacetsRoot = parseBooleanOrDie(body); + Dataverse dataverse = findDataverseOrDie(dvIdtf); + if(dataverse.isMetadataBlockFacetRoot() == blockFacetsRoot) { + return ok(String.format("No update needed, dataverse already consistent with new value. DataverseId: %s blockFacetsRoot: %s", dvIdtf, blockFacetsRoot)); + } + + execCommand(new UpdateMetadataBlockFacetRootCommand(createDataverseRequest(findUserOrDie()), dataverse, blockFacetsRoot)); + return ok(String.format("Metadata block facets root updated. DataverseId: %s blockFacetsRoot: %s", dvIdtf, blockFacetsRoot)); + + } catch (WrappedResponse ex) { + return ex.getResponse(); + } + } + // FIXME: This listContent method is way too optimistic, always returning "ok" and never "error". // TODO: Investigate why there was a change in the timeframe of when pull request #4350 was merged // (2438-4295-dois-for-files branch) such that a contributor API token no longer allows this method diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java index 07215cb919e..c9eb3638b90 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java @@ -11,6 +11,8 @@ import edu.harvard.iq.dataverse.EjbDataverseEngine; import edu.harvard.iq.dataverse.GuestbookResponse; import java.util.List; +import java.util.logging.Logger; + import edu.harvard.iq.dataverse.dataaccess.OptionalAccessService; import javax.faces.context.FacesContext; import javax.ws.rs.core.HttpHeaders; @@ -22,6 +24,7 @@ */ public class DownloadInstance { + private static final Logger logger = Logger.getLogger(DownloadInstance.class.getCanonicalName()); /* private ByteArrayOutputStream outStream = null; @@ -122,6 +125,7 @@ public Boolean checkIfServiceSupportedAndSetConverter(String serviceArg, String for (OptionalAccessService dataService : servicesAvailable) { if (dataService != null) { + logger.fine("Checking service: " + dataService.getServiceName()); if (serviceArg.equals("variables")) { // Special case for the subsetting parameter (variables=): if ("subset".equals(dataService.getServiceName())) { @@ -149,6 +153,7 @@ public Boolean checkIfServiceSupportedAndSetConverter(String serviceArg, String return true; } String argValuePair = serviceArg + "=" + serviceArgValue; + logger.fine("Comparing: " + argValuePair + " and " + dataService.getServiceArguments()); if (argValuePair.startsWith(dataService.getServiceArguments())) { conversionParam = serviceArg; conversionParamValue = serviceArgValue; diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java index 78fdb261d38..01f627ea23b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java @@ -27,9 +27,12 @@ import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; import edu.harvard.iq.dataverse.engine.command.impl.CreateGuestbookResponseCommand; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean.MakeDataCountEntry; import edu.harvard.iq.dataverse.util.FileUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; + import java.io.File; import java.io.FileInputStream; import java.net.URI; @@ -59,6 +62,10 @@ public class DownloadInstanceWriter implements MessageBodyWriter clazz, Type type, Annotation[] throw new NotFoundException("Datafile " + dataFile.getId() + ": Failed to locate and/or open physical file."); } + + boolean redirectSupported = false; + String auxiliaryTag = null; + String auxiliaryType = null; + String auxiliaryFileName = null; // Before we do anything else, check if this download can be handled // by a redirect to remote storage (only supported on S3, as of 5.4): - if (storageIO instanceof S3AccessIO && ((S3AccessIO) storageIO).downloadRedirectEnabled()) { + if (storageIO.downloadRedirectEnabled()) { // Even if the above is true, there are a few cases where a // redirect is not applicable. @@ -101,10 +113,8 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] // for a saved original; but CANNOT if it is a column subsetting // request (must be streamed in real time locally); or a format // conversion that hasn't been cached and saved on S3 yet. - boolean redirectSupported = true; - String auxiliaryTag = null; - String auxiliaryType = null; - String auxiliaryFileName = null; + redirectSupported = true; + if ("imageThumb".equals(di.getConversionParam())) { @@ -112,7 +122,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] int requestedSize = 0; if (!"".equals(di.getConversionParamValue())) { try { - requestedSize = new Integer(di.getConversionParamValue()); + requestedSize = Integer.parseInt(di.getConversionParamValue()); } catch (java.lang.NumberFormatException ex) { // it's ok, the default size will be used. } @@ -120,7 +130,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] auxiliaryTag = ImageThumbConverter.THUMBNAIL_SUFFIX + (requestedSize > 0 ? requestedSize : ImageThumbConverter.DEFAULT_THUMBNAIL_SIZE); - if (isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { + if (storageIO.downloadRedirectEnabled(auxiliaryTag) && isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { auxiliaryType = ImageThumbConverter.THUMBNAIL_MIME_TYPE; String fileName = storageIO.getFileName(); if (fileName != null) { @@ -139,7 +149,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] auxiliaryTag = auxiliaryTag + "_" + auxVersion; } - if (isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { + if (storageIO.downloadRedirectEnabled(auxiliaryTag) && isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { String fileExtension = getFileExtension(di.getAuxiliaryFile()); auxiliaryFileName = storageIO.getFileName() + "." + auxiliaryTag + fileExtension; auxiliaryType = di.getAuxiliaryFile().getContentType(); @@ -162,7 +172,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] // it has been cached already. auxiliaryTag = di.getConversionParamValue(); - if (isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { + if (storageIO.downloadRedirectEnabled(auxiliaryTag) && isAuxiliaryObjectCached(storageIO, auxiliaryTag)) { auxiliaryType = di.getServiceFormatType(di.getConversionParam(), auxiliaryTag); auxiliaryFileName = FileUtil.replaceExtension(storageIO.getFileName(), auxiliaryTag); } else { @@ -177,40 +187,52 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] redirectSupported = false; } } - - if (redirectSupported) { - // definitely close the (potentially still open) input stream, - // since we are not going to use it. The S3 documentation in particular - // emphasizes that it is very important not to leave these - // lying around un-closed, since they are going to fill - // up the S3 connection pool! - storageIO.closeInputStream(); - // [attempt to] redirect: - String redirect_url_str; - try { - redirect_url_str = ((S3AccessIO) storageIO).generateTemporaryS3Url(auxiliaryTag, auxiliaryType, auxiliaryFileName); - } catch (IOException ioex) { - redirect_url_str = null; - } - - if (redirect_url_str == null) { - throw new ServiceUnavailableException(); + } + String redirect_url_str=null; + + if (redirectSupported) { + // definitely close the (potentially still open) input stream, + // since we are not going to use it. The S3 documentation in particular + // emphasizes that it is very important not to leave these + // lying around un-closed, since they are going to fill + // up the S3 connection pool! + storageIO.closeInputStream(); + // [attempt to] redirect: + try { + redirect_url_str = storageIO.generateTemporaryDownloadUrl(auxiliaryTag, auxiliaryType, auxiliaryFileName); + } catch (IOException ioex) { + logger.warning("Unable to generate downloadURL for " + dataFile.getId() + ": " + auxiliaryTag); + //Setting null will let us try to get the file/aux file w/o redirecting + redirect_url_str = null; + } + } + + if (systemConfig.isGlobusFileDownload() && systemConfig.getGlobusStoresList() + .contains(DataAccess.getStorageDriverFromIdentifier(dataFile.getStorageIdentifier()))) { + if (di.getConversionParam() != null) { + if (di.getConversionParam().equals("format")) { + + if ("GlobusTransfer".equals(di.getConversionParamValue())) { + redirect_url_str = globusService.getGlobusAppUrlForDataset(dataFile.getOwner(), false, dataFile); + } } + } + if (redirect_url_str!=null) { - logger.fine("Data Access API: direct S3 url: " + redirect_url_str); + logger.fine("Data Access API: redirect url: " + redirect_url_str); URI redirect_uri; try { redirect_uri = new URI(redirect_url_str); } catch (URISyntaxException ex) { - logger.info("Data Access API: failed to create S3 redirect url (" + redirect_url_str + ")"); + logger.info("Data Access API: failed to create redirect url (" + redirect_url_str + ")"); redirect_uri = null; } if (redirect_uri != null) { // increment the download count, if necessary: if (di.getGbr() != null && !(isThumbnailDownload(di) || isPreprocessedMetadataDownload(di))) { try { - logger.fine("writing guestbook response, for an S3 download redirect."); + logger.fine("writing guestbook response, for a download redirect."); Command cmd = new CreateGuestbookResponseCommand(di.getDataverseRequestService().getDataverseRequest(), di.getGbr(), di.getGbr().getDataFile().getOwner()); di.getCommand().submit(cmd); MakeDataCountEntry entry = new MakeDataCountEntry(di.getRequestUriInfo(), di.getRequestHttpHeaders(), di.getDataverseRequestService(), di.getGbr().getDataFile()); @@ -221,7 +243,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] // finally, issue the redirect: Response response = Response.seeOther(redirect_uri).build(); - logger.fine("Issuing redirect to the file location on S3."); + logger.fine("Issuing redirect to the file location."); throw new RedirectionException(response); } throw new ServiceUnavailableException(); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Files.java b/src/main/java/edu/harvard/iq/dataverse/api/Files.java index 78847119ce4..d8313254ce0 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Files.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Files.java @@ -232,7 +232,7 @@ public Response replaceFileInDataset( } } else { return error(BAD_REQUEST, - "You must upload a file or provide a storageidentifier, filename, and mimetype."); + "You must upload a file or provide a valid storageidentifier, filename, and mimetype."); } } else { newFilename = contentDispositionHeader.getFileName(); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java new file mode 100644 index 00000000000..3912b9102e2 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -0,0 +1,195 @@ +package edu.harvard.iq.dataverse.api; + +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetServiceBean; +import edu.harvard.iq.dataverse.DataverseRoleServiceBean; +import edu.harvard.iq.dataverse.GlobalId; +import edu.harvard.iq.dataverse.MailServiceBean; +import edu.harvard.iq.dataverse.RoleAssigneeServiceBean; +import edu.harvard.iq.dataverse.RoleAssignment; +import edu.harvard.iq.dataverse.UserNotification; +import edu.harvard.iq.dataverse.UserNotificationServiceBean; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import edu.harvard.iq.dataverse.util.json.JSONLDUtil; +import edu.harvard.iq.dataverse.util.json.JsonLDNamespace; +import edu.harvard.iq.dataverse.util.json.JsonLDTerm; + +import java.util.Date; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.io.StringWriter; +import java.sql.Timestamp; +import java.util.logging.Logger; + +import javax.ejb.EJB; +import javax.json.Json; +import javax.json.JsonObject; +import javax.json.JsonValue; +import javax.json.JsonWriter; +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.BadRequestException; +import javax.ws.rs.ServiceUnavailableException; +import javax.ws.rs.Consumes; +import javax.ws.rs.ForbiddenException; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.Response; + +@Path("inbox") +public class LDNInbox extends AbstractApiBean { + + private static final Logger logger = Logger.getLogger(LDNInbox.class.getName()); + + @EJB + SettingsServiceBean settingsService; + + @EJB + DatasetServiceBean datasetService; + + @EJB + MailServiceBean mailService; + + @EJB + UserNotificationServiceBean userNotificationService; + + @EJB + DataverseRoleServiceBean roleService; + + @EJB + RoleAssigneeServiceBean roleAssigneeService; + @Context + protected HttpServletRequest httpRequest; + + @POST + @Path("/") + @Consumes("application/ld+json, application/json-ld") + public Response acceptMessage(String body) { + IpAddress origin = new DataverseRequest(null, httpRequest).getSourceAddress(); + String whitelist = settingsService.get(SettingsServiceBean.Key.LDNMessageHosts.toString(), ""); + // Only do something if we listen to this host + if (whitelist.equals("*") || whitelist.contains(origin.toString())) { + String citingPID = null; + String citingType = null; + boolean sent = false; + + JsonObject jsonld = null; + jsonld = JSONLDUtil.decontextualizeJsonLD(body); + if (jsonld == null) { + // Kludge - something about the coar notify URL causes a + // LOADING_REMOTE_CONTEXT_FAILED error in the titanium library - so replace it + // and try with a local copy + body = body.replace("\"https://purl.org/coar/notify\"", + "{\n" + " \"@vocab\": \"http://purl.org/coar/notify_vocabulary/\",\n" + + " \"ietf\": \"http://www.iana.org/assignments/relation/\",\n" + + " \"coar-notify\": \"http://purl.org/coar/notify_vocabulary/\",\n" + + " \"sorg\": \"http://schema.org/\",\n" + + " \"ReviewAction\": \"coar-notify:ReviewAction\",\n" + + " \"EndorsementAction\": \"coar-notify:EndorsementAction\",\n" + + " \"IngestAction\": \"coar-notify:IngestAction\",\n" + + " \"ietf:cite-as\": {\n" + " \"@type\": \"@id\"\n" + + " }}"); + jsonld = JSONLDUtil.decontextualizeJsonLD(body); + } + if (jsonld == null) { + throw new BadRequestException("Could not parse message to find acceptable citation link to a dataset."); + } + String relationship = "isRelatedTo"; + String name = null; + JsonLDNamespace activityStreams = JsonLDNamespace.defineNamespace("as", + "https://www.w3.org/ns/activitystreams#"); + JsonLDNamespace ietf = JsonLDNamespace.defineNamespace("ietf", "http://www.iana.org/assignments/relation/"); + String objectKey = new JsonLDTerm(activityStreams, "object").getUrl(); + if (jsonld.containsKey(objectKey)) { + JsonObject msgObject = jsonld.getJsonObject(objectKey); + + citingPID = msgObject.getJsonObject(new JsonLDTerm(ietf, "cite-as").getUrl()).getString("@id"); + logger.fine("Citing PID: " + citingPID); + if (msgObject.containsKey("@type")) { + citingType = msgObject.getString("@type"); + if (citingType.startsWith(JsonLDNamespace.schema.getUrl())) { + citingType = citingType.replace(JsonLDNamespace.schema.getUrl(), ""); + } + if (msgObject.containsKey(JsonLDTerm.schemaOrg("name").getUrl())) { + name = msgObject.getString(JsonLDTerm.schemaOrg("name").getUrl()); + } + logger.fine("Citing Type: " + citingType); + String contextKey = new JsonLDTerm(activityStreams, "context").getUrl(); + + if (jsonld.containsKey(contextKey)) { + JsonObject context = jsonld.getJsonObject(contextKey); + for (Map.Entry entry : context.entrySet()) { + + relationship = entry.getKey().replace("_:", ""); + // Assuming only one for now - should check for array and loop + JsonObject citedResource = (JsonObject) entry.getValue(); + String pid = citedResource.getJsonObject(new JsonLDTerm(ietf, "cite-as").getUrl()) + .getString("@id"); + if (citedResource.getString("@type").equals(JsonLDTerm.schemaOrg("Dataset").getUrl())) { + logger.fine("Raw PID: " + pid); + if (pid.startsWith(GlobalId.DOI_RESOLVER_URL)) { + pid = pid.replace(GlobalId.DOI_RESOLVER_URL, GlobalId.DOI_PROTOCOL + ":"); + } else if (pid.startsWith(GlobalId.HDL_RESOLVER_URL)) { + pid = pid.replace(GlobalId.HDL_RESOLVER_URL, GlobalId.HDL_PROTOCOL + ":"); + } + logger.fine("Protocol PID: " + pid); + Optional id = GlobalId.parse(pid); + Dataset dataset = datasetSvc.findByGlobalId(pid); + if (dataset != null) { + JsonObject citingResource = Json.createObjectBuilder().add("@id", citingPID) + .add("@type", citingType).add("relationship", relationship) + .add("name", name).build(); + StringWriter sw = new StringWriter(128); + try (JsonWriter jw = Json.createWriter(sw)) { + jw.write(citingResource); + } + String jsonstring = sw.toString(); + Set ras = roleService.rolesAssignments(dataset); + + roleService.rolesAssignments(dataset).stream() + .filter(ra -> ra.getRole().permissions() + .contains(Permission.PublishDataset)) + .flatMap( + ra -> roleAssigneeService + .getExplicitUsers(roleAssigneeService + .getRoleAssignee(ra.getAssigneeIdentifier())) + .stream()) + .distinct() // prevent double-send + .forEach(au -> { + + if (au.isSuperuser()) { + userNotificationService.sendNotification(au, + new Timestamp(new Date().getTime()), + UserNotification.Type.DATASETMENTIONED, dataset.getId(), + null, null, true, jsonstring); + + } + }); + sent = true; + } + } + } + } + } + } + + if (!sent) { + if (citingPID == null || citingType == null) { + throw new BadRequestException( + "Could not parse message to find acceptable citation link to a dataset."); + } else { + throw new ServiceUnavailableException( + "Unable to process message. Please contact the administrators."); + } + } + } else { + logger.info("Ignoring message from IP address: " + origin.toString()); + throw new ForbiddenException("Inbox does not acept messages from this address"); + } + return ok("Message Received"); + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Users.java b/src/main/java/edu/harvard/iq/dataverse/api/Users.java index b1177531874..d3b938af960 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Users.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Users.java @@ -83,7 +83,7 @@ public Response mergeInAuthenticatedUser(@PathParam("consumedIdentifier") String return error(Response.Status.BAD_REQUEST, "Error calling ChangeUserIdentifierCommand: " + e.getLocalizedMessage()); } - return ok("All account data for " + consumedIdentifier + " has been merged into " + baseIdentifier + " ."); + return ok(String.format("All account data for %s has been merged into %s.", consumedIdentifier, baseIdentifier)); } @POST diff --git a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordServiceBean.java index 96df3ab400a..2e093dbcf36 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordServiceBean.java @@ -9,6 +9,7 @@ import edu.harvard.iq.dataverse.TermsOfUseAndAccess; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.license.License; import edu.harvard.iq.dataverse.license.LicenseServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; @@ -163,7 +164,7 @@ public void setDatasetLicenseAndTermsOfUse(DatasetVersion datasetVersionToMutate terms.setDatasetVersion(datasetVersionToMutate); if (listOfLicensesProvided == null) { - License existingLicense = datasetVersionToMutate.getTermsOfUseAndAccess().getLicense(); + License existingLicense = DatasetUtil.getLicense(datasetVersionToMutate); if (existingLicense != null) { // leave the license alone but set terms of use setTermsOfUse(datasetVersionToMutate, dcterms, existingLicense); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/dto/DataverseMetadataBlockFacetDTO.java b/src/main/java/edu/harvard/iq/dataverse/api/dto/DataverseMetadataBlockFacetDTO.java new file mode 100644 index 00000000000..65b6f0ff58f --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/dto/DataverseMetadataBlockFacetDTO.java @@ -0,0 +1,56 @@ +package edu.harvard.iq.dataverse.api.dto; + +import java.util.List; + +/** + * + * @author adaybujeda + */ +public class DataverseMetadataBlockFacetDTO { + + private Long dataverseId; + private String dataverseAlias; + private boolean isMetadataBlockFacetRoot; + private List metadataBlocks; + + public DataverseMetadataBlockFacetDTO(Long dataverseId, String dataverseAlias, boolean isMetadataBlockFacetRoot, List metadataBlocks) { + this.dataverseId = dataverseId; + this.dataverseAlias = dataverseAlias; + this.isMetadataBlockFacetRoot = isMetadataBlockFacetRoot; + this.metadataBlocks = metadataBlocks; + } + + public Long getDataverseId() { + return dataverseId; + } + + public String getDataverseAlias() { + return dataverseAlias; + } + + public boolean isMetadataBlockFacetRoot() { + return isMetadataBlockFacetRoot; + } + + public List getMetadataBlocks() { + return metadataBlocks; + } + + public static class MetadataBlockDTO { + private String metadataBlockName; + private String metadataBlockFacet; + + public MetadataBlockDTO(String metadataBlockName, String metadataBlockFacet) { + this.metadataBlockName = metadataBlockName; + this.metadataBlockFacet = metadataBlockFacet; + } + + public String getMetadataBlockName() { + return metadataBlockName; + } + + public String getMetadataBlockFacet() { + return metadataBlockFacet; + } + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java index a92e33e223e..a4e78b33a3c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java @@ -1181,7 +1181,7 @@ private void processDataAccs(XMLStreamReader xmlr, DatasetVersionDTO dvDTO) thro String noteType = xmlr.getAttributeValue(null, "type"); if (NOTE_TYPE_TERMS_OF_USE.equalsIgnoreCase(noteType) ) { if ( LEVEL_DV.equalsIgnoreCase(xmlr.getAttributeValue(null, "level"))) { - parseText(xmlr, "notes"); + dvDTO.setTermsOfUse(parseText(xmlr, "notes")); } } else if (NOTE_TYPE_TERMS_OF_ACCESS.equalsIgnoreCase(noteType) ) { if (LEVEL_DV.equalsIgnoreCase(xmlr.getAttributeValue(null, "level"))) { diff --git a/src/main/java/edu/harvard/iq/dataverse/api/util/JsonResponseBuilder.java b/src/main/java/edu/harvard/iq/dataverse/api/util/JsonResponseBuilder.java index 07cf21934d4..aef17d1ab34 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/util/JsonResponseBuilder.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/util/JsonResponseBuilder.java @@ -222,6 +222,7 @@ public JsonResponseBuilder log(Logger logger, Level level, Optional e metadata.deleteCharAt(metadata.length()-1); if (ex.isPresent()) { + ex.get().printStackTrace(); metadata.append("|"); logger.log(level, metadata.toString(), ex); if(includeStackTrace) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java index 142420bc7d9..5c0f3a49f76 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java @@ -496,6 +496,7 @@ public void displayNotification() { case GRANTFILEACCESS: case REJECTFILEACCESS: case DATASETCREATED: + case DATASETMENTIONED: userNotification.setTheObject(datasetService.find(userNotification.getObjectId())); break; @@ -522,6 +523,13 @@ public void displayNotification() { userNotification.setTheObject(datasetVersionService.find(userNotification.getObjectId())); break; + case GLOBUSUPLOADCOMPLETED: + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + case GLOBUSDOWNLOADCOMPLETED: + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + userNotification.setTheObject(datasetService.find(userNotification.getObjectId())); + break; + case CHECKSUMIMPORT: userNotification.setTheObject(datasetVersionService.find(userNotification.getObjectId())); break; diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java index ca247d0c9c2..3e986a15689 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java @@ -64,6 +64,8 @@ public enum DevShibAccountType { UID_WITH_LEADING_SPACE, IDENTIFIER_WITH_LEADING_SPACE, MISSING_REQUIRED_ATTR, + ONE_AFFILIATION, + TWO_AFFILIATIONS, }; public DevShibAccountType getDevShibAccountType() { @@ -146,6 +148,14 @@ public void possiblyMutateRequestInDev(HttpServletRequest request) { ShibUtil.mutateRequestForDevConstantMissingRequiredAttributes(request); break; + case ONE_AFFILIATION: + ShibUtil.mutateRequestForDevConstantOneAffiliation(request); + break; + + case TWO_AFFILIATIONS: + ShibUtil.mutateRequestForDevConstantTwoAffiliations(request); + break; + default: logger.info("Should never reach here"); break; diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibUtil.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibUtil.java index 8d523ceae2f..f8b30710656 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibUtil.java @@ -261,6 +261,28 @@ static void mutateRequestForDevConstantMissingRequiredAttributes(HttpServletRequ request.setAttribute(ShibUtil.usernameAttribute, "missing"); } + static void mutateRequestForDevConstantOneAffiliation(HttpServletRequest request) { + request.setAttribute(ShibUtil.shibIdpAttribute, "https://fake.example.com/idp/shibboleth"); + request.setAttribute(ShibUtil.uniquePersistentIdentifier, "oneAffiliation"); + request.setAttribute(ShibUtil.firstNameAttribute, "Lurneen"); + request.setAttribute(ShibUtil.lastNameAttribute, "Lumpkin"); + request.setAttribute(ShibUtil.emailAttribute, "oneAffiliaton@mailinator.com"); + request.setAttribute(ShibUtil.usernameAttribute, "oneAffiliaton"); + // Affiliation. "ou" is the suggested attribute in :ShibAffiliationAttribute. + request.setAttribute("ou", "Beer-N-Brawl"); + } + + static void mutateRequestForDevConstantTwoAffiliations(HttpServletRequest request) { + request.setAttribute(ShibUtil.shibIdpAttribute, "https://fake.example.com/idp/shibboleth"); + request.setAttribute(ShibUtil.uniquePersistentIdentifier, "twoAffiliatons"); + request.setAttribute(ShibUtil.firstNameAttribute, "Lenny"); + request.setAttribute(ShibUtil.lastNameAttribute, "Leonard"); + request.setAttribute(ShibUtil.emailAttribute, "twoAffiliatons@mailinator.com"); + request.setAttribute(ShibUtil.usernameAttribute, "twoAffiliatons"); + // Affiliation. "ou" is the suggested attribute in :ShibAffiliationAttribute. + request.setAttribute("ou", "SNPP;Stonecutters"); + } + public static Map getRandomUserStatic() { Map fakeUser = new HashMap<>(); String shortRandomString = UUID.randomUUID().toString().substring(0, 8); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/DataAccess.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/DataAccess.java index a422a825259..d046fa4661d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/DataAccess.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/DataAccess.java @@ -22,6 +22,8 @@ import edu.harvard.iq.dataverse.Dataset; import edu.harvard.iq.dataverse.DvObject; +import edu.harvard.iq.dataverse.util.FileUtil; + import java.io.IOException; import java.util.HashMap; import java.util.Properties; @@ -42,9 +44,16 @@ public DataAccess() { }; + public static final String FILE = "file"; + public static final String S3 = "s3"; + static final String SWIFT = "swift"; + static final String REMOTE = "remote"; + static final String TMP = "tmp"; + public static final String SEPARATOR = "://"; //Default to "file" is for tests only - public static final String DEFAULT_STORAGE_DRIVER_IDENTIFIER = System.getProperty("dataverse.files.storage-driver-id", "file"); + public static final String DEFAULT_STORAGE_DRIVER_IDENTIFIER = System.getProperty("dataverse.files.storage-driver-id", FILE); public static final String UNDEFINED_STORAGE_DRIVER_IDENTIFIER = "undefined"; //Used in dataverse.xhtml as a non-null selection option value (indicating a null driver/inheriting the default) + // The getStorageIO() methods initialize StorageIO objects for // datafiles that are already saved using one of the supported Dataverse @@ -53,41 +62,55 @@ public static StorageIO getStorageIO(T dvObject) throws return getStorageIO(dvObject, null); } - //passing DVObject instead of a datafile to accomodate for use of datafiles as well as datasets - public static StorageIO getStorageIO(T dvObject, DataAccessRequest req) throws IOException { + - if (dvObject == null - || dvObject.getStorageIdentifier() == null - || dvObject.getStorageIdentifier().isEmpty()) { - throw new IOException("getDataAccessObject: null or invalid datafile."); - } - String storageIdentifier = dvObject.getStorageIdentifier(); - int separatorIndex = storageIdentifier.indexOf("://"); - String storageDriverId = DEFAULT_STORAGE_DRIVER_IDENTIFIER; //default - if(separatorIndex>0) { - storageDriverId = storageIdentifier.substring(0,separatorIndex); - } - String storageType = getDriverType(storageDriverId); - switch(storageType) { - case "file": - return new FileAccessIO<>(dvObject, req, storageDriverId); - case "s3": - return new S3AccessIO<>(dvObject, req, storageDriverId); - case "swift": - return new SwiftAccessIO<>(dvObject, req, storageDriverId); - case "tmp": - throw new IOException("DataAccess IO attempted on a temporary file that hasn't been permanently saved yet."); + public static String getStorageDriverFromIdentifier(String storageIdentifier) { + + int separatorIndex = storageIdentifier.indexOf(SEPARATOR); + String driverId = DEFAULT_STORAGE_DRIVER_IDENTIFIER; // default + if (separatorIndex > 0) { + driverId = storageIdentifier.substring(0, separatorIndex); } + return driverId; + } + + //passing DVObject instead of a datafile to accomodate for use of datafiles as well as datasets + public static StorageIO getStorageIO(T dvObject, DataAccessRequest req) throws IOException { - // TODO: - // This code will need to be extended with a system of looking up - // available storage plugins by the storage tag embedded in the - // "storage identifier". - // -- L.A. 4.0.2 + if (dvObject == null || dvObject.getStorageIdentifier() == null || dvObject.getStorageIdentifier().isEmpty()) { + throw new IOException("getDataAccessObject: null or invalid datafile."); + } - logger.warning("Could not find storage driver for: " + storageIdentifier); - throw new IOException("getDataAccessObject: Unsupported storage method."); - } + String storageDriverId = getStorageDriverFromIdentifier(dvObject.getStorageIdentifier()); + + return getStorageIO(dvObject, req, storageDriverId); + } + + protected static StorageIO getStorageIO(T dvObject, DataAccessRequest req, + String storageDriverId) throws IOException { + String storageType = getDriverType(storageDriverId); + switch (storageType) { + case FILE: + return new FileAccessIO<>(dvObject, req, storageDriverId); + case S3: + return new S3AccessIO<>(dvObject, req, storageDriverId); + case SWIFT: + return new SwiftAccessIO<>(dvObject, req, storageDriverId); + case REMOTE: + return new RemoteOverlayAccessIO<>(dvObject, req, storageDriverId); + case TMP: + throw new IOException( + "DataAccess IO attempted on a temporary file that hasn't been permanently saved yet."); + } + // TODO: + // This code will need to be extended with a system of looking up + // available storage plugins by the storage tag embedded in the + // "storage identifier". + // -- L.A. 4.0.2 + + logger.warning("Could not find storage driver for: " + storageDriverId); + throw new IOException("getDataAccessObject: Unsupported storage method."); + } // Experimental extension of the StorageIO system allowing direct access to // stored physical files that may not be associated with any DvObjects @@ -98,12 +121,14 @@ public static StorageIO getDirectStorageIO(String fullStorageLocation) String storageLocation=response[1]; String storageType = getDriverType(storageDriverId); switch(storageType) { - case "file": + case FILE: return new FileAccessIO<>(storageLocation, storageDriverId); - case "s3": + case S3: return new S3AccessIO<>(storageLocation, storageDriverId); - case "swift": + case SWIFT: return new SwiftAccessIO<>(storageLocation, storageDriverId); + case REMOTE: + return new RemoteOverlayAccessIO<>(storageLocation, storageDriverId); default: logger.warning("Could not find storage driver for: " + fullStorageLocation); throw new IOException("getDirectStorageIO: Unsupported storage method."); @@ -113,7 +138,7 @@ public static StorageIO getDirectStorageIO(String fullStorageLocation) public static String[] getDriverIdAndStorageLocation(String storageLocation) { //default if no prefix String storageIdentifier=storageLocation; - int separatorIndex = storageLocation.indexOf("://"); + int separatorIndex = storageLocation.indexOf(SEPARATOR); String storageDriverId = ""; //default if(separatorIndex>0) { storageDriverId = storageLocation.substring(0,separatorIndex); @@ -122,11 +147,11 @@ public static String[] getDriverIdAndStorageLocation(String storageLocation) { return new String[]{storageDriverId, storageIdentifier}; } - public static String getStorarageIdFromLocation(String location) { - if(location.contains("://")) { + public static String getStorageIdFromLocation(String location) { + if(location.contains(SEPARATOR)) { //It's a full location with a driverId, so strip and reapply the driver id //NOte that this will strip the bucketname out (which s3 uses) but the S3IOStorage class knows to look at re-insert it - return location.substring(0,location.indexOf("://") +3) + location.substring(location.lastIndexOf('/')+1); + return location.substring(0,location.indexOf(SEPARATOR) +3) + location.substring(location.lastIndexOf('/')+1); } return location.substring(location.lastIndexOf('/')+1); } @@ -137,6 +162,27 @@ public static String getDriverType(String driverId) { } return System.getProperty("dataverse.files." + driverId + ".type", "Undefined"); } + + //This + public static String getDriverPrefix(String driverId) throws IOException { + if(driverId.isEmpty() || driverId.equals("tmp")) { + return "tmp" + SEPARATOR; + } + String storageType = System.getProperty("dataverse.files." + driverId + ".type", "Undefined"); + switch(storageType) { + case FILE: + return FileAccessIO.getDriverPrefix(driverId); + case S3: + return S3AccessIO.getDriverPrefix(driverId); + case SWIFT: + return SwiftAccessIO.getDriverPrefix(driverId); + default: + logger.warning("Could not find storage driver for id: " + driverId); + throw new IOException("getDriverPrefix: Unsupported storage method."); + } + + + } // createDataAccessObject() methods create a *new*, empty DataAccess objects, // for saving new, not yet saved datafiles. @@ -167,7 +213,7 @@ public static StorageIO createNewStorageIO(T dvObject, S * This if will catch any cases where that's attempted. */ // Tests send objects with no storageIdentifier set - if((dvObject.getStorageIdentifier()!=null) && dvObject.getStorageIdentifier().contains("://")) { + if((dvObject.getStorageIdentifier()!=null) && dvObject.getStorageIdentifier().contains(SEPARATOR)) { throw new IOException("Attempt to create new StorageIO for already stored object: " + dvObject.getStorageIdentifier()); } @@ -180,15 +226,18 @@ public static StorageIO createNewStorageIO(T dvObject, S } String storageType = getDriverType(storageDriverId); switch(storageType) { - case "file": + case FILE: storageIO = new FileAccessIO<>(dvObject, null, storageDriverId); break; - case "swift": + case SWIFT: storageIO = new SwiftAccessIO<>(dvObject, null, storageDriverId); break; - case "s3": + case S3: storageIO = new S3AccessIO<>(dvObject, null, storageDriverId); break; + case REMOTE: + storageIO = createNewStorageIO(dvObject, storageTag, RemoteOverlayAccessIO.getBaseStoreIdFor(storageDriverId)) ; + break; default: logger.warning("Could not find storage driver for: " + storageTag); throw new IOException("createDataAccessObject: Unsupported storage method " + storageDriverId); @@ -250,4 +299,79 @@ public static String getStorageDriverLabelFor(String storageDriverId) { } return label; } + + /** + * This method checks to see if an overlay store is being used and, if so, + * defines a base storage identifier for use with auxiliary files, and adds it + * into the returned value + * + * @param newStorageIdentifier + * @return - the newStorageIdentifier (for file, S3, swift stores) - the + * newStorageIdentifier with a new base store identifier inserted (for + * an overlay store) + */ + public static String expandStorageIdentifierIfNeeded(String newStorageIdentifier) { + logger.fine("found: " + newStorageIdentifier); + String driverType = DataAccess + .getDriverType(newStorageIdentifier.substring(0, newStorageIdentifier.indexOf(":"))); + logger.fine("drivertype: " + driverType); + if (driverType.equals(REMOTE)) { + // Add a generated identifier for the aux files + logger.fine("in: " + newStorageIdentifier); + int lastColon = newStorageIdentifier.lastIndexOf(SEPARATOR); + newStorageIdentifier = newStorageIdentifier.substring(0, lastColon + 3) + + FileUtil.generateStorageIdentifier() + "//" + newStorageIdentifier.substring(lastColon + 3); + logger.fine("out: " + newStorageIdentifier); + } + return newStorageIdentifier; + } + + public static boolean uploadToDatasetAllowed(Dataset d, String storageIdentifier) { + boolean allowed=true; + String driverId = DataAccess.getStorageDriverFromIdentifier(storageIdentifier); + String effectiveDriverId = d.getEffectiveStorageDriverId(); + if(!effectiveDriverId.equals(driverId)) { + //Not allowed unless this is a remote store and you're uploading to the basestore + if(getDriverType(driverId).equals(REMOTE)) { + String baseDriverId = RemoteOverlayAccessIO.getBaseStoreIdFor(driverId); + if(!effectiveDriverId.equals(baseDriverId)) { + //Not allowed - wrong base driver + allowed = false; + } else { + //Only allowed if baseStore allows it + allowed = StorageIO.isDirectUploadEnabled(baseDriverId); + } + } else { + //Not allowed - wrong main driver + allowed=false; + } + } else { + //Only allowed if main store allows it + allowed = StorageIO.isDirectUploadEnabled(driverId); + } + return allowed; + } + + + //Method to verify that a submitted storageIdentifier (i.e. in direct/remote uploads) is consistent with the store's configuration. + public static boolean isValidDirectStorageIdentifier(String storageId) { + String driverId = DataAccess.getStorageDriverFromIdentifier(storageId); + String storageType = DataAccess.getDriverType(driverId); + if (storageType.equals("tmp") || storageType.equals("Undefined")) { + return false; + } + switch (storageType) { + case FILE: + return FileAccessIO.isValidIdentifier(driverId, storageId); + case SWIFT: + return SwiftAccessIO.isValidIdentifier(driverId, storageId); + case S3: + return S3AccessIO.isValidIdentifier(driverId, storageId); + case REMOTE: + return RemoteOverlayAccessIO.isValidIdentifier(driverId, storageId); + default: + logger.warning("Request to validate for storage driver: " + driverId); + } + return false; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java index bd0549622f0..d5f00b9868f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java @@ -29,10 +29,13 @@ import java.io.FileOutputStream; // NIO imports: import java.nio.file.Files; +import java.nio.file.InvalidPathException; import java.nio.file.Path; import java.nio.file.Paths; import java.util.List; - +import java.util.logging.Logger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; // Dataverse imports: import edu.harvard.iq.dataverse.DataFile; @@ -49,11 +52,14 @@ public class FileAccessIO extends StorageIO { - public FileAccessIO() { - //Constructor only for testing - super(null, null, null); - } - + private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.dataaccess.FileAccessIO"); + + + public FileAccessIO() { + // Constructor only for testing + super(null, null, null); + } + public FileAccessIO(T dvObject, DataAccessRequest req, String driverId ) { super(dvObject, req, driverId); @@ -64,8 +70,9 @@ public FileAccessIO(T dvObject, DataAccessRequest req, String driverId ) { // "Direct" File Access IO, opened on a physical file not associated with // a specific DvObject public FileAccessIO(String storageLocation, String driverId) { - super(storageLocation, driverId); - this.setIsLocalFile(true); + super(storageLocation, driverId); + this.setIsLocalFile(true); + logger.fine("Storage path: " + storageLocation); physicalPath = Paths.get(storageLocation); } @@ -120,10 +127,10 @@ public void open (DataAccessOption... options) throws IOException { } } else if (isWriteAccess) { // Creates a new directory as needed for a dataset. - Path datasetPath=Paths.get(getDatasetDirectory()); - if (datasetPath != null && !Files.exists(datasetPath)) { - Files.createDirectories(datasetPath); - } + Path datasetPath=Paths.get(getDatasetDirectory()); + if (datasetPath != null && !Files.exists(datasetPath)) { + Files.createDirectories(datasetPath); + } FileOutputStream fout = openLocalFileAsOutputStream(); if (fout == null) { @@ -132,8 +139,8 @@ public void open (DataAccessOption... options) throws IOException { this.setOutputStream(fout); setChannel(fout.getChannel()); - if (!storageIdentifier.startsWith(this.driverId + "://")) { - dvObject.setStorageIdentifier(this.driverId + "://" + storageIdentifier); + if (!storageIdentifier.startsWith(this.driverId + DataAccess.SEPARATOR)) { + dvObject.setStorageIdentifier(this.driverId + DataAccess.SEPARATOR + storageIdentifier); } } @@ -159,17 +166,22 @@ public void open (DataAccessOption... options) throws IOException { // this.setInputStream(fin); } else if (isWriteAccess) { //this checks whether a directory for a dataset exists - Path datasetPath=Paths.get(getDatasetDirectory()); - if (datasetPath != null && !Files.exists(datasetPath)) { - Files.createDirectories(datasetPath); - } - dataset.setStorageIdentifier(this.driverId + "://"+dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage()); + Path datasetPath=Paths.get(getDatasetDirectory()); + if (datasetPath != null && !Files.exists(datasetPath)) { + Files.createDirectories(datasetPath); + } + dataset.setStorageIdentifier(this.driverId + DataAccess.SEPARATOR + dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage()); } } else if (dvObject instanceof Dataverse) { dataverse = this.getDataverse(); } else { - throw new IOException("Data Access: Invalid DvObject type"); + logger.fine("Overlay case: FileAccessIO open for : " + physicalPath.toString()); + Path datasetPath= physicalPath.getParent(); + if (datasetPath != null && !Files.exists(datasetPath)) { + Files.createDirectories(datasetPath); + } + //throw new IOException("Data Access: Invalid DvObject type"); } // This "status" is a leftover from 3.6; we don't have a use for it // in 4.0 yet; and we may not need it at all. @@ -232,7 +244,7 @@ public Channel openAuxChannel(String auxItemTag, DataAccessOption... options) th Path auxPath = getAuxObjectAsPath(auxItemTag); if (isWriteAccessRequested(options)) { - if (dvObject instanceof Dataset && !this.canWrite()) { + if (((dvObject instanceof Dataset) || isDirectAccess()) && !this.canWrite()) { // If this is a dataset-level auxilary file (a cached metadata export, // dataset logo, etc.) there's a chance that no "real" files // have been saved for this dataset yet, and thus the filesystem @@ -293,7 +305,10 @@ public Path getAuxObjectAsPath(String auxItemTag) throws IOException { if (auxItemTag == null || "".equals(auxItemTag)) { throw new IOException("Null or invalid Auxiliary Object Tag."); } - + if(isDirectAccess()) { + //includes overlay case + return Paths.get(physicalPath.toString() + "." + auxItemTag); + } String datasetDirectory = getDatasetDirectory(); if (dvObject.getStorageIdentifier() == null || "".equals(dvObject.getStorageIdentifier())) { @@ -317,7 +332,7 @@ public Path getAuxObjectAsPath(String auxItemTag) throws IOException { } - @Override + @Override public void backupAsAux(String auxItemTag) throws IOException { Path auxPath = getAuxObjectAsPath(auxItemTag); @@ -415,8 +430,8 @@ public void deleteAllAuxObjects() throws IOException { } } - - + + @Override public String getStorageLocation() { // For a local file, the "storage location" is a complete, absolute @@ -425,7 +440,7 @@ public String getStorageLocation() { try { Path testPath = getFileSystemPath(); if (testPath != null) { - return this.driverId + "://" + testPath.toString(); + return this.driverId + DataAccess.SEPARATOR + testPath.toString(); } } catch (IOException ioex) { // just return null, below: @@ -545,7 +560,7 @@ public FileOutputStream openLocalFileAsOutputStream () { } private String getDatasetDirectory() throws IOException { - if (dvObject == null) { + if (isDirectAccess()) { throw new IOException("No DvObject defined in the Data Access Object"); } @@ -572,14 +587,10 @@ private String getDatasetDirectory() throws IOException { } - private String getFilesRootDirectory() { - String filesRootDirectory = System.getProperty("dataverse.files." + this.driverId + ".directory"); - - if (filesRootDirectory == null || filesRootDirectory.equals("")) { - filesRootDirectory = "/tmp/files"; - } - return filesRootDirectory; - } + protected String getFilesRootDirectory() { + String filesRootDirectory = System.getProperty("dataverse.files." + this.driverId + ".directory", "/tmp/files"); + return filesRootDirectory; + } private List listCachedFiles() throws IOException { List auxItems = new ArrayList<>(); @@ -642,10 +653,34 @@ public InputStream getAuxFileAsInputStream(String auxItemTag) throws IOException return in; } private String stripDriverId(String storageIdentifier) { - int separatorIndex = storageIdentifier.indexOf("://"); - if(separatorIndex>0) { - return storageIdentifier.substring(separatorIndex + 3); + int separatorIndex = storageIdentifier.indexOf(DataAccess.SEPARATOR); + if(separatorIndex>0) { + return storageIdentifier.substring(separatorIndex + DataAccess.SEPARATOR.length()); + } + return storageIdentifier; + } + + //Confirm inputs are of the form of a relative file path that doesn't contain . or .. + protected static boolean isValidIdentifier(String driverId, String storageId) { + String pathString = storageId.substring(storageId.lastIndexOf("//") + 2); + String basePath = "/tmp/"; + try { + String rawPathString = basePath + pathString; + Path normalized = Paths.get(rawPathString).normalize(); + if(!rawPathString.equals(normalized.toString())) { + logger.warning("Non-normalized path in submitted identifier " + storageId); + return false; + } + logger.fine(normalized.getFileName().toString()); + if (!usesStandardNamePattern(normalized.getFileName().toString())) { + logger.warning("Unacceptable file name in submitted identifier: " + storageId); + return false; + } + + } catch (InvalidPathException ipe) { + logger.warning("Invalid Path in submitted identifier " + storageId); + return false; } - return storageIdentifier; - } + return true; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/ImageThumbConverter.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/ImageThumbConverter.java index 3197234c5ea..2b4aed3a9a5 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/ImageThumbConverter.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/ImageThumbConverter.java @@ -369,7 +369,7 @@ private static boolean isThumbnailCached(StorageIO storageIO, int size try { cached = storageIO.isAuxObjectCached(THUMBNAIL_SUFFIX + size); } catch (Exception ioex) { - logger.fine("caught Exception while checking for a cached thumbnail (file " + storageIO.getDataFile().getStorageIdentifier() + ")"); + logger.fine("caught Exception while checking for a cached thumbnail (file " + storageIO.getDataFile().getStorageIdentifier() + "): " + ioex.getMessage()); return false; } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java new file mode 100644 index 00000000000..c8e42349318 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java @@ -0,0 +1,634 @@ +package edu.harvard.iq.dataverse.dataaccess; + +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DvObject; +import edu.harvard.iq.dataverse.datavariable.DataVariable; +import edu.harvard.iq.dataverse.util.UrlSignerUtil; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.MalformedURLException; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.channels.Channel; +import java.nio.channels.Channels; +import java.nio.channels.ReadableByteChannel; +import java.nio.channels.WritableByteChannel; +import java.nio.file.Path; +import java.security.KeyManagementException; +import java.security.KeyStoreException; +import java.security.NoSuchAlgorithmException; +import java.util.List; +import java.util.logging.Logger; + +import org.apache.http.Header; +import org.apache.http.client.config.CookieSpecs; +import org.apache.http.client.config.RequestConfig; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpDelete; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.client.methods.HttpHead; +import org.apache.http.client.protocol.HttpClientContext; +import org.apache.http.config.Registry; +import org.apache.http.config.RegistryBuilder; +import org.apache.http.conn.socket.ConnectionSocketFactory; +import org.apache.http.conn.ssl.NoopHostnameVerifier; +import org.apache.http.conn.ssl.SSLConnectionSocketFactory; +import org.apache.http.conn.ssl.TrustAllStrategy; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; +import org.apache.http.protocol.HTTP; +import org.apache.http.ssl.SSLContextBuilder; +import org.apache.http.util.EntityUtils; + +import javax.net.ssl.SSLContext; + +/** + * @author qqmyers + * @param what it stores + */ +/* + * Remote Overlay Driver + * + * StorageIdentifier format: + * ://// + */ +public class RemoteOverlayAccessIO extends StorageIO { + + private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO"); + + private StorageIO baseStore = null; + private String urlPath = null; + private String baseUrl = null; + + private static HttpClientContext localContext = HttpClientContext.create(); + private PoolingHttpClientConnectionManager cm = null; + CloseableHttpClient httpclient = null; + private int timeout = 1200; + private RequestConfig config = RequestConfig.custom().setConnectTimeout(timeout * 1000) + .setConnectionRequestTimeout(timeout * 1000).setSocketTimeout(timeout * 1000) + .setCookieSpec(CookieSpecs.STANDARD).setExpectContinueEnabled(true).build(); + private static boolean trustCerts = false; + private int httpConcurrency = 4; + + public RemoteOverlayAccessIO(T dvObject, DataAccessRequest req, String driverId) throws IOException { + super(dvObject, req, driverId); + this.setIsLocalFile(false); + configureStores(req, driverId, null); + logger.fine("Parsing storageidentifier: " + dvObject.getStorageIdentifier()); + urlPath = dvObject.getStorageIdentifier().substring(dvObject.getStorageIdentifier().lastIndexOf("//") + 2); + validatePath(urlPath); + + logger.fine("Base URL: " + urlPath); + } + + public RemoteOverlayAccessIO(String storageLocation, String driverId) throws IOException { + super(null, null, driverId); + this.setIsLocalFile(false); + configureStores(null, driverId, storageLocation); + + urlPath = storageLocation.substring(storageLocation.lastIndexOf("//") + 2); + validatePath(urlPath); + logger.fine("Base URL: " + urlPath); + } + + private void validatePath(String path) throws IOException { + try { + URI absoluteURI = new URI(baseUrl + "/" + urlPath); + if(!absoluteURI.normalize().toString().startsWith(baseUrl)) { + throw new IOException("storageidentifier doesn't start with " + this.driverId + "'s base-url"); + } + } catch(URISyntaxException use) { + throw new IOException("Could not interpret storageidentifier in remote store " + this.driverId); + } + } + + + @Override + public void open(DataAccessOption... options) throws IOException { + + baseStore.open(options); + + DataAccessRequest req = this.getRequest(); + + if (isWriteAccessRequested(options)) { + isWriteAccess = true; + isReadAccess = false; + } else { + isWriteAccess = false; + isReadAccess = true; + } + + if (dvObject instanceof DataFile) { + String storageIdentifier = dvObject.getStorageIdentifier(); + + DataFile dataFile = this.getDataFile(); + + if (req != null && req.getParameter("noVarHeader") != null) { + baseStore.setNoVarHeader(true); + } + + if (storageIdentifier == null || "".equals(storageIdentifier)) { + throw new FileNotFoundException("Data Access: No local storage identifier defined for this datafile."); + } + + // Fix new DataFiles: DataFiles that have not yet been saved may use this method + // when they don't have their storageidentifier in the final form + // So we fix it up here. ToDo: refactor so that storageidentifier is generated + // by the appropriate StorageIO class and is final from the start. + logger.fine("StorageIdentifier is: " + storageIdentifier); + + if (isReadAccess) { + if (dataFile.getFilesize() >= 0) { + this.setSize(dataFile.getFilesize()); + } else { + logger.fine("Setting size"); + this.setSize(getSizeFromHttpHeader()); + } + if (dataFile.getContentType() != null && dataFile.getContentType().equals("text/tab-separated-values") + && dataFile.isTabularData() && dataFile.getDataTable() != null && (!this.noVarHeader())) { + + List datavariables = dataFile.getDataTable().getDataVariables(); + String varHeaderLine = generateVariableHeader(datavariables); + this.setVarHeader(varHeaderLine); + } + + } + + this.setMimeType(dataFile.getContentType()); + + try { + this.setFileName(dataFile.getFileMetadata().getLabel()); + } catch (Exception ex) { + this.setFileName("unknown"); + } + } else if (dvObject instanceof Dataset) { + throw new IOException( + "Data Access: RemoteOverlay Storage driver does not support dvObject type Dataverse yet"); + } else if (dvObject instanceof Dataverse) { + throw new IOException( + "Data Access: RemoteOverlay Storage driver does not support dvObject type Dataverse yet"); + } else { + this.setSize(getSizeFromHttpHeader()); + } + } + + private long getSizeFromHttpHeader() { + long size = -1; + HttpHead head = new HttpHead(baseUrl + "/" + urlPath); + try { + CloseableHttpResponse response = getSharedHttpClient().execute(head, localContext); + + try { + int code = response.getStatusLine().getStatusCode(); + logger.fine("Response for HEAD: " + code); + switch (code) { + case 200: + Header[] headers = response.getHeaders(HTTP.CONTENT_LEN); + logger.fine("Num headers: " + headers.length); + String sizeString = response.getHeaders(HTTP.CONTENT_LEN)[0].getValue(); + logger.fine("Content-Length: " + sizeString); + size = Long.parseLong(response.getHeaders(HTTP.CONTENT_LEN)[0].getValue()); + logger.fine("Found file size: " + size); + break; + default: + logger.warning("Response from " + head.getURI().toString() + " was " + code); + } + } finally { + EntityUtils.consume(response.getEntity()); + } + } catch (IOException e) { + logger.warning(e.getMessage()); + } + return size; + } + + @Override + public InputStream getInputStream() throws IOException { + if (super.getInputStream() == null) { + try { + HttpGet get = new HttpGet(generateTemporaryDownloadUrl(null, null, null)); + CloseableHttpResponse response = getSharedHttpClient().execute(get, localContext); + + int code = response.getStatusLine().getStatusCode(); + switch (code) { + case 200: + setInputStream(response.getEntity().getContent()); + break; + default: + logger.warning("Response from " + get.getURI().toString() + " was " + code); + throw new IOException("Cannot retrieve: " + baseUrl + "/" + urlPath + " code: " + code); + } + } catch (Exception e) { + logger.warning(e.getMessage()); + e.printStackTrace(); + throw new IOException("Error retrieving: " + baseUrl + "/" + urlPath + " " + e.getMessage()); + + } + setChannel(Channels.newChannel(super.getInputStream())); + } + return super.getInputStream(); + } + + @Override + public Channel getChannel() throws IOException { + if (super.getChannel() == null) { + getInputStream(); + } + return channel; + } + + @Override + public ReadableByteChannel getReadChannel() throws IOException { + // Make sure StorageIO.channel variable exists + getChannel(); + return super.getReadChannel(); + } + + @Override + public void delete() throws IOException { + // Delete is best-effort - we tell the remote server and it may or may not + // implement this call + if (!isDirectAccess()) { + throw new IOException("Direct Access IO must be used to permanently delete stored file objects"); + } + try { + HttpDelete del = new HttpDelete(baseUrl + "/" + urlPath); + CloseableHttpResponse response = getSharedHttpClient().execute(del, localContext); + try { + int code = response.getStatusLine().getStatusCode(); + switch (code) { + case 200: + logger.fine("Sent DELETE for " + baseUrl + "/" + urlPath); + default: + logger.fine("Response from DELETE on " + del.getURI().toString() + " was " + code); + } + } finally { + EntityUtils.consume(response.getEntity()); + } + } catch (Exception e) { + logger.warning(e.getMessage()); + throw new IOException("Error deleting: " + baseUrl + "/" + urlPath); + + } + + // Delete all the cached aux files as well: + deleteAllAuxObjects(); + + } + + @Override + public Channel openAuxChannel(String auxItemTag, DataAccessOption... options) throws IOException { + return baseStore.openAuxChannel(auxItemTag, options); + } + + @Override + public boolean isAuxObjectCached(String auxItemTag) throws IOException { + return baseStore.isAuxObjectCached(auxItemTag); + } + + @Override + public long getAuxObjectSize(String auxItemTag) throws IOException { + return baseStore.getAuxObjectSize(auxItemTag); + } + + @Override + public Path getAuxObjectAsPath(String auxItemTag) throws IOException { + return baseStore.getAuxObjectAsPath(auxItemTag); + } + + @Override + public void backupAsAux(String auxItemTag) throws IOException { + baseStore.backupAsAux(auxItemTag); + } + + @Override + public void revertBackupAsAux(String auxItemTag) throws IOException { + baseStore.revertBackupAsAux(auxItemTag); + } + + @Override + // this method copies a local filesystem Path into this DataAccess Auxiliary + // location: + public void savePathAsAux(Path fileSystemPath, String auxItemTag) throws IOException { + baseStore.savePathAsAux(fileSystemPath, auxItemTag); + } + + @Override + public void saveInputStreamAsAux(InputStream inputStream, String auxItemTag, Long filesize) throws IOException { + baseStore.saveInputStreamAsAux(inputStream, auxItemTag, filesize); + } + + /** + * @param inputStream InputStream we want to save + * @param auxItemTag String representing this Auxiliary type ("extension") + * @throws IOException if anything goes wrong. + */ + @Override + public void saveInputStreamAsAux(InputStream inputStream, String auxItemTag) throws IOException { + baseStore.saveInputStreamAsAux(inputStream, auxItemTag); + } + + @Override + public List listAuxObjects() throws IOException { + return baseStore.listAuxObjects(); + } + + @Override + public void deleteAuxObject(String auxItemTag) throws IOException { + baseStore.deleteAuxObject(auxItemTag); + } + + @Override + public void deleteAllAuxObjects() throws IOException { + baseStore.deleteAllAuxObjects(); + } + + @Override + public String getStorageLocation() throws IOException { + String fullStorageLocation = dvObject.getStorageIdentifier(); + logger.fine("storageidentifier: " + fullStorageLocation); + int driverIndex = fullStorageLocation.lastIndexOf(DataAccess.SEPARATOR); + if(driverIndex >=0) { + fullStorageLocation = fullStorageLocation.substring(fullStorageLocation.lastIndexOf(DataAccess.SEPARATOR) + DataAccess.SEPARATOR.length()); + } + if (this.getDvObject() instanceof Dataset) { + throw new IOException("RemoteOverlayAccessIO: Datasets are not a supported dvObject"); + } else if (this.getDvObject() instanceof DataFile) { + fullStorageLocation = StorageIO.getDriverPrefix(this.driverId) + fullStorageLocation; + } else if (dvObject instanceof Dataverse) { + throw new IOException("RemoteOverlayAccessIO: Dataverses are not a supported dvObject"); + } + logger.fine("fullStorageLocation: " + fullStorageLocation); + return fullStorageLocation; + } + + @Override + public Path getFileSystemPath() throws UnsupportedDataAccessOperationException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: this is a remote DataAccess IO object, it has no local filesystem path associated with it."); + } + + @Override + public boolean exists() { + logger.fine("Exists called"); + return (getSizeFromHttpHeader() != -1); + } + + @Override + public WritableByteChannel getWriteChannel() throws UnsupportedDataAccessOperationException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: there are no write Channels associated with S3 objects."); + } + + @Override + public OutputStream getOutputStream() throws UnsupportedDataAccessOperationException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: there are no output Streams associated with S3 objects."); + } + + @Override + public InputStream getAuxFileAsInputStream(String auxItemTag) throws IOException { + return baseStore.getAuxFileAsInputStream(auxItemTag); + } + + @Override + public boolean downloadRedirectEnabled() { + String optionValue = System.getProperty("dataverse.files." + this.driverId + ".download-redirect"); + if ("true".equalsIgnoreCase(optionValue)) { + return true; + } + return false; + } + + public boolean downloadRedirectEnabled(String auxObjectTag) { + return baseStore.downloadRedirectEnabled(auxObjectTag); + } + + @Override + public String generateTemporaryDownloadUrl(String auxiliaryTag, String auxiliaryType, String auxiliaryFileName) + throws IOException { + + // ToDo - support remote auxiliary Files + if (auxiliaryTag == null) { + String secretKey = System.getProperty("dataverse.files." + this.driverId + ".secret-key"); + if (secretKey == null) { + return baseUrl + "/" + urlPath; + } else { + return UrlSignerUtil.signUrl(baseUrl + "/" + urlPath, getUrlExpirationMinutes(), null, "GET", + secretKey); + } + } else { + return baseStore.generateTemporaryDownloadUrl(auxiliaryTag, auxiliaryType, auxiliaryFileName); + } + } + + int getUrlExpirationMinutes() { + String optionValue = System.getProperty("dataverse.files." + this.driverId + ".url-expiration-minutes"); + if (optionValue != null) { + Integer num; + try { + num = Integer.parseInt(optionValue); + } catch (NumberFormatException ex) { + num = null; + } + if (num != null) { + return num; + } + } + return 60; + } + + private void configureStores(DataAccessRequest req, String driverId, String storageLocation) throws IOException { + baseUrl = System.getProperty("dataverse.files." + this.driverId + ".base-url"); + if (baseUrl == null) { + throw new IOException("dataverse.files." + this.driverId + ".base-url is required"); + } else { + try { + new URI(baseUrl); + } catch (Exception e) { + logger.warning( + "Trouble interpreting base-url for store: " + this.driverId + " : " + e.getLocalizedMessage()); + throw new IOException("Can't interpret base-url as a URI"); + } + + } + + if (baseStore == null) { + String baseDriverId = getBaseStoreIdFor(driverId); + String fullStorageLocation = null; + String baseDriverType = System.getProperty("dataverse.files." + baseDriverId + ".type", DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER); + + if(dvObject instanceof Dataset) { + baseStore = DataAccess.getStorageIO(dvObject, req, baseDriverId); + } else { + if (this.getDvObject() != null) { + fullStorageLocation = getStoragePath(); + + // S3 expects :/// + switch (baseDriverType) { + case DataAccess.S3: + fullStorageLocation = baseDriverId + DataAccess.SEPARATOR + + System.getProperty("dataverse.files." + baseDriverId + ".bucket-name") + "/" + + fullStorageLocation; + break; + case DataAccess.FILE: + fullStorageLocation = baseDriverId + DataAccess.SEPARATOR + + System.getProperty("dataverse.files." + baseDriverId + ".directory", "/tmp/files") + "/" + + fullStorageLocation; + break; + default: + logger.warning("Not Implemented: RemoteOverlay store with base store type: " + + System.getProperty("dataverse.files." + baseDriverId + ".type")); + throw new IOException("Not implemented"); + } + + } else if (storageLocation != null) { + // ://// + //remoteDriverId:// is removed if coming through directStorageIO + int index = storageLocation.indexOf(DataAccess.SEPARATOR); + if(index > 0) { + storageLocation = storageLocation.substring(index + DataAccess.SEPARATOR.length()); + } + //THe base store needs the baseStoreIdentifier and not the relative URL + fullStorageLocation = storageLocation.substring(0, storageLocation.indexOf("//")); + + switch (baseDriverType) { + case DataAccess.S3: + fullStorageLocation = baseDriverId + DataAccess.SEPARATOR + + System.getProperty("dataverse.files." + baseDriverId + ".bucket-name") + "/" + + fullStorageLocation; + break; + case DataAccess.FILE: + fullStorageLocation = baseDriverId + DataAccess.SEPARATOR + + System.getProperty("dataverse.files." + baseDriverId + ".directory", "/tmp/files") + "/" + + fullStorageLocation; + break; + default: + logger.warning("Not Implemented: RemoteOverlay store with base store type: " + + System.getProperty("dataverse.files." + baseDriverId + ".type")); + throw new IOException("Not implemented"); + } + } + baseStore = DataAccess.getDirectStorageIO(fullStorageLocation); + } + if (baseDriverType.contentEquals(DataAccess.S3)) { + ((S3AccessIO) baseStore).setMainDriver(false); + } + } + remoteStoreName = System.getProperty("dataverse.files." + this.driverId + ".remote-store-name"); + try { + remoteStoreUrl = new URL(System.getProperty("dataverse.files." + this.driverId + ".remote-store-url")); + } catch(MalformedURLException mfue) { + logger.fine("Unable to read remoteStoreUrl for driver: " + this.driverId); + } + } + + //Convenience method to assemble the path, starting with the DOI authority/identifier/, that is needed to create a base store via DataAccess.getDirectStorageIO - the caller has to add the store type specific prefix required. + private String getStoragePath() throws IOException { + String fullStoragePath = dvObject.getStorageIdentifier(); + logger.fine("storageidentifier: " + fullStoragePath); + int driverIndex = fullStoragePath.lastIndexOf(DataAccess.SEPARATOR); + if(driverIndex >=0) { + fullStoragePath = fullStoragePath.substring(fullStoragePath.lastIndexOf(DataAccess.SEPARATOR) + DataAccess.SEPARATOR.length()); + } + int suffixIndex = fullStoragePath.indexOf("//"); + if(suffixIndex >=0) { + fullStoragePath = fullStoragePath.substring(0, suffixIndex); + } + if (this.getDvObject() instanceof Dataset) { + fullStoragePath = this.getDataset().getAuthorityForFileStorage() + "/" + + this.getDataset().getIdentifierForFileStorage() + "/" + fullStoragePath; + } else if (this.getDvObject() instanceof DataFile) { + fullStoragePath = this.getDataFile().getOwner().getAuthorityForFileStorage() + "/" + + this.getDataFile().getOwner().getIdentifierForFileStorage() + "/" + fullStoragePath; + }else if (dvObject instanceof Dataverse) { + throw new IOException("RemoteOverlayAccessIO: Dataverses are not a supported dvObject"); + } + logger.fine("fullStoragePath: " + fullStoragePath); + return fullStoragePath; + } + + public CloseableHttpClient getSharedHttpClient() { + if (httpclient == null) { + try { + initHttpPool(); + httpclient = HttpClients.custom().setConnectionManager(cm).setDefaultRequestConfig(config).build(); + + } catch (NoSuchAlgorithmException | KeyStoreException | KeyManagementException ex) { + logger.warning(ex.getMessage()); + } + } + return httpclient; + } + + private void initHttpPool() throws NoSuchAlgorithmException, KeyManagementException, KeyStoreException { + if (trustCerts) { + // use the TrustSelfSignedStrategy to allow Self Signed Certificates + SSLContext sslContext; + SSLConnectionSocketFactory connectionFactory; + + sslContext = SSLContextBuilder.create().loadTrustMaterial(new TrustAllStrategy()).build(); + // create an SSL Socket Factory to use the SSLContext with the trust self signed + // certificate strategy + // and allow all hosts verifier. + connectionFactory = new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE); + + Registry registry = RegistryBuilder.create() + .register("https", connectionFactory).build(); + cm = new PoolingHttpClientConnectionManager(registry); + } else { + cm = new PoolingHttpClientConnectionManager(); + } + cm.setDefaultMaxPerRoute(httpConcurrency); + cm.setMaxTotal(httpConcurrency > 20 ? httpConcurrency : 20); + } + + @Override + public void savePath(Path fileSystemPath) throws IOException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: savePath() not implemented in this storage driver."); + + } + + @Override + public void saveInputStream(InputStream inputStream) throws IOException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: saveInputStream() not implemented in this storage driver."); + + } + + @Override + public void saveInputStream(InputStream inputStream, Long filesize) throws IOException { + throw new UnsupportedDataAccessOperationException( + "RemoteOverlayAccessIO: saveInputStream(InputStream, Long) not implemented in this storage driver."); + + } + + protected static boolean isValidIdentifier(String driverId, String storageId) { + String urlPath = storageId.substring(storageId.lastIndexOf("//") + 2); + String baseUrl = System.getProperty("dataverse.files." + driverId + ".base-url"); + try { + URI absoluteURI = new URI(baseUrl + "/" + urlPath); + if(!absoluteURI.normalize().toString().startsWith(baseUrl)) { + logger.warning("storageidentifier doesn't start with " + driverId + "'s base-url: " + storageId); + return false; + } + } catch(URISyntaxException use) { + logger.warning("Could not interpret storageidentifier in remote store " + driverId + " : " + storageId); + return false; + } + return true; + } + + public static String getBaseStoreIdFor(String driverId) { + return System.getProperty("dataverse.files." + driverId + ".base-store"); + } + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java index ea19d29b41e..3c9cef04980 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java @@ -84,6 +84,8 @@ public class S3AccessIO extends StorageIO { private static final Config config = ConfigProvider.getConfig(); private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.dataaccess.S3AccessIO"); + + private boolean mainDriver = true; private static HashMap driverClientMap = new HashMap(); private static HashMap driverTMMap = new HashMap(); @@ -120,6 +122,7 @@ public S3AccessIO(T dvObject, DataAccessRequest req, String driverId) { public S3AccessIO(String storageLocation, String driverId) { this(null, null, driverId); // TODO: validate the storage location supplied + logger.fine("Instantiating with location: " + storageLocation); bucketName = storageLocation.substring(0,storageLocation.indexOf('/')); minPartSize = getMinPartSize(driverId); key = storageLocation.substring(storageLocation.indexOf('/')+1); @@ -174,22 +177,22 @@ public void open(DataAccessOption... options) throws IOException { //Fix new DataFiles: DataFiles that have not yet been saved may use this method when they don't have their storageidentifier in the final ://: form // So we fix it up here. ToDo: refactor so that storageidentifier is generated by the appropriate StorageIO class and is final from the start. String newStorageIdentifier = null; - if (storageIdentifier.startsWith(this.driverId + "://")) { - if(!storageIdentifier.substring((this.driverId + "://").length()).contains(":")) { + if (storageIdentifier.startsWith(this.driverId + DataAccess.SEPARATOR)) { + if(!storageIdentifier.substring((this.driverId + DataAccess.SEPARATOR).length()).contains(":")) { //Driver id but no bucket if(bucketName!=null) { - newStorageIdentifier=this.driverId + "://" + bucketName + ":" + storageIdentifier.substring((this.driverId + "://").length()); + newStorageIdentifier=this.driverId + DataAccess.SEPARATOR + bucketName + ":" + storageIdentifier.substring((this.driverId + DataAccess.SEPARATOR).length()); } else { throw new IOException("S3AccessIO: DataFile (storage identifier " + storageIdentifier + ") is not associated with a bucket."); } } // else we're OK (assumes bucket name in storageidentifier matches the driver's bucketname) } else { - if(!storageIdentifier.substring((this.driverId + "://").length()).contains(":")) { + if(!storageIdentifier.substring((this.driverId + DataAccess.SEPARATOR).length()).contains(":")) { //No driver id or bucket - newStorageIdentifier= this.driverId + "://" + bucketName + ":" + storageIdentifier; + newStorageIdentifier= this.driverId + DataAccess.SEPARATOR + bucketName + ":" + storageIdentifier; } else { //Just the bucketname - newStorageIdentifier= this.driverId + "://" + storageIdentifier; + newStorageIdentifier= this.driverId + DataAccess.SEPARATOR + storageIdentifier; } } if(newStorageIdentifier != null) { @@ -235,39 +238,44 @@ public void open(DataAccessOption... options) throws IOException { } else if (dvObject instanceof Dataset) { Dataset dataset = this.getDataset(); key = dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage(); - dataset.setStorageIdentifier(this.driverId + "://" + key); + dataset.setStorageIdentifier(this.driverId + DataAccess.SEPARATOR + key); } else if (dvObject instanceof Dataverse) { throw new IOException("Data Access: Storage driver does not support dvObject type Dataverse yet"); } else { - // Direct access, e.g. for external upload - no associated DVobject yet, but we want to be able to get the size - // With small files, it looks like we may call before S3 says it exists, so try some retries before failing - if(key!=null) { - ObjectMetadata objectMetadata = null; - int retries = 20; - while(retries > 0) { - try { - objectMetadata = s3.getObjectMetadata(bucketName, key); - if(retries != 20) { - logger.warning("Success for key: " + key + " after " + ((20-retries)*3) + " seconds"); - } - retries = 0; - } catch (SdkClientException sce) { - if(retries > 1) { - retries--; - try { - Thread.sleep(3000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - logger.warning("Retrying after: " + sce.getMessage()); - } else { - throw new IOException("Cannot get S3 object " + key + " ("+sce.getMessage()+")"); - } - } - } - this.setSize(objectMetadata.getContentLength()); - }else { - throw new IOException("Data Access: Invalid DvObject type"); + if (isMainDriver()) { + // Direct access, e.g. for external upload - no associated DVobject yet, but we + // want to be able to get the size + // With small files, it looks like we may call before S3 says it exists, so try + // some retries before failing + if (key != null) { + ObjectMetadata objectMetadata = null; + int retries = 20; + while (retries > 0) { + try { + objectMetadata = s3.getObjectMetadata(bucketName, key); + if (retries != 20) { + logger.warning( + "Success for key: " + key + " after " + ((20 - retries) * 3) + " seconds"); + } + retries = 0; + } catch (SdkClientException sce) { + if (retries > 1) { + retries--; + try { + Thread.sleep(3000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + logger.warning("Retrying after: " + sce.getMessage()); + } else { + throw new IOException("Cannot get S3 object " + key + " (" + sce.getMessage() + ")"); + } + } + } + this.setSize(objectMetadata.getContentLength()); + } else { + throw new IOException("Data Access: Invalid DvObject type"); + } } } } @@ -437,6 +445,7 @@ public void delete() throws IOException { @Override public Channel openAuxChannel(String auxItemTag, DataAccessOption... options) throws IOException { if (isWriteAccessRequested(options)) { + //Need size to write to S3 throw new UnsupportedDataAccessOperationException("S3AccessIO: write mode openAuxChannel() not yet implemented in this storage driver."); } @@ -723,7 +732,7 @@ public String getStorageLocation() throws IOException { throw new IOException("Failed to obtain the S3 key for the file"); } - return this.driverId + "://" + bucketName + "/" + locationKey; + return this.driverId + DataAccess.SEPARATOR + bucketName + "/" + locationKey; } @Override @@ -822,7 +831,7 @@ private static String getMainFileKey(String baseKey, String storageIdentifier, S throw new FileNotFoundException("Data Access: No local storage identifier defined for this datafile."); } - if (storageIdentifier.indexOf(driverId + "://")>=0) { + if (storageIdentifier.indexOf(driverId + DataAccess.SEPARATOR)>=0) { //String driverId = storageIdentifier.substring(0, storageIdentifier.indexOf("://")+3); //As currently implemented (v4.20), the bucket is part of the identifier and we could extract it and compare it with getBucketName() as a check - //Only one bucket per driver is supported (though things might work if the profile creds work with multiple buckets, then again it's not clear when logic is reading from the driver property or from the DataFile). @@ -834,6 +843,7 @@ private static String getMainFileKey(String baseKey, String storageIdentifier, S return key; } + @Override public boolean downloadRedirectEnabled() { String optionValue = System.getProperty("dataverse.files." + this.driverId + ".download-redirect"); if ("true".equalsIgnoreCase(optionValue)) { @@ -842,6 +852,10 @@ public boolean downloadRedirectEnabled() { return false; } + public boolean downloadRedirectEnabled(String auxObjectTag) { + return downloadRedirectEnabled(); + } + /** * Generates a temporary URL for a direct S3 download; * either for the main physical file, or (optionally) for an auxiliary. @@ -851,7 +865,7 @@ public boolean downloadRedirectEnabled() { * @return redirect url * @throws IOException. */ - public String generateTemporaryS3Url(String auxiliaryTag, String auxiliaryType, String auxiliaryFileName) throws IOException { + public String generateTemporaryDownloadUrl(String auxiliaryTag, String auxiliaryType, String auxiliaryFileName) throws IOException { //Questions: // Q. Should this work for private and public? // A. Yes! Since the URL has a limited, short life span. -- L.A. @@ -923,9 +937,9 @@ public String generateTemporaryS3Url(String auxiliaryTag, String auxiliaryType, // them for some servers, we check whether the protocol is in the url and then // normalizing to use the part without the protocol String endpointServer = endpoint; - int protocolEnd = endpoint.indexOf("://"); + int protocolEnd = endpoint.indexOf(DataAccess.SEPARATOR); if (protocolEnd >=0 ) { - endpointServer = endpoint.substring(protocolEnd + 3); + endpointServer = endpoint.substring(protocolEnd + DataAccess.SEPARATOR.length()); } logger.fine("Endpoint: " + endpointServer); // We're then replacing @@ -984,9 +998,9 @@ private String generateTemporaryS3UploadUrl(String key, Date expiration) throws // them for some servers, we check whether the protocol is in the url and then // normalizing to use the part without the protocol String endpointServer = endpoint; - int protocolEnd = endpoint.indexOf("://"); + int protocolEnd = endpoint.indexOf(DataAccess.SEPARATOR); if (protocolEnd >=0 ) { - endpointServer = endpoint.substring(protocolEnd + 3); + endpointServer = endpoint.substring(protocolEnd + DataAccess.SEPARATOR.length()); } logger.fine("Endpoint: " + endpointServer); // We're then replacing @@ -1253,4 +1267,44 @@ public static void completeMultipartUpload(String globalId, String storageIdenti s3Client.completeMultipartUpload(req); } + public boolean isMainDriver() { + return mainDriver; + } + + public void setMainDriver(boolean mainDriver) { + this.mainDriver = mainDriver; + } + + public static String getDriverPrefix(String driverId) { + return driverId+ DataAccess.SEPARATOR + getBucketName(driverId) + ":"; + } + + //Confirm inputs are of the form s3://demo-dataverse-bucket:176e28068b0-1c3f80357c42 + protected static boolean isValidIdentifier(String driverId, String storageId) { + String storageBucketAndId = storageId.substring(storageId.lastIndexOf("//") + 2); + String bucketName = getBucketName(driverId); + if(bucketName==null) { + logger.warning("No bucket defined for " + driverId); + return false; + } + int index = storageBucketAndId.lastIndexOf(":"); + if(index<=0) { + logger.warning("No bucket defined in submitted identifier: " + storageId); + return false; + } + String idBucket = storageBucketAndId.substring(0, index); + String id = storageBucketAndId.substring(index+1); + logger.fine(id); + if(!bucketName.equals(idBucket)) { + logger.warning("Incorrect bucket in submitted identifier: " + storageId); + return false; + } + if (!usesStandardNamePattern(id)) { + logger.warning("Unacceptable identifier pattern in submitted identifier: " + storageId); + return false; + } + return true; + } + + } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java index b0e9648285c..90e4a54dbe8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java @@ -30,16 +30,17 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.net.URL; import java.nio.channels.Channel; import java.nio.channels.ReadableByteChannel; import java.nio.channels.WritableByteChannel; import java.nio.file.Path; +import java.util.HashMap; import java.util.Iterator; import java.util.List; - - -//import org.apache.commons.httpclient.Header; -//import org.apache.commons.httpclient.methods.GetMethod; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; /** @@ -66,7 +67,7 @@ public StorageIO(T dvObject, DataAccessRequest req, String driverId) { this.req = new DataAccessRequest(); } if (this.driverId == null) { - this.driverId = "file"; + this.driverId = DataAccess.FILE; } } @@ -78,7 +79,10 @@ public StorageIO(T dvObject, DataAccessRequest req, String driverId) { protected boolean isReadAccess = false; protected boolean isWriteAccess = false; - + //A public store is one in which files may be accessible outside Dataverse and therefore accessible without regard to Dataverse's access controls related to restriction and embargoes. + //Currently, this is just used to warn users at upload time rather than disable restriction/embargo. + static protected Map driverPublicAccessMap = new HashMap(); + public boolean canRead() { return isReadAccess; } @@ -183,7 +187,7 @@ public boolean canWrite() { public abstract void deleteAllAuxObjects() throws IOException; private DataAccessRequest req; - private InputStream in; + private InputStream in = null; private OutputStream out; protected Channel channel; protected DvObject dvObject; @@ -222,6 +226,8 @@ public boolean canWrite() { private String swiftFileName; private String remoteUrl; + protected String remoteStoreName = null; + protected URL remoteStoreUrl = null; // For HTTP-based downloads: /*private GetMethod method = null; @@ -330,6 +336,14 @@ public String getSwiftContainerName(){ return swiftContainerName; } + public String getRemoteStoreName() { + return remoteStoreName; + } + + public URL getRemoteStoreUrl() { + return remoteStoreUrl; + } + /*public GetMethod getHTTPMethod() { return method; } @@ -564,4 +578,48 @@ public boolean isBelowIngestSizeLimit() { return true; } } + + public boolean downloadRedirectEnabled() { + return false; + } + + public boolean downloadRedirectEnabled(String auxObjectTag) { + return false; + } + + public String generateTemporaryDownloadUrl(String auxiliaryTag, String auxiliaryType, String auxiliaryFileName) throws IOException { + throw new UnsupportedDataAccessOperationException("Direct download not implemented for this storage type"); + } + + + public static boolean isPublicStore(String driverId) { + //Read once and cache + if(!driverPublicAccessMap.containsKey(driverId)) { + driverPublicAccessMap.put(driverId, Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".public"))); + } + return driverPublicAccessMap.get(driverId); + } + + public static String getDriverPrefix(String driverId) { + return driverId+ DataAccess.SEPARATOR; + } + + public static boolean isDirectUploadEnabled(String driverId) { + return Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".upload-redirect")); + } + + //Check that storageIdentifier is consistent with store's config + //False will prevent direct uploads + protected static boolean isValidIdentifier(String driverId, String storageId) { + return false; + } + + //Utility to verify the standard UUID pattern for stored files. + protected static boolean usesStandardNamePattern(String identifier) { + + Pattern r = Pattern.compile("^[a-f,0-9]{11}-[a-f,0-9]{12}$"); + Matcher m = r.matcher(identifier); + return m.find(); + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java index 3bc29cb9836..b1725b040a3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java @@ -508,7 +508,7 @@ private StoredObject initializeSwiftFileObject(boolean writeAccess, String auxIt if (dvObject instanceof DataFile) { Dataset owner = this.getDataFile().getOwner(); - if (storageIdentifier.startsWith(this.driverId + "://")) { + if (storageIdentifier.startsWith(this.driverId + DataAccess.SEPARATOR)) { // This is a call on an already existing swift object. String[] swiftStorageTokens = storageIdentifier.substring(8).split(":", 3); @@ -552,14 +552,14 @@ private StoredObject initializeSwiftFileObject(boolean writeAccess, String auxIt //setSwiftContainerName(swiftFolderPath); //swiftFileName = dataFile.getDisplayName(); //Storage Identifier is now updated after the object is uploaded on Swift. - dvObject.setStorageIdentifier(this.driverId + "://" + swiftDefaultEndpoint + ":" + swiftFolderPath + ":" + swiftFileName); + dvObject.setStorageIdentifier(this.driverId + DataAccess.SEPARATOR + swiftDefaultEndpoint + ":" + swiftFolderPath + ":" + swiftFileName); } else { throw new IOException("SwiftAccessIO: unknown access mode."); } } else if (dvObject instanceof Dataset) { Dataset dataset = this.getDataset(); - if (storageIdentifier.startsWith(this.driverId + "://")) { + if (storageIdentifier.startsWith(this.driverId + DataAccess.SEPARATOR)) { // This is a call on an already existing swift object. //TODO: determine how storage identifier will give us info @@ -601,7 +601,7 @@ private StoredObject initializeSwiftFileObject(boolean writeAccess, String auxIt swiftPseudoFolderPathSeparator + dataset.getIdentifierForFileStorage(); swiftFileName = auxItemTag; - dvObject.setStorageIdentifier(this.driverId + "://" + swiftEndPoint + ":" + swiftFolderPath); + dvObject.setStorageIdentifier(this.driverId + DataAccess.SEPARATOR + swiftEndPoint + ":" + swiftFolderPath); } else { throw new IOException("SwiftAccessIO: unknown access mode."); } @@ -628,7 +628,7 @@ private StoredObject initializeSwiftFileObject(boolean writeAccess, String auxIt other swiftContainerName Object Store pseudo-folder can be created, which is not provide by the joss Java swift library as of yet. */ - if (storageIdentifier.startsWith(this.driverId + "://")) { + if (storageIdentifier.startsWith(this.driverId + DataAccess.SEPARATOR)) { // An existing swift object; the container must already exist as well. this.swiftContainer = account.getContainer(swiftContainerName); } else { @@ -874,7 +874,7 @@ public String getSwiftContainerName() { } return null; } - + //https://gist.github.com/ishikawa/88599 public static String toHexString(byte[] bytes) { Formatter formatter = new Formatter(); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java index ccf947b8868..7683aab7dfa 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java @@ -5,6 +5,7 @@ import edu.harvard.iq.dataverse.DatasetField; import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.FileMetadata; +import edu.harvard.iq.dataverse.TermsOfUseAndAccess; import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; import edu.harvard.iq.dataverse.dataaccess.DataAccess; import static edu.harvard.iq.dataverse.dataaccess.DataAccess.getStorageIO; @@ -36,6 +37,7 @@ import edu.harvard.iq.dataverse.datasetutility.FileSizeChecker; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.StringUtil; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; @@ -456,13 +458,13 @@ public static List getDatasetSummaryFields(DatasetVersion datasetV return datasetFields; } - public static boolean isAppropriateStorageDriver(Dataset dataset){ - // ToDo - rsync was written before multiple store support and currently is hardcoded to use the "s3" store. + public static boolean isRsyncAppropriateStorageDriver(Dataset dataset){ + // ToDo - rsync was written before multiple store support and currently is hardcoded to use the DataAccess.S3 store. // When those restrictions are lifted/rsync can be configured per store, this test should check that setting // instead of testing for the 's3" store, //This method is used by both the dataset and edit files page so one change here //will fix both - return dataset.getEffectiveStorageDriverId().equals("s3"); + return dataset.getEffectiveStorageDriverId().equals(DataAccess.S3); } /** @@ -476,16 +478,16 @@ public static boolean isAppropriateStorageDriver(Dataset dataset){ public static String getDownloadSize(DatasetVersion dsv, boolean original) { return FileSizeChecker.bytesToHumanReadable(getDownloadSizeNumeric(dsv, original)); } - + public static Long getDownloadSizeNumeric(DatasetVersion dsv, boolean original) { return getDownloadSizeNumericBySelectedFiles(dsv.getFileMetadatas(), original); } - + public static Long getDownloadSizeNumericBySelectedFiles(List fileMetadatas, boolean original) { long bytes = 0l; for (FileMetadata fileMetadata : fileMetadatas) { DataFile dataFile = fileMetadata.getDataFile(); - if (original && dataFile.isTabularData()) { + if (original && dataFile.isTabularData()) { bytes += dataFile.getOriginalFileSize() == null ? 0 : dataFile.getOriginalFileSize(); } else { bytes += dataFile.getFilesize(); @@ -538,14 +540,23 @@ public static boolean validateDatasetMetadataExternally(Dataset ds, String execu } + public static License getLicense(DatasetVersion dsv) { + License license = null; + TermsOfUseAndAccess tua = dsv.getTermsOfUseAndAccess(); + if(tua!=null) { + license = tua.getLicense(); + } + return license; + } + public static String getLicenseName(DatasetVersion dsv) { - License license = dsv.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(dsv); return license != null ? license.getName() : BundleUtil.getStringFromBundle("license.custom"); } public static String getLicenseURI(DatasetVersion dsv) { - License license = dsv.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(dsv); // Return the URI // For standard licenses, just return the stored URI return (license != null) ? license.getUri().toString() @@ -560,12 +571,12 @@ public static String getLicenseURI(DatasetVersion dsv) { } public static String getLicenseIcon(DatasetVersion dsv) { - License license = dsv.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(dsv); return license != null && license.getIconUrl() != null ? license.getIconUrl().toString() : null; } public static String getLicenseDescription(DatasetVersion dsv) { - License license = dsv.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(dsv); return license != null ? license.getShortDescription() : BundleUtil.getStringFromBundle("license.custom.description"); } diff --git a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java index b270393e5e1..a6ca27050a1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java +++ b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java @@ -19,12 +19,10 @@ import edu.harvard.iq.dataverse.api.Files; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; -import edu.harvard.iq.dataverse.engine.command.impl.AbstractCreateDatasetCommand; -import edu.harvard.iq.dataverse.engine.command.impl.CreateNewDatasetCommand; -import edu.harvard.iq.dataverse.engine.command.impl.DeleteDataFileCommand; import edu.harvard.iq.dataverse.engine.command.impl.RestrictFileCommand; import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetVersionCommand; import edu.harvard.iq.dataverse.ingest.IngestServiceBean; @@ -43,7 +41,6 @@ import java.util.Iterator; import java.util.List; import java.util.Objects; -import java.util.ResourceBundle; import java.util.Set; import java.util.logging.Level; import java.util.logging.Logger; @@ -59,7 +56,6 @@ import javax.ws.rs.core.Response; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; -import org.apache.commons.lang3.StringUtils; import org.apache.commons.io.IOUtils; import org.ocpsoft.common.util.Strings; @@ -778,7 +774,7 @@ private boolean runAddReplacePhase2(boolean tabIngest){ } } } - + msgt("step_090_notifyUser"); if (!this.step_090_notifyUser()){ return false; @@ -1379,7 +1375,7 @@ private boolean step_040_auto_checkForDuplicates(){ String fileType = fileToReplace.getOriginalFileFormat() != null ? fileToReplace.getOriginalFileFormat() : fileToReplace.getContentType(); if (!finalFileList.get(0).getContentType().equalsIgnoreCase(fileType)) { String friendlyType = fileToReplace.getOriginalFormatLabel() != null ? fileToReplace.getOriginalFormatLabel() : fileToReplace.getFriendlyType(); - + List errParams = Arrays.asList(friendlyType, finalFileList.get(0).getFriendlyType()); @@ -2049,6 +2045,10 @@ public Response addFiles(String jsonData, Dataset dataset, User authUser) { String newStorageIdentifier = null; if (optionalFileParams.hasStorageIdentifier()) { newStorageIdentifier = optionalFileParams.getStorageIdentifier(); + newStorageIdentifier = DataAccess.expandStorageIdentifierIfNeeded(newStorageIdentifier); + if(!DataAccess.uploadToDatasetAllowed(dataset, newStorageIdentifier)) { + addErrorSevere("Dataset store configuration does not allow provided storageIdentifier."); + } if (optionalFileParams.hasFileName()) { newFilename = optionalFileParams.getFileName(); if (optionalFileParams.hasMimetype()) { @@ -2057,14 +2057,10 @@ public Response addFiles(String jsonData, Dataset dataset, User authUser) { } msgt("ADD! = " + newFilename); - - runAddFileByDataset(dataset, - newFilename, - newFileContentType, - newStorageIdentifier, - null, - optionalFileParams, true); - + if (!hasError()) { + runAddFileByDataset(dataset, newFilename, newFileContentType, newStorageIdentifier, + null, optionalFileParams, true); + } if (hasError()) { JsonObjectBuilder fileoutput = Json.createObjectBuilder() .add("storageIdentifier", newStorageIdentifier) @@ -2088,8 +2084,8 @@ public Response addFiles(String jsonData, Dataset dataset, User authUser) { .add("fileDetails", successresult.getJsonArray("files").getJsonObject(0)); jarr.add(fileoutput); } - } successNumberofFiles = successNumberofFiles + 1; + } } else { JsonObjectBuilder fileoutput = Json.createObjectBuilder() .add("errorMessage", "You must provide a storageidentifier, filename, and mimetype.") diff --git a/src/main/java/edu/harvard/iq/dataverse/datasetutility/OptionalFileParams.java b/src/main/java/edu/harvard/iq/dataverse/datasetutility/OptionalFileParams.java index 35687151090..959dbc4e262 100644 --- a/src/main/java/edu/harvard/iq/dataverse/datasetutility/OptionalFileParams.java +++ b/src/main/java/edu/harvard/iq/dataverse/datasetutility/OptionalFileParams.java @@ -16,6 +16,7 @@ import edu.harvard.iq.dataverse.DataFileTag; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.api.Util; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.util.BundleUtil; import java.lang.reflect.Type; @@ -371,8 +372,15 @@ private void loadParamsFromJson(String jsonData) throws DataFileTagException{ // get storage identifier as string // ------------------------------- if ((jsonObj.has(STORAGE_IDENTIFIER_ATTR_NAME)) && (!jsonObj.get(STORAGE_IDENTIFIER_ATTR_NAME).isJsonNull())){ + // Basic sanity check that driver specified is defined and the overall + // identifier is consistent with that store's config. Note that being able to + // specify a driver that does not support direct uploads is currently used with + // out-of-band uploads, e.g. for bulk migration. + String storageId = jsonObj.get(STORAGE_IDENTIFIER_ATTR_NAME).getAsString(); + if (DataAccess.isValidDirectStorageIdentifier(storageId)) { + this.storageIdentifier = storageId; + } - this.storageIdentifier = jsonObj.get(STORAGE_IDENTIFIER_ATTR_NAME).getAsString(); } // ------------------------------- @@ -396,7 +404,7 @@ private void loadParamsFromJson(String jsonData) throws DataFileTagException{ // ------------------------------- if ((jsonObj.has(LEGACY_CHECKSUM_ATTR_NAME)) && (!jsonObj.get(LEGACY_CHECKSUM_ATTR_NAME).isJsonNull())){ - this.checkSumValue = jsonObj.get(LEGACY_CHECKSUM_ATTR_NAME).getAsString(); + this.checkSumValue = jsonObj.get(LEGACY_CHECKSUM_ATTR_NAME).getAsString().toLowerCase(); this.checkSumType= ChecksumType.MD5; } // ------------------------------- @@ -404,7 +412,7 @@ private void loadParamsFromJson(String jsonData) throws DataFileTagException{ // ------------------------------- else if ((jsonObj.has(CHECKSUM_OBJECT_NAME)) && (!jsonObj.get(CHECKSUM_OBJECT_NAME).isJsonNull())){ - this.checkSumValue = ((JsonObject) jsonObj.get(CHECKSUM_OBJECT_NAME)).get(CHECKSUM_OBJECT_VALUE).getAsString(); + this.checkSumValue = ((JsonObject) jsonObj.get(CHECKSUM_OBJECT_NAME)).get(CHECKSUM_OBJECT_VALUE).getAsString().toLowerCase(); this.checkSumType = ChecksumType.fromString(((JsonObject) jsonObj.get(CHECKSUM_OBJECT_NAME)).get(CHECKSUM_OBJECT_TYPE).getAsString()); } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractCreateDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractCreateDatasetCommand.java index ec544d9490a..1465cbd74e2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractCreateDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractCreateDatasetCommand.java @@ -102,7 +102,7 @@ public Dataset execute(CommandContext ctxt) throws CommandException { } if (theDataset.getStorageIdentifier() == null) { String driverId = theDataset.getEffectiveStorageDriverId(); - theDataset.setStorageIdentifier(driverId + "://" + theDataset.getAuthorityForFileStorage() + "/" + theDataset.getIdentifierForFileStorage()); + theDataset.setStorageIdentifier(driverId + DataAccess.SEPARATOR + theDataset.getAuthorityForFileStorage() + "/" + theDataset.getIdentifierForFileStorage()); } if (theDataset.getIdentifier()==null) { theDataset.setIdentifier(ctxt.datasets().generateDatasetIdentifier(theDataset, idServiceBean)); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java new file mode 100644 index 00000000000..89666f02db2 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java @@ -0,0 +1,381 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.SettingsWrapper; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; +import edu.harvard.iq.dataverse.branding.BrandingUtil; +import edu.harvard.iq.dataverse.engine.command.Command; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import edu.harvard.iq.dataverse.util.json.JsonUtil; +import edu.harvard.iq.dataverse.workflow.step.Failure; +import edu.harvard.iq.dataverse.workflow.step.WorkflowStepResult; + +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.charset.StandardCharsets; +import java.security.KeyFactory; +import java.security.KeyManagementException; +import java.security.KeyStoreException; +import java.security.NoSuchAlgorithmException; +import java.security.interfaces.RSAPrivateKey; +import java.security.spec.InvalidKeySpecException; +import java.security.spec.PKCS8EncodedKeySpec; +import java.time.Instant; +import java.util.Base64; +import java.util.Date; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.logging.Logger; + +import javax.json.Json; +import javax.json.JsonObject; +import javax.json.JsonObjectBuilder; +import javax.json.JsonValue; +import javax.net.ssl.SSLContext; + +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.http.client.ClientProtocolException; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.conn.ssl.NoopHostnameVerifier; +import org.apache.http.conn.ssl.TrustAllStrategy; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.ssl.SSLContextBuilder; + +import org.erdtman.jcs.JsonCanonicalizer; + +import com.auth0.jwt.JWT; +import com.auth0.jwt.algorithms.Algorithm; +import com.auth0.jwt.exceptions.JWTCreationException; + +@RequiredPermissions(Permission.PublishDataset) +public class DRSSubmitToArchiveCommand extends S3SubmitToArchiveCommand implements Command { + + private static final Logger logger = Logger.getLogger(DRSSubmitToArchiveCommand.class.getName()); + private static final String DRS_CONFIG = ":DRSArchiverConfig"; + private static final String ADMIN_METADATA = "admin_metadata"; + private static final String S3_BUCKET_NAME = "s3_bucket_name"; + private static final String S3_PATH = "s3_path"; + private static final String COLLECTIONS = "collections"; + private static final String PACKAGE_ID = "package_id"; + private static final String SINGLE_VERSION = "single_version"; + private static final String DRS_ENDPOINT = "DRS_endpoint"; + + + private static final String RSA_KEY = "dataverse.archiver.drs.rsa_key"; + + private static final String TRUST_CERT = "trust_cert"; + private static final String TIMEOUT = "timeout"; + + public DRSSubmitToArchiveCommand(DataverseRequest aRequest, DatasetVersion version) { + super(aRequest, version); + } + + @Override + public WorkflowStepResult performArchiveSubmission(DatasetVersion dv, ApiToken token, + Map requestedSettings) { + logger.fine("In DRSSubmitToArchiveCommand..."); + JsonObject drsConfigObject = null; + + try { + drsConfigObject = JsonUtil.getJsonObject(requestedSettings.get(DRS_CONFIG)); + } catch (Exception e) { + logger.warning("Unable to parse " + DRS_CONFIG + " setting as a Json object"); + } + if (drsConfigObject != null) { + JsonObject adminMetadata = drsConfigObject.getJsonObject(ADMIN_METADATA); + Set collections = adminMetadata.getJsonObject(COLLECTIONS).keySet(); + Dataset dataset = dv.getDataset(); + Dataverse ancestor = dataset.getOwner(); + String alias = getArchivableAncestor(ancestor, collections); + String spaceName = getSpaceName(dataset); + String packageId = getFileName(spaceName, dv); + + if (alias != null) { + if (drsConfigObject.getBoolean(SINGLE_VERSION, false)) { + for (DatasetVersion version : dataset.getVersions()) { + if (version.getArchivalCopyLocation() != null) { + return new Failure("DRS Archiver fail: version " + version.getFriendlyVersionNumber() + + " already archived."); + } + } + } + + JsonObject collectionConfig = adminMetadata.getJsonObject(COLLECTIONS).getJsonObject(alias); + + WorkflowStepResult s3Result = super.performArchiveSubmission(dv, token, requestedSettings); + + JsonObjectBuilder statusObject = Json.createObjectBuilder(); + statusObject.add(DatasetVersion.ARCHIVAL_STATUS, DatasetVersion.ARCHIVAL_STATUS_FAILURE); + statusObject.add(DatasetVersion.ARCHIVAL_STATUS_MESSAGE, "Bag not transferred"); + + if (s3Result == WorkflowStepResult.OK) { + //This will be overwritten if the further steps are successful + statusObject.add(DatasetVersion.ARCHIVAL_STATUS, DatasetVersion.ARCHIVAL_STATUS_FAILURE); + statusObject.add(DatasetVersion.ARCHIVAL_STATUS_MESSAGE, "Bag transferred, DRS ingest call failed"); + + // Now contact DRS + boolean trustCert = drsConfigObject.getBoolean(TRUST_CERT, false); + int jwtTimeout = drsConfigObject.getInt(TIMEOUT, 5); + JsonObjectBuilder job = Json.createObjectBuilder(); + + job.add(S3_BUCKET_NAME, adminMetadata.getString(S3_BUCKET_NAME)); + + job.add(PACKAGE_ID, packageId); + job.add(S3_PATH, spaceName); + + // We start with the default admin_metadata + JsonObjectBuilder amob = Json.createObjectBuilder(adminMetadata); + // Remove collections and then override any params for the given alias + amob.remove(COLLECTIONS); + // Allow override of bucket name + if (collectionConfig.containsKey(S3_BUCKET_NAME)) { + job.add(S3_BUCKET_NAME, collectionConfig.get(S3_BUCKET_NAME)); + } + + for (Entry entry : collectionConfig.entrySet()) { + if (!entry.getKey().equals(S3_BUCKET_NAME)) { + amob.add(entry.getKey(), entry.getValue()); + } + } + job.add(ADMIN_METADATA, amob); + + String drsConfigString = JsonUtil.prettyPrint(job.build()); + + CloseableHttpClient client = null; + if (trustCert) { + // use the TrustSelfSignedStrategy to allow Self Signed Certificates + try { + SSLContext sslContext = SSLContextBuilder.create().loadTrustMaterial(new TrustAllStrategy()) + .build(); + client = HttpClients.custom().setSSLContext(sslContext) + .setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE).build(); + } catch (KeyManagementException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (NoSuchAlgorithmException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (KeyStoreException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + if (client == null) { + client = HttpClients.createDefault(); + } + HttpPost ingestPost; + try { + ingestPost = new HttpPost(); + ingestPost.setURI(new URI(drsConfigObject.getString(DRS_ENDPOINT))); + + byte[] encoded = Base64.getDecoder().decode(System.getProperty(RSA_KEY).replaceAll("[\\r\\n]", "")); + + KeyFactory keyFactory = KeyFactory.getInstance("RSA"); + PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encoded); + RSAPrivateKey privKey = (RSAPrivateKey) keyFactory.generatePrivate(keySpec); + //RSAPublicKey publicKey; + /* + * If public key is needed: encoded = Base64.decodeBase64(publicKeyPEM); + * + * KeyFactory keyFactory = KeyFactory.getInstance("RS256"); X509EncodedKeySpec + * keySpec = new X509EncodedKeySpec(encoded); return (RSAPublicKey) + * keyFactory.generatePublic(keySpec); RSAPublicKey publicKey = new + * RSAPublicKey(System.getProperty(RS256_KEY)); + */ + Algorithm algorithmRSA = Algorithm.RSA256(null, privKey); + + String body = drsConfigString; + String jwtString = createJWTString(algorithmRSA, BrandingUtil.getInstallationBrandName(), body, jwtTimeout); + logger.fine("JWT: " + jwtString); + + ingestPost.setHeader("Authorization", "Bearer " + jwtString); + + logger.fine("Body: " + body); + ingestPost.setEntity(new StringEntity(body, "utf-8")); + ingestPost.setHeader("Content-Type", "application/json"); + + try (CloseableHttpResponse response = client.execute(ingestPost)) { + int code = response.getStatusLine().getStatusCode(); + String responseBody = new String(response.getEntity().getContent().readAllBytes(), + StandardCharsets.UTF_8); + if (code == 202) { + logger.fine("Status: " + code); + logger.fine("Response" + responseBody); + JsonObject responseObject = JsonUtil.getJsonObject(responseBody); + if (responseObject.containsKey(DatasetVersion.ARCHIVAL_STATUS) + && responseObject.containsKey(DatasetVersion.ARCHIVAL_STATUS_MESSAGE)) { + String status = responseObject.getString(DatasetVersion.ARCHIVAL_STATUS); + if (status.equals(DatasetVersion.ARCHIVAL_STATUS_PENDING) || status.equals(DatasetVersion.ARCHIVAL_STATUS_FAILURE) + || status.equals(DatasetVersion.ARCHIVAL_STATUS_SUCCESS)) { + statusObject.addAll(Json.createObjectBuilder(responseObject)); + switch (status) { + case DatasetVersion.ARCHIVAL_STATUS_PENDING: + logger.info("DRS Ingest successfully started for: " + packageId + " : " + + responseObject.toString()); + break; + case DatasetVersion.ARCHIVAL_STATUS_FAILURE: + logger.severe("DRS Ingest Failed for: " + packageId + " : " + + responseObject.toString()); + return new Failure("DRS Archiver fail in Ingest call"); + case DatasetVersion.ARCHIVAL_STATUS_SUCCESS: + // We don't expect this from DRS + logger.warning("Unexpected Status: " + status); + } + } else { + logger.severe("DRS Ingest Failed for: " + packageId + " with returned status: " + + status); + return new Failure( + "DRS Archiver fail in Ingest call with returned status: " + status); + } + } else { + logger.severe("DRS Ingest Failed for: " + packageId + + " - response does not include status and message"); + return new Failure( + "DRS Archiver fail in Ingest call \" - response does not include status and message"); + } + } else { + logger.severe("DRS Ingest Failed for: " + packageId + " with status code: " + code); + logger.fine("Response" + responseBody); + return new Failure("DRS Archiver fail in Ingest call with status code: " + code); + } + } catch (ClientProtocolException e2) { + e2.printStackTrace(); + } catch (IOException e2) { + e2.printStackTrace(); + } + } catch (URISyntaxException e) { + return new Failure( + "DRS Archiver workflow step failed: unable to parse " + DRS_ENDPOINT ); + } catch (JWTCreationException exception) { + // Invalid Signing configuration / Couldn't convert Claims. + return new Failure( + "DRS Archiver JWT Creation failure: " + exception.getMessage() ); + + } + // execute + catch (InvalidKeySpecException e) { + e.printStackTrace(); + } catch (NoSuchAlgorithmException e) { + e.printStackTrace(); + } catch (IOException e) { + e.printStackTrace(); + } finally { + //Set status after success or failure + dv.setArchivalCopyLocation(statusObject.build().toString()); + } + } else { + logger.warning("DRS: S3 archiving failed - will not call ingest: " + packageId); + dv.setArchivalCopyLocation(statusObject.build().toString()); + return new Failure("DRS Archiver fail in initial S3 Archiver transfer"); + } + + } else { + logger.fine("DRS Archiver: No matching collection found - will not archive: " + packageId); + return WorkflowStepResult.OK; + } + } else { + logger.warning(DRS_CONFIG + " not found"); + return new Failure("DRS Submission not configured - no " + DRS_CONFIG + " found."); + } + return WorkflowStepResult.OK; + } + + @Override + protected String getFileName(String spaceName, DatasetVersion dv) { + return spaceName + (".v" + dv.getFriendlyVersionNumber()).replace('.', '_'); + } + + @Override + protected String getDataCiteFileName(String spaceName, DatasetVersion dv) { + return spaceName + ("_datacite.v" + dv.getFriendlyVersionNumber()).replace('.','_'); + } + + + public static String createJWTString(Algorithm algorithmRSA, String installationBrandName, String body, int expirationInMinutes) throws IOException { + String canonicalBody = new JsonCanonicalizer(body).getEncodedString(); + logger.fine("Canonical body: " + canonicalBody); + String digest = DigestUtils.sha256Hex(canonicalBody); + return JWT.create().withIssuer(BrandingUtil.getInstallationBrandName()).withIssuedAt(Date.from(Instant.now())) + .withExpiresAt(Date.from(Instant.now().plusSeconds(60 * expirationInMinutes))) + .withKeyId("defaultDataverse").withClaim("bodySHA256Hash", digest).sign(algorithmRSA); + } + + private static String getArchivableAncestor(Dataverse ancestor, Set collections) { + String alias = ancestor.getAlias(); + while (ancestor != null && !collections.contains(alias)) { + ancestor = ancestor.getOwner(); + if (ancestor != null) { + alias = ancestor.getAlias(); + } else { + alias = null; + } + } + return alias; + } + + //Overrides inherited method to also check whether the dataset is in a collection for which the DRS Archiver is configured + public static boolean isArchivable(Dataset d, SettingsWrapper sw) { + JsonObject drsConfigObject = null; + + try { + String config = sw.get(DRS_CONFIG, null); + if (config != null) { + drsConfigObject = JsonUtil.getJsonObject(config); + } + } catch (Exception e) { + logger.warning("Unable to parse " + DRS_CONFIG + " setting as a Json object"); + } + if (drsConfigObject != null) { + JsonObject adminMetadata = drsConfigObject.getJsonObject(ADMIN_METADATA); + if (adminMetadata != null) { + JsonObject collectionObj = adminMetadata.getJsonObject(COLLECTIONS); + if (collectionObj != null) { + Set collections = collectionObj.keySet(); + return getArchivableAncestor(d.getOwner(), collections) != null; + } + } + } + return false; + } + + // DRS Archiver supports single-version semantics if the SINGLE_VERSION key in + // the DRS_CONFIG is true + // These methods make that choices visible on the page (cached via + // SettingsWrapper) or in the API (using SettingServiceBean), both using the + // same underlying logic + + public static boolean isSingleVersion(SettingsWrapper sw) { + String config = sw.get(DRS_CONFIG, null); + return isSingleVersion(config); + } + + public static boolean isSingleVersion(SettingsServiceBean ss) { + String config = ss.get(DRS_CONFIG, null); + return isSingleVersion(config); + } + + private static boolean isSingleVersion(String config) { + JsonObject drsConfigObject = null; + try { + if (config != null) { + drsConfigObject = JsonUtil.getJsonObject(config); + } + } catch (Exception e) { + logger.warning("Unable to parse " + DRS_CONFIG + " setting as a Json object"); + } + if (drsConfigObject != null) { + return drsConfigObject.getBoolean(SINGLE_VERSION, false); + } + return false; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java index 3e17cc638a4..12bb3fb6a0a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java @@ -37,6 +37,9 @@ import java.util.concurrent.Future; import org.apache.solr.client.solrj.SolrServerException; +import javax.ejb.EJB; +import javax.inject.Inject; + /** * @@ -48,7 +51,9 @@ public class FinalizeDatasetPublicationCommand extends AbstractPublishDatasetCommand { private static final Logger logger = Logger.getLogger(FinalizeDatasetPublicationCommand.class.getName()); - + + + /** * mirror field from {@link PublishDatasetCommand} of same name */ diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlockFacetsCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlockFacetsCommand.java new file mode 100644 index 00000000000..abc444dc538 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlockFacetsCommand.java @@ -0,0 +1,40 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.engine.command.AbstractCommand; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * + * @author adaybujeda + */ +public class ListMetadataBlockFacetsCommand extends AbstractCommand> { + + private final Dataverse dv; + + public ListMetadataBlockFacetsCommand(DataverseRequest aRequest, Dataverse aDataverse) { + super(aRequest, aDataverse); + dv = aDataverse; + } + + @Override + public List execute(CommandContext ctxt) throws CommandException { + return dv.getMetadataBlockFacets(); + } + + @Override + public Map> getRequiredPermissions() { + return Collections.singletonMap("", + dv.isReleased() ? Collections.emptySet() + : Collections.singleton(Permission.ViewUnpublishedDataverse)); + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/RedetectFileTypeCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/RedetectFileTypeCommand.java index 8eeca0cb4cd..286b107a5fd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/RedetectFileTypeCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/RedetectFileTypeCommand.java @@ -12,7 +12,8 @@ import edu.harvard.iq.dataverse.export.ExportException; import edu.harvard.iq.dataverse.export.ExportService; import edu.harvard.iq.dataverse.util.EjbUtil; -import edu.harvard.iq.dataverse.util.FileTypeDetection; +import edu.harvard.iq.dataverse.util.FileUtil; + import java.io.File; import java.io.FileOutputStream; import java.io.IOException; @@ -62,7 +63,7 @@ public DataFile execute(CommandContext ctxt) throws CommandException { } logger.fine("target file: " + localFile); - String newlyDetectedContentType = FileTypeDetection.determineFileType(localFile); + String newlyDetectedContentType = FileUtil.determineFileType(localFile, fileToRedetect.getDisplayName()); fileToRedetect.setContentType(newlyDetectedContentType); } catch (IOException ex) { throw new CommandException("Exception while attempting to get the bytes of the file during file type redetection: " + ex.getLocalizedMessage(), this); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommand.java new file mode 100644 index 00000000000..2e5b6b59ebe --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommand.java @@ -0,0 +1,65 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.engine.command.AbstractCommand; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; + +import java.util.Collections; +import java.util.List; +import java.util.stream.Collectors; + +/** + * + * @author adaybujeda + */ +@RequiredPermissions( Permission.EditDataverse ) +public class UpdateMetadataBlockFacetRootCommand extends AbstractCommand { + + private final Dataverse editedDv; + private final boolean metadataBlockFacetRoot; + + public UpdateMetadataBlockFacetRootCommand(DataverseRequest aRequest, Dataverse editedDv, boolean metadataBlockFacetRoot) { + super(aRequest, editedDv); + this.editedDv = editedDv; + this.metadataBlockFacetRoot = metadataBlockFacetRoot; + } + + @Override + public Dataverse execute(CommandContext ctxt) throws CommandException { + if(editedDv.isMetadataBlockFacetRoot() != metadataBlockFacetRoot) { + // Update metadata block facets when root changes value + // if you set root to be false (i.e. inherit), it should clear the blocks. + // if you set to true (i.e. use your own), it should make a copy of what is in the parent + List newBlockFacets = Collections.emptyList(); + if (metadataBlockFacetRoot) { + newBlockFacets = editedDv.getMetadataBlockFacets().stream().map(blockFacet -> { + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setDataverse(editedDv); + metadataBlockFacet.setMetadataBlock(blockFacet.getMetadataBlock()); + return metadataBlockFacet; + }).collect(Collectors.toList()); + } + editedDv.setMetadataBlockFacets(newBlockFacets); + + editedDv.setMetadataBlockFacetRoot(metadataBlockFacetRoot); + return ctxt.dataverses().save(editedDv); + } + + return editedDv; + } + + // Visible for testing + public Dataverse getEditedDataverse() { + return this.editedDv; + } + + // Visible for testing + public boolean getMetadataBlockFacetRoot() { + return metadataBlockFacetRoot; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommand.java new file mode 100644 index 00000000000..72a41f5cc3c --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommand.java @@ -0,0 +1,52 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.engine.command.AbstractCommand; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; + +import java.util.List; + +/** + * + * @author adaybujeda + */ +@RequiredPermissions( Permission.EditDataverse ) +public class UpdateMetadataBlockFacetsCommand extends AbstractCommand { + + private final Dataverse editedDv; + private final List metadataBlockFacets; + + public UpdateMetadataBlockFacetsCommand(DataverseRequest aRequest, Dataverse editedDv, List metadataBlockFacets) { + super(aRequest, editedDv); + this.editedDv = editedDv; + this.metadataBlockFacets = metadataBlockFacets; + } + + @Override + public Dataverse execute(CommandContext ctxt) throws CommandException { + if (!editedDv.isMetadataBlockFacetRoot()) { + throw new IllegalCommandException("Cannot update metadata blocks facets when dataverse has metadata block facet root set to false", this); + } + + editedDv.setMetadataBlockFacets(metadataBlockFacets); + Dataverse updated = ctxt.dataverses().save(editedDv); + return updated; + } + + // Visible for testing + public Dataverse getEditedDataverse() { + return this.editedDv; + } + + // Visible for testing + public List getMetadataBlockFacets() { + return this.metadataBlockFacets; + } + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/export/DublinCoreExporter.java b/src/main/java/edu/harvard/iq/dataverse/export/DublinCoreExporter.java index 7c4ebfdd44d..113e669f511 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/DublinCoreExporter.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/DublinCoreExporter.java @@ -18,7 +18,7 @@ public class DublinCoreExporter implements Exporter { - + @Override public String getProviderName() { return "oai_dc"; diff --git a/src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java b/src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java index 1952acb67a3..4bbcd653ac3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java @@ -129,7 +129,7 @@ private static void dtoddi(DatasetDTO datasetDto, OutputStream outputStream) thr xmlw.writeAttribute("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance"); xmlw.writeAttribute("xsi:schemaLocation", DDIExporter.DEFAULT_XML_NAMESPACE + " " + DDIExporter.DEFAULT_XML_SCHEMALOCATION); writeAttribute(xmlw, "version", DDIExporter.DEFAULT_XML_VERSION); - if(isMetadataLanguageSet(datasetDto.getMetadataLanguage())) { + if(DvObjectContainer.isMetadataLanguageSet(datasetDto.getMetadataLanguage())) { writeAttribute(xmlw, "xml:lang", datasetDto.getMetadataLanguage()); } createStdyDscr(xmlw, datasetDto); @@ -151,7 +151,7 @@ public static void datasetJson2ddi(JsonObject datasetDtoAsJson, DatasetVersion v xmlw.writeAttribute("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance"); xmlw.writeAttribute("xsi:schemaLocation", DDIExporter.DEFAULT_XML_NAMESPACE + " " + DDIExporter.DEFAULT_XML_SCHEMALOCATION); writeAttribute(xmlw, "version", DDIExporter.DEFAULT_XML_VERSION); - if(isMetadataLanguageSet(datasetDto.getMetadataLanguage())) { + if(DvObjectContainer.isMetadataLanguageSet(datasetDto.getMetadataLanguage())) { writeAttribute(xmlw, "xml:lang", datasetDto.getMetadataLanguage()); } createStdyDscr(xmlw, datasetDto); @@ -161,14 +161,6 @@ public static void datasetJson2ddi(JsonObject datasetDtoAsJson, DatasetVersion v xmlw.writeEndElement(); // codeBook xmlw.flush(); } - - - private static boolean isMetadataLanguageSet(String mdLang) { - if(mdLang!=null && !mdLang.equals(DvObjectContainer.UNDEFINED_METADATA_LANGUAGE_CODE)) { - return true; - } - return false; - } /** * @todo This is just a stub, copied from DDIExportServiceBean. It should @@ -944,7 +936,7 @@ private static void writeDistributorsElement(XMLStreamWriter xmlw, DatasetVersio } if (!distributorName.isEmpty()) { xmlw.writeStartElement("distrbtr"); - if(isMetadataLanguageSet(lang)) { + if(DvObjectContainer.isMetadataLanguageSet(lang)) { writeAttribute(xmlw, "xml:lang", lang); } if (!distributorAffiliation.isEmpty()) { @@ -1064,7 +1056,7 @@ private static void writeAbstractElement(XMLStreamWriter xmlw, DatasetVersionDTO if(!descriptionDate.isEmpty()){ writeAttribute(xmlw,"date",descriptionDate); } - if(isMetadataLanguageSet(lang)) { + if(DvObjectContainer.isMetadataLanguageSet(lang)) { writeAttribute(xmlw, "xml:lang", lang); } xmlw.writeCharacters(descriptionText); @@ -1538,7 +1530,7 @@ private static void writeFullElement (XMLStreamWriter xmlw, String name, String //For the simplest Elements we can if (!StringUtilisEmpty(value)) { xmlw.writeStartElement(name); - if(isMetadataLanguageSet(lang)) { + if(DvObjectContainer.isMetadataLanguageSet(lang)) { writeAttribute(xmlw, "xml:lang", lang); } xmlw.writeCharacters(value); diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java index c996e332bdb..7f94b1bbbbf 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java @@ -276,63 +276,6 @@ public JsonObjectBuilder toJson() { return jab; } - public enum ReservedWord { - - // TODO: Research if a format like "{reservedWord}" is easily parse-able or if another format would be - // better. The choice of curly braces is somewhat arbitrary, but has been observed in documenation for - // various REST APIs. For example, "Variable substitutions will be made when a variable is named in {brackets}." - // from https://swagger.io/specification/#fixed-fields-29 but that's for URLs. - FILE_ID("fileId"), - FILE_PID("filePid"), - SITE_URL("siteUrl"), - API_TOKEN("apiToken"), - // datasetId is the database id - DATASET_ID("datasetId"), - // datasetPid is the DOI or Handle - DATASET_PID("datasetPid"), - DATASET_VERSION("datasetVersion"), - FILE_METADATA_ID("fileMetadataId"), - LOCALE_CODE("localeCode"); - - private final String text; - private final String START = "{"; - private final String END = "}"; - - private ReservedWord(final String text) { - this.text = START + text + END; - } - - /** - * This is a centralized method that enforces that only reserved words - * are allowed to be used by external tools. External tool authors - * cannot pass their own query parameters through Dataverse such as - * "mode=mode1". - * - * @throws IllegalArgumentException - */ - public static ReservedWord fromString(String text) throws IllegalArgumentException { - if (text != null) { - for (ReservedWord reservedWord : ReservedWord.values()) { - if (text.equals(reservedWord.text)) { - return reservedWord; - } - } - } - // TODO: Consider switching to a more informative message that enumerates the valid reserved words. - boolean moreInformativeMessage = false; - if (moreInformativeMessage) { - throw new IllegalArgumentException("Unknown reserved word: " + text + ". A reserved word must be one of these values: " + Arrays.asList(ReservedWord.values()) + "."); - } else { - throw new IllegalArgumentException("Unknown reserved word: " + text); - } - } - - @Override - public String toString() { - return text; - } - } - public String getDescriptionLang() { String description = ""; if (this.toolName != null) { diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java index 9a5fcd2cac8..33d8c2d0d54 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java @@ -2,17 +2,14 @@ import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.FileMetadata; -import edu.harvard.iq.dataverse.GlobalId; import edu.harvard.iq.dataverse.authorization.users.ApiToken; -import edu.harvard.iq.dataverse.externaltools.ExternalTool.ReservedWord; -import edu.harvard.iq.dataverse.util.BundleUtil; -import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.URLTokenUtil; + import java.io.StringReader; import java.util.ArrayList; import java.util.List; -import java.util.logging.Logger; + import javax.json.Json; import javax.json.JsonArray; import javax.json.JsonObject; @@ -23,18 +20,9 @@ * instantiated. Applies logic based on an {@link ExternalTool} specification, * such as constructing a URL to access that file. */ -public class ExternalToolHandler { - - private static final Logger logger = Logger.getLogger(ExternalToolHandler.class.getCanonicalName()); +public class ExternalToolHandler extends URLTokenUtil { private final ExternalTool externalTool; - private final DataFile dataFile; - private final Dataset dataset; - private final FileMetadata fileMetadata; - - private ApiToken apiToken; - private String localeCode; - /** * File level tool * @@ -44,22 +32,8 @@ public class ExternalToolHandler { * used anonymously. */ public ExternalToolHandler(ExternalTool externalTool, DataFile dataFile, ApiToken apiToken, FileMetadata fileMetadata, String localeCode) { + super(dataFile, apiToken, fileMetadata, localeCode); this.externalTool = externalTool; - if (dataFile == null) { - String error = "A DataFile is required."; - logger.warning("Error in ExternalToolHandler constructor: " + error); - throw new IllegalArgumentException(error); - } - if (fileMetadata == null) { - String error = "A FileMetadata is required."; - logger.warning("Error in ExternalToolHandler constructor: " + error); - throw new IllegalArgumentException(error); - } - this.dataFile = dataFile; - this.apiToken = apiToken; - this.fileMetadata = fileMetadata; - dataset = fileMetadata.getDatasetVersion().getDataset(); - this.localeCode = localeCode; } /** @@ -71,33 +45,8 @@ public ExternalToolHandler(ExternalTool externalTool, DataFile dataFile, ApiToke * used anonymously. */ public ExternalToolHandler(ExternalTool externalTool, Dataset dataset, ApiToken apiToken, String localeCode) { + super(dataset, apiToken, localeCode); this.externalTool = externalTool; - if (dataset == null) { - String error = "A Dataset is required."; - logger.warning("Error in ExternalToolHandler constructor: " + error); - throw new IllegalArgumentException(error); - } - this.dataset = dataset; - this.apiToken = apiToken; - this.dataFile = null; - this.fileMetadata = null; - this.localeCode = localeCode; - } - - public DataFile getDataFile() { - return dataFile; - } - - public FileMetadata getFileMetadata() { - return fileMetadata; - } - - public ApiToken getApiToken() { - return apiToken; - } - - public String getLocaleCode() { - return localeCode; } // TODO: rename to handleRequest() to someday handle sending headers as well as query parameters. @@ -131,60 +80,6 @@ public String getQueryParametersForUrl(boolean preview) { } } - private String getQueryParam(String key, String value) { - ReservedWord reservedWord = ReservedWord.fromString(value); - switch (reservedWord) { - case FILE_ID: - // getDataFile is never null for file tools because of the constructor - return key + "=" + getDataFile().getId(); - case FILE_PID: - GlobalId filePid = getDataFile().getGlobalId(); - if (filePid != null) { - return key + "=" + getDataFile().getGlobalId(); - } - break; - case SITE_URL: - return key + "=" + SystemConfig.getDataverseSiteUrlStatic(); - case API_TOKEN: - String apiTokenString = null; - ApiToken theApiToken = getApiToken(); - if (theApiToken != null) { - apiTokenString = theApiToken.getTokenString(); - return key + "=" + apiTokenString; - } - break; - case DATASET_ID: - return key + "=" + dataset.getId(); - case DATASET_PID: - return key + "=" + dataset.getGlobalId().asString(); - case DATASET_VERSION: - String versionString = null; - if(fileMetadata!=null) { //true for file case - versionString = fileMetadata.getDatasetVersion().getFriendlyVersionNumber(); - } else { //Dataset case - return the latest visible version (unless/until the dataset case allows specifying a version) - if (getApiToken() != null) { - versionString = dataset.getLatestVersion().getFriendlyVersionNumber(); - } else { - versionString = dataset.getLatestVersionForCopy().getFriendlyVersionNumber(); - } - } - if (("DRAFT").equals(versionString)) { - versionString = ":draft"; // send the token needed in api calls that can be substituted for a numeric - // version. - } - return key + "=" + versionString; - case FILE_METADATA_ID: - if(fileMetadata!=null) { //true for file case - return key + "=" + fileMetadata.getId(); - } - case LOCALE_CODE: - return key + "=" + getLocaleCode(); - default: - break; - } - return null; - } - public String getToolUrlWithQueryParams() { return externalTool.getToolUrl() + getQueryParametersForUrl(); } @@ -209,8 +104,6 @@ public void setApiToken(ApiToken apiToken) { public String getExploreScript() { String toolUrl = this.getToolUrlWithQueryParams(); logger.fine("Exploring with " + toolUrl); - String msg = BundleUtil.getStringFromBundle("externaltools.enable.browser.popups"); - String script = "const newWin = window.open('" + toolUrl + "', target='_blank'); if (!newWin || newWin.closed || typeof newWin.closed == \"undefined\") {alert(\"" + msg + "\");}"; - return script; + return getScriptForUrl(toolUrl); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java index 95fd900e4d2..d49d66c26f7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java @@ -3,8 +3,9 @@ import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.DataFileServiceBean; import edu.harvard.iq.dataverse.authorization.users.ApiToken; -import edu.harvard.iq.dataverse.externaltools.ExternalTool.ReservedWord; import edu.harvard.iq.dataverse.externaltools.ExternalTool.Type; +import edu.harvard.iq.dataverse.util.URLTokenUtil; +import edu.harvard.iq.dataverse.util.URLTokenUtil.ReservedWord; import edu.harvard.iq.dataverse.externaltools.ExternalTool.Scope; import java.io.StringReader; diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java b/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java new file mode 100644 index 00000000000..9a963000541 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java @@ -0,0 +1,33 @@ +package edu.harvard.iq.dataverse.globus; + +import java.util.ArrayList; + +public class AccessList { + private int length; + private String endpoint; + private ArrayList DATA; + + public void setDATA(ArrayList DATA) { + this.DATA = DATA; + } + + public void setEndpoint(String endpoint) { + this.endpoint = endpoint; + } + + public void setLength(int length) { + this.length = length; + } + + public String getEndpoint() { + return endpoint; + } + + public ArrayList getDATA() { + return DATA; + } + + public int getLength() { + return length; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java b/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java new file mode 100644 index 00000000000..877fc68e4a1 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java @@ -0,0 +1,88 @@ +package edu.harvard.iq.dataverse.globus; + +import java.util.ArrayList; + +public class AccessToken implements java.io.Serializable { + + private String accessToken; + private String idToken; + private Long expiresIn; + private String resourceServer; + private String tokenType; + private String state; + private String scope; + private String refreshToken; + private ArrayList otherTokens; + + public String getAccessToken() { + return accessToken; + } + + String getIdToken() { + return idToken; + } + + Long getExpiresIn() { + return expiresIn; + } + + String getResourceServer() { + return resourceServer; + } + + String getTokenType() { + return tokenType; + } + + String getState() { + return state; + } + + String getScope() { + return scope; + } + + String getRefreshToken() { + return refreshToken; + } + + ArrayList getOtherTokens() { + return otherTokens; + } + + public void setAccessToken(String accessToken) { + this.accessToken = accessToken; + } + + public void setExpiresIn(Long expiresIn) { + this.expiresIn = expiresIn; + } + + public void setIdToken(String idToken) { + this.idToken = idToken; + } + + public void setOtherTokens(ArrayList otherTokens) { + this.otherTokens = otherTokens; + } + + public void setRefreshToken(String refreshToken) { + this.refreshToken = refreshToken; + } + + public void setResourceServer(String resourceServer) { + this.resourceServer = resourceServer; + } + + public void setScope(String scope) { + this.scope = scope; + } + + public void setState(String state) { + this.state = state; + } + + public void setTokenType(String tokenType) { + this.tokenType = tokenType; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java b/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java new file mode 100644 index 00000000000..0b8373cba09 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java @@ -0,0 +1,29 @@ +package edu.harvard.iq.dataverse.globus; + +public class FileDetailsHolder { + + private String hash; + private String mime; + private String storageID; + + public FileDetailsHolder(String id, String hash, String mime) { + + this.storageID = id; + this.hash = hash; + this.mime = mime; + + } + + public String getStorageID() { + return this.storageID; + } + + public String getHash() { + return hash; + } + + public String getMime() { + return mime; + } + +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java new file mode 100644 index 00000000000..9d80c5cc280 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java @@ -0,0 +1,1265 @@ +package edu.harvard.iq.dataverse.globus; + +import com.google.gson.FieldNamingPolicy; +import com.google.gson.GsonBuilder; +import edu.harvard.iq.dataverse.*; + +import javax.ejb.Asynchronous; +import javax.ejb.EJB; +import javax.ejb.Stateless; +import javax.ejb.TransactionAttribute; +import javax.ejb.TransactionAttributeType; +import javax.inject.Inject; +import javax.inject.Named; +import javax.json.Json; +import javax.json.JsonArray; +import javax.json.JsonArrayBuilder; +import javax.json.JsonObject; +import javax.json.JsonPatch; +import javax.servlet.http.HttpServletRequest; + +import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json; +import static edu.harvard.iq.dataverse.util.json.JsonPrinter.toJsonArray; + +import java.io.*; + +import java.net.HttpURLConnection; +import java.net.MalformedURLException; +import java.net.URL; +import java.net.URLEncoder; +import java.sql.Timestamp; +import java.text.SimpleDateFormat; +import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executor; +import java.util.concurrent.Executors; +import java.util.logging.FileHandler; +import java.util.logging.Level; +import java.util.logging.Logger; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import com.google.gson.Gson; +import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; +import edu.harvard.iq.dataverse.dataaccess.StorageIO; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import edu.harvard.iq.dataverse.util.FileUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.URLTokenUtil; +import edu.harvard.iq.dataverse.util.json.JsonUtil; + +@Stateless +@Named("GlobusServiceBean") +public class GlobusServiceBean implements java.io.Serializable { + + @EJB + protected DatasetServiceBean datasetSvc; + + @EJB + protected SettingsServiceBean settingsSvc; + + @Inject + DataverseSession session; + + @EJB + protected AuthenticationServiceBean authSvc; + + @EJB + EjbDataverseEngine commandEngine; + + @EJB + UserNotificationServiceBean userNotificationService; + + private static final Logger logger = Logger.getLogger(GlobusServiceBean.class.getCanonicalName()); + private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); + + private String code; + private String userTransferToken; + private String state; + + public String getState() { + return state; + } + + public void setState(String state) { + this.state = state; + } + + public String getCode() { + return code; + } + + public void setCode(String code) { + this.code = code; + } + + public String getUserTransferToken() { + return userTransferToken; + } + + public void setUserTransferToken(String userTransferToken) { + this.userTransferToken = userTransferToken; + } + + ArrayList checkPermisions(AccessToken clientTokenUser, String directory, String globusEndpoint, + String principalType, String principal) throws MalformedURLException { + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access_list"); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + ArrayList ids = new ArrayList(); + if (result.status == 200) { + AccessList al = parseJson(result.jsonResponse, AccessList.class, false); + + for (int i = 0; i < al.getDATA().size(); i++) { + Permissions pr = al.getDATA().get(i); + if ((pr.getPath().equals(directory + "/") || pr.getPath().equals(directory)) + && pr.getPrincipalType().equals(principalType) + && ((principal == null) || (principal != null && pr.getPrincipal().equals(principal)))) { + ids.add(pr.getId()); + } else { + logger.info(pr.getPath() + " === " + directory + " == " + pr.getPrincipalType()); + continue; + } + } + } + + return ids; + } + + public void updatePermision(AccessToken clientTokenUser, String directory, String principalType, String perm) + throws MalformedURLException { + if (directory != null && !directory.equals("")) { + directory = directory + "/"; + } + logger.info("Start updating permissions." + " Directory is " + directory); + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + ArrayList rules = checkPermisions(clientTokenUser, directory, globusEndpoint, principalType, null); + logger.info("Size of rules " + rules.size()); + int count = 0; + while (count < rules.size()) { + logger.info("Start removing rules " + rules.get(count)); + Permissions permissions = new Permissions(); + permissions.setDATA_TYPE("access"); + permissions.setPermissions(perm); + permissions.setPath(directory); + + Gson gson = new GsonBuilder().create(); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(count)); + logger.info("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(count)); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "PUT", gson.toJson(permissions)); + if (result.status != 200) { + logger.warning("Cannot update access rule " + rules.get(count)); + } else { + logger.info("Access rule " + rules.get(count) + " was updated"); + } + count++; + } + } + + public void deletePermision(String ruleId, Logger globusLogger) throws MalformedURLException { + + if (ruleId.length() > 0) { + AccessToken clientTokenUser = getClientToken(); + globusLogger.info("Start deleting permissions."); + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + + URL url = new URL( + "https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + ruleId); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "DELETE", null); + if (result.status != 200) { + globusLogger.warning("Cannot delete access rule " + ruleId); + } else { + globusLogger.info("Access rule " + ruleId + " was deleted successfully"); + } + } + + } + + public int givePermission(String principalType, String principal, String perm, AccessToken clientTokenUser, + String directory, String globusEndpoint) throws MalformedURLException { + + ArrayList rules = checkPermisions(clientTokenUser, directory, globusEndpoint, principalType, principal); + + Permissions permissions = new Permissions(); + permissions.setDATA_TYPE("access"); + permissions.setPrincipalType(principalType); + permissions.setPrincipal(principal); + permissions.setPath(directory + "/"); + permissions.setPermissions(perm); + + Gson gson = new GsonBuilder().create(); + MakeRequestResponse result = null; + if (rules.size() == 0) { + logger.info("Start creating the rule"); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access"); + result = makeRequest(url, "Bearer", clientTokenUser.getOtherTokens().get(0).getAccessToken(), "POST", + gson.toJson(permissions)); + + if (result.status == 400) { + logger.severe("Path " + permissions.getPath() + " is not valid"); + } else if (result.status == 409) { + logger.warning("ACL already exists or Endpoint ACL already has the maximum number of access rules"); + } + + return result.status; + } else { + logger.info("Start Updating the rule"); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(0)); + result = makeRequest(url, "Bearer", clientTokenUser.getOtherTokens().get(0).getAccessToken(), "PUT", + gson.toJson(permissions)); + + if (result.status == 400) { + logger.severe("Path " + permissions.getPath() + " is not valid"); + } else if (result.status == 409) { + logger.warning("ACL already exists or Endpoint ACL already has the maximum number of access rules"); + } + logger.info("Result status " + result.status); + } + + return result.status; + } + + public boolean getSuccessfulTransfers(AccessToken clientTokenUser, String taskId) throws MalformedURLException { + + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint_manager/task/" + taskId + + "/successful_transfers"); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + + if (result.status == 200) { + logger.info(" SUCCESS ====== "); + return true; + } + return false; + } + + public GlobusTask getTask(AccessToken clientTokenUser, String taskId, Logger globusLogger) throws MalformedURLException { + + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint_manager/task/" + taskId); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + + GlobusTask task = null; + + if (result.status == 200) { + task = parseJson(result.jsonResponse, GlobusTask.class, false); + } + if (result.status != 200) { + globusLogger.warning("Cannot find information for the task " + taskId + " : Reason : " + + result.jsonResponse.toString()); + } + + return task; + } + + public AccessToken getClientToken() throws MalformedURLException { + String globusBasicToken = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + URL url = new URL( + "https://auth.globus.org/v2/oauth2/token?scope=openid+email+profile+urn:globus:auth:scope:transfer.api.globus.org:all&grant_type=client_credentials"); + + MakeRequestResponse result = makeRequest(url, "Basic", globusBasicToken, "POST", null); + AccessToken clientTokenUser = null; + if (result.status == 200) { + clientTokenUser = parseJson(result.jsonResponse, AccessToken.class, true); + } + return clientTokenUser; + } + + public AccessToken getAccessToken(HttpServletRequest origRequest, String globusBasicToken) + throws UnsupportedEncodingException, MalformedURLException { + String serverName = origRequest.getServerName(); + if (serverName.equals("localhost")) { + logger.severe("Changing localhost to utoronto"); + serverName = "utl-192-123.library.utoronto.ca"; + } + + String redirectURL = "https://" + serverName + "/globus.xhtml"; + + redirectURL = URLEncoder.encode(redirectURL, "UTF-8"); + + URL url = new URL("https://auth.globus.org/v2/oauth2/token?code=" + code + "&redirect_uri=" + redirectURL + + "&grant_type=authorization_code"); + logger.info(url.toString()); + + MakeRequestResponse result = makeRequest(url, "Basic", globusBasicToken, "POST", null); + AccessToken accessTokenUser = null; + + if (result.status == 200) { + logger.info("Access Token: \n" + result.toString()); + accessTokenUser = parseJson(result.jsonResponse, AccessToken.class, true); + logger.info(accessTokenUser.getAccessToken()); + } + + return accessTokenUser; + + } + + public MakeRequestResponse makeRequest(URL url, String authType, String authCode, String method, + String jsonString) { + String str = null; + HttpURLConnection connection = null; + int status = 0; + try { + connection = (HttpURLConnection) url.openConnection(); + // Basic + // NThjMGYxNDQtN2QzMy00ZTYzLTk3MmUtMjljNjY5YzJjNGJiOktzSUVDMDZtTUxlRHNKTDBsTmRibXBIbjZvaWpQNGkwWVVuRmQyVDZRSnc9 + logger.info(authType + " " + authCode); + connection.setRequestProperty("Authorization", authType + " " + authCode); + // connection.setRequestProperty("Content-Type", + // "application/x-www-form-urlencoded"); + connection.setRequestMethod(method); + if (jsonString != null) { + connection.setRequestProperty("Content-Type", "application/json"); + connection.setRequestProperty("Accept", "application/json"); + logger.info(jsonString); + connection.setDoOutput(true); + OutputStreamWriter wr = new OutputStreamWriter(connection.getOutputStream()); + wr.write(jsonString); + wr.flush(); + } + + status = connection.getResponseCode(); + logger.info("Status now " + status); + InputStream result = connection.getInputStream(); + if (result != null) { + logger.info("Result is not null"); + str = readResultJson(result).toString(); + logger.info("str is "); + logger.info(result.toString()); + } else { + logger.info("Result is null"); + str = null; + } + + logger.info("status: " + status); + } catch (IOException ex) { + logger.info("IO"); + logger.severe(ex.getMessage()); + logger.info(ex.getCause().toString()); + logger.info(ex.getStackTrace().toString()); + } finally { + if (connection != null) { + connection.disconnect(); + } + } + MakeRequestResponse r = new MakeRequestResponse(str, status); + return r; + + } + + private StringBuilder readResultJson(InputStream in) { + StringBuilder sb = null; + try { + + BufferedReader br = new BufferedReader(new InputStreamReader(in)); + sb = new StringBuilder(); + String line; + while ((line = br.readLine()) != null) { + sb.append(line + "\n"); + } + br.close(); + logger.info(sb.toString()); + } catch (IOException e) { + sb = null; + logger.severe(e.getMessage()); + } + return sb; + } + + private T parseJson(String sb, Class jsonParserClass, boolean namingPolicy) { + if (sb != null) { + Gson gson = null; + if (namingPolicy) { + gson = new GsonBuilder().setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES).create(); + + } else { + gson = new GsonBuilder().create(); + } + T jsonClass = gson.fromJson(sb, jsonParserClass); + return jsonClass; + } else { + logger.severe("Bad respond from token rquest"); + return null; + } + } + + public String getDirectory(String datasetId) { + Dataset dataset = null; + String directory = null; + try { + dataset = datasetSvc.find(Long.parseLong(datasetId)); + if (dataset == null) { + logger.severe("Dataset not found " + datasetId); + return null; + } + String storeId = dataset.getStorageIdentifier(); + storeId.substring(storeId.indexOf("//") + 1); + directory = storeId.substring(storeId.indexOf("//") + 1); + logger.info(storeId); + logger.info(directory); + logger.info("Storage identifier:" + dataset.getIdentifierForFileStorage()); + return directory; + + } catch (NumberFormatException nfe) { + logger.severe(nfe.getMessage()); + + return null; + } + + } + + class MakeRequestResponse { + public String jsonResponse; + public int status; + + MakeRequestResponse(String jsonResponse, int status) { + this.jsonResponse = jsonResponse; + this.status = status; + } + + } + + private MakeRequestResponse findDirectory(String directory, AccessToken clientTokenUser, String globusEndpoint) + throws MalformedURLException { + URL url = new URL(" https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/ls?path=" + + directory + "/"); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + logger.info("find directory status:" + result.status); + + return result; + } + + public boolean giveGlobusPublicPermissions(String datasetId) + throws UnsupportedEncodingException, MalformedURLException { + + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + String globusBasicToken = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + if (globusEndpoint.equals("") || globusBasicToken.equals("")) { + return false; + } + AccessToken clientTokenUser = getClientToken(); + if (clientTokenUser == null) { + logger.severe("Cannot get client token "); + return false; + } + + String directory = getDirectory(datasetId); + logger.info(directory); + + MakeRequestResponse status = findDirectory(directory, clientTokenUser, globusEndpoint); + + if (status.status == 200) { + + /* + * FilesList fl = parseJson(status.jsonResponse, FilesList.class, false); + * ArrayList files = fl.getDATA(); if (files != null) { for (FileG file: + * files) { if (!file.getName().contains("cached") && + * !file.getName().contains(".thumb")) { int perStatus = + * givePermission("all_authenticated_users", "", "r", clientTokenUser, directory + * + "/" + file.getName(), globusEndpoint); logger.info("givePermission status " + * + perStatus + " for " + file.getName()); if (perStatus == 409) { + * logger.info("Permissions already exist or limit was reached for " + + * file.getName()); } else if (perStatus == 400) { + * logger.info("No file in Globus " + file.getName()); } else if (perStatus != + * 201) { logger.info("Cannot get permission for " + file.getName()); } } } } + */ + + int perStatus = givePermission("all_authenticated_users", "", "r", clientTokenUser, directory, + globusEndpoint); + logger.info("givePermission status " + perStatus); + if (perStatus == 409) { + logger.info("Permissions already exist or limit was reached"); + } else if (perStatus == 400) { + logger.info("No directory in Globus"); + } else if (perStatus != 201 && perStatus != 200) { + logger.info("Cannot give read permission"); + return false; + } + + } else if (status.status == 404) { + logger.info("There is no globus directory"); + } else { + logger.severe("Cannot find directory in globus, status " + status); + return false; + } + + return true; + } + + // Generates the URL to launch the Globus app + public String getGlobusAppUrlForDataset(Dataset d) { + return getGlobusAppUrlForDataset(d, true, null); + } + + public String getGlobusAppUrlForDataset(Dataset d, boolean upload, DataFile df) { + String localeCode = session.getLocaleCode(); + ApiToken apiToken = null; + User user = session.getUser(); + + if (user instanceof AuthenticatedUser) { + apiToken = authSvc.findApiTokenByUser((AuthenticatedUser) user); + + if ((apiToken == null) || (apiToken.getExpireTime().before(new Date()))) { + logger.fine("Created apiToken for user: " + user.getIdentifier()); + apiToken = authSvc.generateApiTokenForUser((AuthenticatedUser) user); + } + } + String storePrefix = ""; + String driverId = d.getEffectiveStorageDriverId(); + try { + storePrefix = DataAccess.getDriverPrefix(driverId); + } catch (Exception e) { + logger.warning("GlobusAppUrlForDataset: Failed to get storePrefix for " + driverId); + } + //Use URLTokenUtil for params currently in common with external tools. + URLTokenUtil tokenUtil = new URLTokenUtil(d, df, apiToken, localeCode); + String appUrl; + if (upload) { + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/upload?datasetPid={datasetPid}&siteUrl={siteUrl}&apiToken={apiToken}&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}"; + } else { + if (df == null) { + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/download?datasetPid={datasetPid}&siteUrl={siteUrl}" + + ((apiToken != null) ? "&apiToken={apiToken}" : "") + + "&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}"; + } else { + String rawStorageId = df.getStorageIdentifier(); + rawStorageId=rawStorageId.substring(rawStorageId.lastIndexOf(":")+1); + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/download-file?datasetPid={datasetPid}&siteUrl={siteUrl}" + + ((apiToken != null) ? "&apiToken={apiToken}" : "") + + "&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}&fileId={fileId}&storageIdentifier=" + + rawStorageId + "&fileName=" + df.getCurrentName(); + } + } + return tokenUtil.replaceTokensWithValues(appUrl) + "&storePrefix=" + storePrefix; + } + + public String getGlobusDownloadScript(Dataset dataset, ApiToken apiToken) { + return URLTokenUtil.getScriptForUrl(getGlobusAppUrlForDataset(dataset, false, null)); + + } + + @Asynchronous + @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) + public void globusUpload(String jsonData, ApiToken token, Dataset dataset, String httpRequestUrl, + AuthenticatedUser authUser) throws ExecutionException, InterruptedException, MalformedURLException { + + Integer countAll = 0; + Integer countSuccess = 0; + Integer countError = 0; + String logTimestamp = logFormatter.format(new Date()); + Logger globusLogger = Logger.getLogger( + "edu.harvard.iq.dataverse.upload.client.DatasetServiceBean." + "GlobusUpload" + logTimestamp); + String logFileName = "../logs" + File.separator + "globusUpload_id_" + dataset.getId() + "_" + logTimestamp + + ".log"; + FileHandler fileHandler; + boolean fileHandlerSuceeded; + try { + fileHandler = new FileHandler(logFileName); + globusLogger.setUseParentHandlers(false); + fileHandlerSuceeded = true; + } catch (IOException | SecurityException ex) { + Logger.getLogger(DatasetServiceBean.class.getName()).log(Level.SEVERE, null, ex); + return; + } + + if (fileHandlerSuceeded) { + globusLogger.addHandler(fileHandler); + } else { + globusLogger = logger; + } + + globusLogger.info("Starting an globusUpload "); + + String datasetIdentifier = dataset.getStorageIdentifier(); + + // ToDo - use DataAccess methods? + String storageType = datasetIdentifier.substring(0, datasetIdentifier.indexOf("://") + 3); + datasetIdentifier = datasetIdentifier.substring(datasetIdentifier.indexOf("://") + 3); + + Thread.sleep(5000); + + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(jsonData)) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + logger.log(Level.SEVERE, "Error parsing dataset json. Json: {0}"); + } + logger.info("json: " + JsonUtil.prettyPrint(jsonObject)); + + String taskIdentifier = jsonObject.getString("taskIdentifier"); + + String ruleId = ""; + try { + ruleId = jsonObject.getString("ruleId"); + } catch (NullPointerException npe) { + logger.warning("NPE for jsonData object"); + } + + // globus task status check + GlobusTask task = globusStatusCheck(taskIdentifier, globusLogger); + String taskStatus = getTaskStatus(task); + + if (ruleId.length() > 0) { + deletePermision(ruleId, globusLogger); + } + + // If success, switch to an EditInProgress lock - do this before removing the + // GlobusUpload lock + // Keeping a lock through the add datafiles API call avoids a conflicting edit + // and keeps any open dataset page refreshing until the datafile appears + if (!(taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE"))) { + datasetSvc.addDatasetLock(dataset, + new DatasetLock(DatasetLock.Reason.EditInProgress, authUser, "Completing Globus Upload")); + } + + DatasetLock gLock = dataset.getLockFor(DatasetLock.Reason.GlobusUpload); + if (gLock == null) { + logger.log(Level.WARNING, "No lock found for dataset"); + } else { + logger.log(Level.FINE, "Removing GlobusUpload lock " + gLock.getId()); + /* + * Note: This call to remove a lock only works immediately because it is in + * another service bean. Despite the removeDatasetLocks method having the + * REQUIRES_NEW transaction annotation, when the globusUpload method and that + * method were in the same bean (globusUpload was in the DatasetServiceBean to + * start), the globus lock was still seen in the API call initiated in the + * addFilesAsync method called within the globusUpload method. I.e. it appeared + * that the lock removal was not committed/visible outside this method until + * globusUpload itself ended. + */ + datasetSvc.removeDatasetLocks(dataset, DatasetLock.Reason.GlobusUpload); + } + + if (taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE")) { + String comment = "Reason : " + taskStatus.split("#")[1] + "
Short Description : " + + taskStatus.split("#")[2]; + userNotificationService.sendNotification((AuthenticatedUser) authUser, new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSUPLOADCOMPLETEDWITHERRORS, dataset.getId(), comment, true); + globusLogger.info("Globus task failed "); + + } else { + try { + // + + List inputList = new ArrayList(); + JsonArray filesJsonArray = jsonObject.getJsonArray("files"); + + if (filesJsonArray != null) { + + for (JsonObject fileJsonObject : filesJsonArray.getValuesAs(JsonObject.class)) { + + // storageIdentifier s3://gcs5-bucket1:1781cfeb8a7-748c270a227c from + // externalTool + String storageIdentifier = fileJsonObject.getString("storageIdentifier"); + String[] bits = storageIdentifier.split(":"); + String bucketName = bits[1].replace("/", ""); + String fileId = bits[bits.length - 1]; + + // fullpath s3://gcs5-bucket1/10.5072/FK2/3S6G2E/1781cfeb8a7-4ad9418a5873 + String fullPath = storageType + bucketName + "/" + datasetIdentifier + "/" + fileId; + String fileName = fileJsonObject.getString("fileName"); + + inputList.add(fileId + "IDsplit" + fullPath + "IDsplit" + fileName); + } + + // calculateMissingMetadataFields: checksum, mimetype + JsonObject newfilesJsonObject = calculateMissingMetadataFields(inputList, globusLogger); + JsonArray newfilesJsonArray = newfilesJsonObject.getJsonArray("files"); + + JsonArrayBuilder jsonDataSecondAPI = Json.createArrayBuilder(); + + for (JsonObject fileJsonObject : filesJsonArray.getValuesAs(JsonObject.class)) { + + countAll++; + String storageIdentifier = fileJsonObject.getString("storageIdentifier"); + String fileName = fileJsonObject.getString("fileName"); + String directoryLabel = fileJsonObject.getString("directoryLabel"); + String[] bits = storageIdentifier.split(":"); + String fileId = bits[bits.length - 1]; + + List newfileJsonObject = IntStream.range(0, newfilesJsonArray.size()) + .mapToObj(index -> ((JsonObject) newfilesJsonArray.get(index)).getJsonObject(fileId)) + .filter(Objects::nonNull).collect(Collectors.toList()); + + if (newfileJsonObject != null) { + if (!newfileJsonObject.get(0).getString("hash").equalsIgnoreCase("null")) { + JsonPatch path = Json.createPatchBuilder() + .add("/md5Hash", newfileJsonObject.get(0).getString("hash")).build(); + fileJsonObject = path.apply(fileJsonObject); + path = Json.createPatchBuilder() + .add("/mimeType", newfileJsonObject.get(0).getString("mime")).build(); + fileJsonObject = path.apply(fileJsonObject); + jsonDataSecondAPI.add(fileJsonObject); + countSuccess++; + } else { + globusLogger.info(fileName + + " will be skipped from adding to dataset by second API due to missing values "); + countError++; + } + } else { + globusLogger.info(fileName + + " will be skipped from adding to dataset by second API due to missing values "); + countError++; + } + } + + String newjsonData = jsonDataSecondAPI.build().toString(); + + globusLogger.info("Successfully generated new JsonData for Second API call"); + + String command = "curl -H \"X-Dataverse-key:" + token.getTokenString() + "\" -X POST " + + httpRequestUrl + "/api/datasets/:persistentId/addFiles?persistentId=doi:" + + datasetIdentifier + " -F jsonData='" + newjsonData + "'"; + System.out.println("*******====command ==== " + command); + + String output = addFilesAsync(command, globusLogger); + if (output.equalsIgnoreCase("ok")) { + // if(!taskSkippedFiles) + if (countError == 0) { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSUPLOADCOMPLETED, + dataset.getId(), countSuccess + " files added out of " + countAll, true); + } else { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSUPLOADCOMPLETEDWITHERRORS, dataset.getId(), + countSuccess + " files added out of " + countAll, true); + } + globusLogger.info("Successfully completed api/datasets/:persistentId/addFiles call "); + } else { + globusLogger.log(Level.SEVERE, + "******* Error while executing api/datasets/:persistentId/add call ", command); + } + + } + + globusLogger.info("Files processed: " + countAll.toString()); + globusLogger.info("Files added successfully: " + countSuccess.toString()); + globusLogger.info("Files failures: " + countError.toString()); + globusLogger.info("Finished upload via Globus job."); + + if (fileHandlerSuceeded) { + fileHandler.close(); + } + + } catch (Exception e) { + logger.info("Exception from globusUpload call "); + e.printStackTrace(); + globusLogger.info("Exception from globusUpload call " + e.getMessage()); + datasetSvc.removeDatasetLocks(dataset, DatasetLock.Reason.EditInProgress); + } + } + } + + public String addFilesAsync(String curlCommand, Logger globusLogger) + throws ExecutionException, InterruptedException { + CompletableFuture addFilesFuture = CompletableFuture.supplyAsync(() -> { + try { + Thread.sleep(2000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + return (addFiles(curlCommand, globusLogger)); + }, executor).exceptionally(ex -> { + globusLogger.fine("Something went wrong : " + ex.getLocalizedMessage()); + ex.printStackTrace(); + return null; + }); + + String result = addFilesFuture.get(); + + return result; + } + + private String addFiles(String curlCommand, Logger globusLogger) { + ProcessBuilder processBuilder = new ProcessBuilder(); + Process process = null; + String line; + String status = ""; + + try { + globusLogger.info("Call to : " + curlCommand); + processBuilder.command("bash", "-c", curlCommand); + process = processBuilder.start(); + process.waitFor(); + + BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream())); + + StringBuilder sb = new StringBuilder(); + while ((line = br.readLine()) != null) + sb.append(line); + globusLogger.info(" API Output : " + sb.toString()); + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(sb.toString())) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + globusLogger.log(Level.SEVERE, "Error parsing dataset json."); + } + + status = jsonObject.getString("status"); + } catch (Exception ex) { + globusLogger.log(Level.SEVERE, + "******* Unexpected Exception while executing api/datasets/:persistentId/add call ", ex); + } + + return status; + } + + @Asynchronous + public void globusDownload(String jsonData, Dataset dataset, User authUser) throws MalformedURLException { + + String logTimestamp = logFormatter.format(new Date()); + Logger globusLogger = Logger.getLogger( + "edu.harvard.iq.dataverse.upload.client.DatasetServiceBean." + "GlobusDownload" + logTimestamp); + + String logFileName = "../logs" + File.separator + "globusDownload_id_" + dataset.getId() + "_" + logTimestamp + + ".log"; + FileHandler fileHandler; + boolean fileHandlerSuceeded; + try { + fileHandler = new FileHandler(logFileName); + globusLogger.setUseParentHandlers(false); + fileHandlerSuceeded = true; + } catch (IOException | SecurityException ex) { + Logger.getLogger(DatasetServiceBean.class.getName()).log(Level.SEVERE, null, ex); + return; + } + + if (fileHandlerSuceeded) { + globusLogger.addHandler(fileHandler); + } else { + globusLogger = logger; + } + + globusLogger.info("Starting an globusDownload "); + + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(jsonData)) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + globusLogger.log(Level.SEVERE, "Error parsing dataset json. Json: {0}"); + } + + String taskIdentifier = jsonObject.getString("taskIdentifier"); + String ruleId = ""; + + try { + jsonObject.getString("ruleId"); + } catch (NullPointerException npe) { + + } + + // globus task status check + GlobusTask task = globusStatusCheck(taskIdentifier, globusLogger); + String taskStatus = getTaskStatus(task); + + if (ruleId.length() > 0) { + deletePermision(ruleId, globusLogger); + } + + if (taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE")) { + String comment = "Reason : " + taskStatus.split("#")[1] + "
Short Description : " + + taskStatus.split("#")[2]; + userNotificationService.sendNotification((AuthenticatedUser) authUser, new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSDOWNLOADCOMPLETEDWITHERRORS, dataset.getId(), comment, true); + globusLogger.info("Globus task failed during download process"); + } else { + boolean taskSkippedFiles = (task.getSkip_source_errors() == null) ? false : task.getSkip_source_errors(); + if (!taskSkippedFiles) { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSDOWNLOADCOMPLETED, + dataset.getId()); + } else { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSDOWNLOADCOMPLETEDWITHERRORS, + dataset.getId(), ""); + } + } + } + + Executor executor = Executors.newFixedThreadPool(10); + + private GlobusTask globusStatusCheck(String taskId, Logger globusLogger) throws MalformedURLException { + boolean taskCompletion = false; + String status = ""; + GlobusTask task = null; + int pollingInterval = SystemConfig.getIntLimitFromStringOrDefault(settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusPollingInterval), 50); + do { + try { + globusLogger.info("checking globus transfer task " + taskId); + Thread.sleep(pollingInterval * 1000); + AccessToken clientTokenUser = getClientToken(); + // success = globusServiceBean.getSuccessfulTransfers(clientTokenUser, taskId); + task = getTask(clientTokenUser, taskId, globusLogger); + if (task != null) { + status = task.getStatus(); + if (status != null) { + // The task is in progress. + if (status.equalsIgnoreCase("ACTIVE")) { + if (task.getNice_status().equalsIgnoreCase("ok") + || task.getNice_status().equalsIgnoreCase("queued")) { + taskCompletion = false; + } else { + taskCompletion = true; + // status = "FAILED" + "#" + task.getNice_status() + "#" + + // task.getNice_status_short_description(); + } + } else { + // The task is either succeeded, failed or inactive. + taskCompletion = true; + // status = status + "#" + task.getNice_status() + "#" + + // task.getNice_status_short_description(); + } + } else { + // status = "FAILED"; + taskCompletion = true; + } + } else { + // status = "FAILED"; + taskCompletion = true; + } + } catch (Exception ex) { + ex.printStackTrace(); + } + + } while (!taskCompletion); + + globusLogger.info("globus transfer task completed successfully"); + return task; + } + + private String getTaskStatus(GlobusTask task) { + String status = null; + if (task != null) { + status = task.getStatus(); + if (status != null) { + // The task is in progress. + if (status.equalsIgnoreCase("ACTIVE")) { + status = "FAILED" + "#" + task.getNice_status() + "#" + task.getNice_status_short_description(); + } else { + // The task is either succeeded, failed or inactive. + status = status + "#" + task.getNice_status() + "#" + task.getNice_status_short_description(); + } + } else { + status = "FAILED"; + } + } else { + status = "FAILED"; + } + return status; + } + + public JsonObject calculateMissingMetadataFields(List inputList, Logger globusLogger) + throws InterruptedException, ExecutionException, IOException { + + List> hashvalueCompletableFutures = inputList.stream() + .map(iD -> calculateDetailsAsync(iD, globusLogger)).collect(Collectors.toList()); + + CompletableFuture allFutures = CompletableFuture + .allOf(hashvalueCompletableFutures.toArray(new CompletableFuture[hashvalueCompletableFutures.size()])); + + CompletableFuture> allCompletableFuture = allFutures.thenApply(future -> { + return hashvalueCompletableFutures.stream().map(completableFuture -> completableFuture.join()) + .collect(Collectors.toList()); + }); + + CompletableFuture completableFuture = allCompletableFuture.thenApply(files -> { + return files.stream().map(d -> json(d)).collect(toJsonArray()); + }); + + JsonArrayBuilder filesObject = (JsonArrayBuilder) completableFuture.get(); + + JsonObject output = Json.createObjectBuilder().add("files", filesObject).build(); + + return output; + + } + + private CompletableFuture calculateDetailsAsync(String id, Logger globusLogger) { + // logger.info(" calcualte additional details for these globus id ==== " + id); + + return CompletableFuture.supplyAsync(() -> { + try { + Thread.sleep(2000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + try { + return (calculateDetails(id, globusLogger)); + } catch (InterruptedException | IOException e) { + e.printStackTrace(); + } + return null; + }, executor).exceptionally(ex -> { + return null; + }); + } + + private FileDetailsHolder calculateDetails(String id, Logger globusLogger) + throws InterruptedException, IOException { + int count = 0; + String checksumVal = ""; + InputStream in = null; + String fileId = id.split("IDsplit")[0]; + String fullPath = id.split("IDsplit")[1]; + String fileName = id.split("IDsplit")[2]; + + // ToDo: what if the file doesnot exists in s3 + // ToDo: what if checksum calculation failed + + do { + try { + StorageIO dataFileStorageIO = DataAccess.getDirectStorageIO(fullPath); + in = dataFileStorageIO.getInputStream(); + checksumVal = FileUtil.calculateChecksum(in, DataFile.ChecksumType.MD5); + count = 3; + } catch (IOException ioex) { + count = 3; + logger.info(ioex.getMessage()); + globusLogger.info("S3AccessIO: DataFile (fullPAth " + fullPath + + ") does not appear to be an S3 object associated with driver: "); + } catch (Exception ex) { + count = count + 1; + ex.printStackTrace(); + logger.info(ex.getMessage()); + Thread.sleep(5000); + } + + } while (count < 3); + + if (checksumVal.length() == 0) { + checksumVal = "NULL"; + } + + String mimeType = calculatemime(fileName); + globusLogger.info(" File Name " + fileName + " File Details " + fileId + " checksum = " + checksumVal + + " mimeType = " + mimeType); + return new FileDetailsHolder(fileId, checksumVal, mimeType); + // getBytes(in)+"" ); + // calculatemime(fileName)); + } + + public String calculatemime(String fileName) throws InterruptedException { + + String finalType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT; + String type = FileUtil.determineFileTypeByNameAndExtension(fileName); + + if (type!=null && !type.isBlank()) { + if (FileUtil.useRecognizedType(finalType, type)) { + finalType = type; + } + } + + return finalType; + } + /* + * public boolean globusFinishTransfer(Dataset dataset, AuthenticatedUser user) + * throws MalformedURLException { + * + * logger.info("=====Tasklist == dataset id :" + dataset.getId()); String + * directory = null; + * + * try { + * + * List fileMetadatas = new ArrayList<>(); + * + * StorageIO datasetSIO = DataAccess.getStorageIO(dataset); + * + * + * + * DatasetVersion workingVersion = dataset.getEditVersion(); + * + * if (workingVersion.getCreateTime() != null) { + * workingVersion.setCreateTime(new Timestamp(new Date().getTime())); } + * + * directory = dataset.getAuthorityForFileStorage() + "/" + + * dataset.getIdentifierForFileStorage(); + * + * System.out.println("======= directory ==== " + directory + + * " ==== datasetId :" + dataset.getId()); Map checksumMapOld + * = new HashMap<>(); + * + * Iterator fmIt = workingVersion.getFileMetadatas().iterator(); + * + * while (fmIt.hasNext()) { FileMetadata fm = fmIt.next(); if (fm.getDataFile() + * != null && fm.getDataFile().getId() != null) { String chksum = + * fm.getDataFile().getChecksumValue(); if (chksum != null) { + * checksumMapOld.put(chksum, 1); } } } + * + * List dFileList = new ArrayList<>(); boolean update = false; for + * (S3ObjectSummary s3ObjectSummary : datasetSIO.listAuxObjects("")) { + * + * String s3ObjectKey = s3ObjectSummary.getKey(); + * + * + * String t = s3ObjectKey.replace(directory, ""); + * + * if (t.indexOf(".") > 0) { long totalSize = s3ObjectSummary.getSize(); String + * filePath = s3ObjectKey; String fileName = + * filePath.split("/")[filePath.split("/").length - 1]; String fullPath = + * datasetSIO.getStorageLocation() + "/" + fileName; + * + * logger.info("Full path " + fullPath); StorageIO dataFileStorageIO = + * DataAccess.getDirectStorageIO(fullPath); InputStream in = + * dataFileStorageIO.getInputStream(); + * + * String checksumVal = FileUtil.calculateChecksum(in, + * DataFile.ChecksumType.MD5); //String checksumVal = s3ObjectSummary.getETag(); + * logger.info("The checksum is " + checksumVal); if + * ((checksumMapOld.get(checksumVal) != null)) { logger.info("datasetId :" + + * dataset.getId() + "======= filename ==== " + filePath + + * " == file already exists "); } else if (filePath.contains("cached") || + * filePath.contains(".thumb")) { logger.info(filePath + " is ignored"); } else + * { update = true; logger.info("datasetId :" + dataset.getId() + + * "======= filename ==== " + filePath + " == new file "); try { + * + * DataFile datafile = new DataFile(DataFileServiceBean.MIME_TYPE_GLOBUS_FILE); + * //MIME_TYPE_GLOBUS datafile.setModificationTime(new Timestamp(new + * Date().getTime())); datafile.setCreateDate(new Timestamp(new + * Date().getTime())); datafile.setPermissionModificationTime(new Timestamp(new + * Date().getTime())); + * + * FileMetadata fmd = new FileMetadata(); + * + * + * fmd.setLabel(fileName); fmd.setDirectoryLabel(filePath.replace(directory, + * "").replace(File.separator + fileName, "")); + * + * fmd.setDataFile(datafile); + * + * datafile.getFileMetadatas().add(fmd); + * + * FileUtil.generateS3PackageStorageIdentifierForGlobus(datafile); + * logger.info("==== datasetId :" + dataset.getId() + "======= filename ==== " + * + filePath + " == added to datafile, filemetadata "); + * + * try { // We persist "SHA1" rather than "SHA-1". + * //datafile.setChecksumType(DataFile.ChecksumType.SHA1); + * datafile.setChecksumType(DataFile.ChecksumType.MD5); + * datafile.setChecksumValue(checksumVal); } catch (Exception cksumEx) { + * logger.info("==== datasetId :" + dataset.getId() + + * "======Could not calculate checksumType signature for the new file "); } + * + * datafile.setFilesize(totalSize); + * + * dFileList.add(datafile); + * + * } catch (Exception ioex) { logger.info("datasetId :" + dataset.getId() + + * "======Failed to process and/or save the file " + ioex.getMessage()); return + * false; + * + * } } } } if (update) { + * + * List filesAdded = new ArrayList<>(); + * + * if (dFileList != null && dFileList.size() > 0) { + * + * // Dataset dataset = version.getDataset(); + * + * for (DataFile dataFile : dFileList) { + * + * if (dataFile.getOwner() == null) { dataFile.setOwner(dataset); + * + * workingVersion.getFileMetadatas().add(dataFile.getFileMetadata()); + * dataFile.getFileMetadata().setDatasetVersion(workingVersion); + * dataset.getFiles().add(dataFile); + * + * } + * + * filesAdded.add(dataFile); + * + * } + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ===== Done! Finished saving new files to the dataset."); } + * + * fileMetadatas.clear(); for (DataFile addedFile : filesAdded) { + * fileMetadatas.add(addedFile.getFileMetadata()); } filesAdded = null; + * + * if (workingVersion.isDraft()) { + * + * logger.info("Async: ==== datasetId :" + dataset.getId() + + * " ==== inside draft version "); + * + * Timestamp updateTime = new Timestamp(new Date().getTime()); + * + * workingVersion.setLastUpdateTime(updateTime); + * dataset.setModificationTime(updateTime); + * + * + * for (FileMetadata fileMetadata : fileMetadatas) { + * + * if (fileMetadata.getDataFile().getCreateDate() == null) { + * fileMetadata.getDataFile().setCreateDate(updateTime); + * fileMetadata.getDataFile().setCreator((AuthenticatedUser) user); } + * fileMetadata.getDataFile().setModificationTime(updateTime); } + * + * + * } else { logger.info("datasetId :" + dataset.getId() + + * " ==== inside released version "); + * + * for (int i = 0; i < workingVersion.getFileMetadatas().size(); i++) { for + * (FileMetadata fileMetadata : fileMetadatas) { if + * (fileMetadata.getDataFile().getStorageIdentifier() != null) { + * + * if (fileMetadata.getDataFile().getStorageIdentifier().equals(workingVersion. + * getFileMetadatas().get(i).getDataFile().getStorageIdentifier())) { + * workingVersion.getFileMetadatas().set(i, fileMetadata); } } } } + * + * + * } + * + * + * try { Command cmd; logger.info("Async: ==== datasetId :" + + * dataset.getId() + + * " ======= UpdateDatasetVersionCommand START in globus function "); cmd = new + * UpdateDatasetVersionCommand(dataset, new DataverseRequest(user, + * (HttpServletRequest) null)); ((UpdateDatasetVersionCommand) + * cmd).setValidateLenient(true); //new DataverseRequest(authenticatedUser, + * (HttpServletRequest) null) //dvRequestService.getDataverseRequest() + * commandEngine.submit(cmd); } catch (CommandException ex) { + * logger.log(Level.WARNING, "==== datasetId :" + dataset.getId() + + * "======CommandException updating DatasetVersion from batch job: " + + * ex.getMessage()); return false; } + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ======= GLOBUS CALL COMPLETED SUCCESSFULLY "); + * + * //return true; } + * + * } catch (Exception e) { String message = e.getMessage(); + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ======= GLOBUS CALL Exception ============== " + message); + * e.printStackTrace(); return false; //return + * error(Response.Status.INTERNAL_SERVER_ERROR, + * "Uploaded files have passed checksum validation but something went wrong while attempting to move the files into Dataverse. Message was '" + * + message + "'."); } + * + * String globusBasicToken = + * settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + * AccessToken clientTokenUser = getClientToken(globusBasicToken); + * updatePermision(clientTokenUser, directory, "identity", "r"); return true; } + * + */ +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java new file mode 100644 index 00000000000..c2b01779f4a --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java @@ -0,0 +1,92 @@ +package edu.harvard.iq.dataverse.globus; + +public class GlobusTask { + + private String DATA_TYPE; + private String type; + private String status; + private String owner_id; + private String request_time; + private String task_id; + private String destination_endpoint_display_name; + private boolean skip_source_errors; + private String nice_status; + private String nice_status_short_description; + + public String getDestination_endpoint_display_name() { + return destination_endpoint_display_name; + } + + public void setDestination_endpoint_display_name(String destination_endpoint_display_name) { + this.destination_endpoint_display_name = destination_endpoint_display_name; + } + + public void setRequest_time(String request_time) { + this.request_time = request_time; + } + + public String getRequest_time() { + return request_time; + } + + public String getTask_id() { + return task_id; + } + + public void setTask_id(String task_id) { + this.task_id = task_id; + } + + public String getDATA_TYPE() { + return DATA_TYPE; + } + + public void setDATA_TYPE(String DATA_TYPE) { + this.DATA_TYPE = DATA_TYPE; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public String getOwner_id() { + return owner_id; + } + + public void setOwner_id(String owner_id) { + this.owner_id = owner_id; + } + + public Boolean getSkip_source_errors() { + return skip_source_errors; + } + + public void setSkip_source_errors(Boolean skip_source_errors) { + this.skip_source_errors = skip_source_errors; + } + + public String getNice_status() { + return nice_status; + } + + public void setNice_status(String nice_status) { + this.nice_status = nice_status; + } + + public String getNice_status_short_description() { + return nice_status_short_description; + } + +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java b/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java new file mode 100644 index 00000000000..b8bb5193fa4 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java @@ -0,0 +1,58 @@ +package edu.harvard.iq.dataverse.globus; + +public class Permissions { + private String DATA_TYPE; + private String principal_type; + private String principal; + private String id; + private String path; + private String permissions; + + public void setPath(String path) { + this.path = path; + } + + public void setDATA_TYPE(String DATA_TYPE) { + this.DATA_TYPE = DATA_TYPE; + } + + public void setPermissions(String permissions) { + this.permissions = permissions; + } + + public void setPrincipal(String principal) { + this.principal = principal; + } + + public void setPrincipalType(String principalType) { + this.principal_type = principalType; + } + + public String getPath() { + return path; + } + + public String getDATA_TYPE() { + return DATA_TYPE; + } + + public String getPermissions() { + return permissions; + } + + public String getPrincipal() { + return principal; + } + + public String getPrincipalType() { + return principal_type; + } + + public void setId(String id) { + this.id = id; + } + + public String getId() { + return id; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java index 1add8e53ef0..b03bae618a4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java @@ -290,10 +290,6 @@ public List saveAndAddFilesToDataset(DatasetVersion version, } } } - - if (unattached) { - dataFile.setOwner(null); - } // Any necessary post-processing: // performPostProcessingTasks(dataFile); } else { @@ -302,6 +298,7 @@ public List saveAndAddFilesToDataset(DatasetVersion version, //Populate metadata dataAccess.open(DataAccessOption.READ_ACCESS); //set file size + logger.fine("Setting file size: " + dataAccess.getSize()); dataFile.setFilesize(dataAccess.getSize()); if(dataAccess instanceof S3AccessIO) { ((S3AccessIO)dataAccess).removeTempTag(); @@ -311,18 +308,20 @@ public List saveAndAddFilesToDataset(DatasetVersion version, + ioex.getMessage() + ")"); } savedSuccess = true; - dataFile.setOwner(null); } logger.fine("Done! Finished saving new files in permanent storage and adding them to the dataset."); boolean belowLimit = false; try { + //getting StorageIO may require knowing the owner (so this must come before owner is potentially set back to null belowLimit = dataFile.getStorageIO().isBelowIngestSizeLimit(); } catch (IOException e) { logger.warning("Error getting ingest limit for file: " + dataFile.getIdentifier() + " : " + e.getMessage()); } - + if (unattached) { + dataFile.setOwner(null); + } if (savedSuccess && belowLimit) { // These are all brand new files, so they should all have // one filemetadata total. -- L.A. diff --git a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java index bd66e822c20..484e5768eb1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java @@ -49,6 +49,7 @@ import java.util.List; import java.util.Locale; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.concurrent.Future; import java.util.function.Function; @@ -816,6 +817,7 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set langs = settingsService.getConfiguredLanguages(); Map cvocMap = datasetFieldService.getCVocConf(false); + Set metadataBlocksWithValue = new HashSet<>(); for (DatasetField dsf : datasetVersion.getFlatDatasetFields()) { DatasetFieldType dsfType = dsf.getDatasetFieldType(); @@ -823,6 +825,11 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set dataversePaths = retrieveDVOPaths(dataset); diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchFields.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchFields.java index 63b5a777b0e..2e75a81ed5f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchFields.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchFields.java @@ -206,6 +206,7 @@ public class SearchFields { * A dataverse, a dataset, or a file. */ public static final String TYPE = "dvObjectType"; + public static final String METADATA_TYPES = "metadata_type_ss"; public static final String NAME_SORT = "nameSort"; // PUBLICATION_YEAR used to be called PUBLICATION_DATE. public static final String PUBLICATION_YEAR = "publicationDate"; diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java index 6da4960679d..9bb83c88add 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java @@ -32,6 +32,7 @@ import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.logging.Logger; import javax.ejb.EJB; @@ -1150,9 +1151,20 @@ public List getFriendlyNamesFromFilterQuery(String filterQuery) { friendlyNames.add(key); } } + String noLeadingQuote = value.replaceAll("^\"", ""); String noTrailingQuote = noLeadingQuote.replaceAll("\"$", ""); String valueWithoutQuotes = noTrailingQuote; + + if (key.equals(SearchFields.METADATA_TYPES) && getDataverse() != null && getDataverse().getMetadataBlockFacets() != null) { + Optional friendlyName = getDataverse().getMetadataBlockFacets().stream().filter(block -> block.getMetadataBlock().getName().equals(valueWithoutQuotes)).findFirst().map(block -> block.getMetadataBlock().getLocaleDisplayFacet()); + logger.fine(String.format("action=getFriendlyNamesFromFilterQuery key=%s value=%s friendlyName=%s", key, value, friendlyName)); + if(friendlyName.isPresent()) { + friendlyNames.add(friendlyName.get()); + return friendlyNames; + } + } + friendlyNames.add(valueWithoutQuotes); return friendlyNames; } diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java index 8dc367ec5c9..ca158198204 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java @@ -7,6 +7,7 @@ import edu.harvard.iq.dataverse.DatasetVersionServiceBean; import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.DataverseFacet; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; import edu.harvard.iq.dataverse.DvObjectServiceBean; import edu.harvard.iq.dataverse.authorization.groups.Group; import edu.harvard.iq.dataverse.authorization.groups.GroupServiceBean; @@ -26,9 +27,11 @@ import java.util.Collections; import java.util.Date; import java.util.HashMap; +import java.util.LinkedList; import java.util.List; import java.util.ListIterator; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.MissingResourceException; import java.util.logging.Level; @@ -207,6 +210,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List metadataBlockFacets = new LinkedList<>(); //I'm not sure if just adding null here is good for hte permissions system... i think it needs something if(dataverses != null) { for(Dataverse dataverse : dataverses) { @@ -244,6 +249,8 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List 0) { if(metadataBlockName.length() > 0 ) { localefriendlyName = getLocaleTitle(datasetFieldName,facetFieldCount.getName(), metadataBlockName); - } else { + } else if (facetField.getName().equals(SearchFields.METADATA_TYPES)) { + Optional metadataBlockFacet = metadataBlockFacets.stream().filter(blockFacet -> blockFacet.getMetadataBlock().getName().equals(facetFieldCount.getName())).findFirst(); + if (metadataBlockFacet.isEmpty()) { + // metadata block facet is not configured to be displayed => ignore + continue; + } + + localefriendlyName = metadataBlockFacet.get().getMetadataBlock().getLocaleDisplayFacet(); + } else { try { localefriendlyName = BundleUtil.getStringFromPropertyFile(facetFieldCount.getName(), "Bundle"); } catch (Exception e) { @@ -694,7 +709,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List 0) { facetCategory.setFriendlyName(friendlyName); } else { @@ -749,7 +764,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List 0) { - FacetLabel facetLabel = new FacetLabel(start + "-" + end, new Long(rangeFacetCount.getCount())); + FacetLabel facetLabel = new FacetLabel(start + "-" + end, Long.valueOf(rangeFacetCount.getCount())); // special [12 TO 34] syntax for range facets facetLabel.setFilterQuery(rangeFacet.getName() + ":" + "[" + start + " TO " + end + "]"); facetLabelList.add(facetLabel); diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java new file mode 100644 index 00000000000..223e4b86da9 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -0,0 +1,352 @@ +package edu.harvard.iq.dataverse.settings; + +import org.eclipse.microprofile.config.ConfigProvider; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +/** + * Enum to store each and every JVM-based setting as a reference, + * much like the enum {@link SettingsServiceBean.Key} for DB settings. + * + * To be able to have more control over JVM settings names, + * avoid typos, maybe create lists of settings and so on, + * this enum will provide the place to add any old and new + * settings that are destined to be made at the JVM level. + * + * Further future extensions of this enum class include + * - adding predicates for validation and + * - adding data manipulation for aliased config names. + * + * To create a setting, simply add it within a scope: + * {@link JvmSettings#JvmSettings(JvmSettings, String)} + * + * Settings that might get renamed may provide their old names as aliases: + * {@link JvmSettings#JvmSettings(JvmSettings, String, String...)} + * + * Some scopes or settings may need one or more placeholders, simply don't give + * a key in these cases: + * {@link JvmSettings#JvmSettings(JvmSettings)} + * + */ +public enum JvmSettings { + // the upmost root scope - every setting shall start with it. + PREFIX("dataverse"), + + // GENERAL SETTINGS + VERSION(PREFIX, "version"), + BUILD(PREFIX, "build"), + + ; + + private static final String SCOPE_SEPARATOR = "."; + public static final String PLACEHOLDER_KEY = "%s"; + private static final Pattern OLD_NAME_PLACEHOLDER_PATTERN = Pattern.compile("%(\\d\\$)?s"); + + private final String key; + private final String scopedKey; + private final JvmSettings parent; + private final List oldNames; + private final int placeholders; + + /** + * Create a root scope. + * @param key The scopes name. + */ + JvmSettings(String key) { + this.key = key; + this.scopedKey = key; + this.parent = null; + this.oldNames = List.of(); + this.placeholders = 0; + } + + /** + * Create a scope or setting with a placeholder for a variable argument in it. + * Used to create "configurable objects" with certain attributes using dynamic, programmatic lookup. + * + * Any placeholder present in a settings full scoped key will be replaced when looked up + * via {@link #lookup(Class, String...)}. + * + * @param scope The parent scope. + */ + JvmSettings(JvmSettings scope) { + this.key = PLACEHOLDER_KEY; + this.scopedKey = scope.scopedKey + SCOPE_SEPARATOR + this.key; + this.parent = scope; + this.oldNames = List.of(); + this.placeholders = scope.placeholders + 1; + } + + /** + * Create a scope or setting with name it and associate with a parent scope. + * @param scope The parent scope. + * @param key The name of this scope or setting. + */ + JvmSettings(JvmSettings scope, String key) { + this.key = key; + this.scopedKey = scope.scopedKey + SCOPE_SEPARATOR + key; + this.parent = scope; + this.oldNames = List.of(); + this.placeholders = scope.placeholders; + } + + /** + * Create a setting with name it and associate with a parent scope. + * (Could also be a scope, but old names for scopes aren't the way this is designed.) + * + * When old names are given, these need to be given as fully scoped setting names! (Otherwise + * it would not be possible to switch between completely different scopes.) + * + * @param scope The parent scope of this setting. + * @param key The name of the setting. + * @param oldNames Any previous names this setting was known as. + * Must be given as fully scopes names, not just the old unscoped key/name. + * Used by {@link edu.harvard.iq.dataverse.settings.source.AliasConfigSource} to allow backward + * compatible, non-breaking deprecation and switching to new setting names. + */ + JvmSettings(JvmSettings scope, String key, String... oldNames) { + this.key = key; + this.scopedKey = scope.scopedKey + SCOPE_SEPARATOR + key; + this.parent = scope; + this.oldNames = Arrays.stream(oldNames).collect(Collectors.toUnmodifiableList()); + this.placeholders = scope.placeholders; + } + + private static final List aliased = new ArrayList<>(); + static { + for (JvmSettings setting : JvmSettings.values()) { + if (!setting.oldNames.isEmpty()) { + aliased.add(setting); + } + } + } + + /** + * Get all settings having old names to include them in {@link edu.harvard.iq.dataverse.settings.source.AliasConfigSource} + * @return List of settings with old alias names. Can be empty, but will not be null. + */ + public static List getAliasedSettings() { + return Collections.unmodifiableList(aliased); + } + + /** + * Return a list of old names to be used as aliases for backward compatibility. + * Will return empty list if no old names present. + * + * This method should only be used by {@link edu.harvard.iq.dataverse.settings.source.AliasConfigSource}. + * In case of a setting containing placeholder(s), it will check any old names given in the definition + * for presence of at least one placeholder plus it doesn't use more placeholders than available. + * (Old names containing placeholders for settings without any are checked, too.) + * + * Violations will result in a {@link IllegalArgumentException} and will be noticed during any test execution. + * A developer must fix the old name definition before shipping the code. + * + * @return List of old names, may be empty, but never null. + * @throws IllegalArgumentException When an old name has no or too many placeholders for this setting. + */ + public List getOldNames() { + if (needsVarArgs()) { + for (String name : oldNames) { + long matches = OLD_NAME_PLACEHOLDER_PATTERN.matcher(name).results().count(); + + if (matches == 0) { + throw new IllegalArgumentException("JvmSettings." + this.name() + "'s old name '" + + name + "' needs at least one placeholder"); + } else if (matches > this.placeholders) { + throw new IllegalArgumentException("JvmSettings." + this.name() + "'s old name '" + + name + "' has more placeholders than the current name"); + } + } + } else if (! this.oldNames.stream().noneMatch(OLD_NAME_PLACEHOLDER_PATTERN.asPredicate())) { + throw new IllegalArgumentException("JvmSettings." + this.name() + " has no placeholder but old name requires it"); + } + + return oldNames; + } + + /** + * Retrieve the scoped key for this setting. Scopes are separated by dots. + * If the setting contains placeholders, these will be represented as {@link #PLACEHOLDER_KEY}. + * + * @return The scoped key (or the key if no scope). Example: dataverse.subscope.subsubscope.key + */ + public String getScopedKey() { + return this.scopedKey; + } + + public Pattern getPatternizedKey() { + return Pattern.compile( + getScopedKey() + .replace(SCOPE_SEPARATOR, "\\.") + .replace(PLACEHOLDER_KEY, "(.+?)")); + } + + + /** + * Does this setting carry and placeholders for variable arguments? + * @return True if so, False otherwise. + */ + public boolean needsVarArgs() { + return this.placeholders > 0; + } + + /** + * Return the number of placeholders / variable arguments are necessary to lookup this setting. + * An exact match in the number of arguments will be necessary for a successful lookup. + * @return Number of placeholders for this scoped setting. + */ + public int numberOfVarArgs() { + return placeholders; + } + + /** + * Lookup this setting via MicroProfile Config as a required option (it will fail if not present). + * @throws java.util.NoSuchElementException - if the property is not defined or is defined as an empty string + * @return The setting as a String + */ + public String lookup() { + return lookup(String.class); + } + + /** + * Lookup this setting via MicroProfile Config as an optional setting. + * @return The setting as String wrapped in a (potentially empty) Optional + */ + public Optional lookupOptional() { + return lookupOptional(String.class); + } + + /** + * Lookup this setting via MicroProfile Config as a required option (it will fail if not present). + * + * @param klass The target type class to convert the setting to if found and not null + * @return The setting as an instance of {@link T} + * @param Target type to convert the setting to (you can create custom converters) + * + * @throws java.util.NoSuchElementException When the property is not defined or is defined as an empty string. + * @throws IllegalArgumentException When the settings value could not be converted to target type. + */ + public T lookup(Class klass) { + if (needsVarArgs()) { + throw new IllegalArgumentException("Cannot lookup a setting containing placeholders with this method."); + } + + // This must be done with the full-fledged lookup, as we cannot store the config in an instance or static + // variable, as the alias config source depends on this enum (circular dependency). This is easiest + // avoided by looking up the static cached config at the cost of a method invocation. + return ConfigProvider.getConfig().getValue(this.getScopedKey(), klass); + } + + /** + * Lookup this setting via MicroProfile Config as an optional setting. + * + * @param klass The target type class to convert the setting to if found and not null + * @param Target type to convert the setting to (you can create custom converters) + * @return The setting as an instance of {@link Optional} or an empty Optional + * + * @throws IllegalArgumentException When the settings value could not be converted to target type. + */ + public Optional lookupOptional(Class klass) { + if (needsVarArgs()) { + throw new IllegalArgumentException("Cannot lookup a setting containing variable arguments with this method."); + } + + // This must be done with the full-fledged lookup, as we cannot store the config in an instance or static + // variable, as the alias config source depends on this enum (circular dependency). This is easiest + // avoided by looking up the static cached config at the cost of a method invocation. + return ConfigProvider.getConfig().getOptionalValue(this.getScopedKey(), klass); + } + + /** + * Lookup a required setting containing placeholders for arguments like a name and return as plain String. + * To use type conversion, use {@link #lookup(Class, String...)}. + * + * @param arguments The var args to replace the placeholders of this setting. + * @return The value of the setting. + * + * @throws java.util.NoSuchElementException When the setting has not been set in any MPCONFIG source or is an empty string. + * @throws IllegalArgumentException When using it on a setting without placeholders. + * @throws IllegalArgumentException When not providing as many arguments as there are placeholders. + */ + public String lookup(String... arguments) { + return lookup(String.class, arguments); + } + + /** + * Lookup an optional setting containing placeholders for arguments like a name and return as plain String. + * To use type conversion, use {@link #lookupOptional(Class, String...)}. + * + * @param arguments The var args to replace the placeholders of this setting. + * @return The value as an instance of {@link Optional} or an empty Optional + * + * @throws IllegalArgumentException When using it on a setting without placeholders. + * @throws IllegalArgumentException When not providing as many arguments as there are placeholders. + */ + public Optional lookupOptional(String... arguments) { + return lookupOptional(String.class, arguments); + } + + /** + * Lookup a required setting containing placeholders for arguments like a name and return as converted type. + * To avoid type conversion, use {@link #lookup(String...)}. + * + * @param klass The target type class. + * @param arguments The var args to replace the placeholders of this setting. + * @param Target type to convert the setting to (you can create custom converters) + * @return The value of the setting, converted to the given type. + * + * @throws java.util.NoSuchElementException When the setting has not been set in any MPCONFIG source or is an empty string. + * @throws IllegalArgumentException When using it on a setting without placeholders. + * @throws IllegalArgumentException When not providing as many arguments as there are placeholders. + * @throws IllegalArgumentException When the settings value could not be converted to the target type. + */ + public T lookup(Class klass, String... arguments) { + if (needsVarArgs()) { + if (arguments == null || arguments.length != placeholders) { + throw new IllegalArgumentException("You must specify " + placeholders + " placeholder lookup arguments."); + } + return ConfigProvider.getConfig().getValue(this.insert(arguments), klass); + } + throw new IllegalArgumentException("Cannot lookup a setting without variable arguments with this method."); + } + + /** + * Lookup an optional setting containing placeholders for arguments like a name and return as converted type. + * To avoid type conversion, use {@link #lookupOptional(String...)}. + * + * @param klass The target type class. + * @param arguments The var args to replace the placeholders of this setting. + * @param Target type to convert the setting to (you can create custom converters) + * @return The value as an instance of {@link Optional} or an empty Optional + * + * @throws IllegalArgumentException When using it on a setting without placeholders. + * @throws IllegalArgumentException When not providing as many arguments as there are placeholders. + * @throws IllegalArgumentException When the settings value could not be converted to the target type. + */ + public Optional lookupOptional(Class klass, String... arguments) { + if (needsVarArgs()) { + if (arguments == null || arguments.length != placeholders) { + throw new IllegalArgumentException("You must specify " + placeholders + " placeholder lookup arguments."); + } + return ConfigProvider.getConfig().getOptionalValue(this.insert(arguments), klass); + } + throw new IllegalArgumentException("Cannot lookup a setting without variable arguments with this method."); + } + + /** + * Inject arguments into the placeholders of this setting. Will not do anything when no placeholders present. + * + * @param arguments The variable arguments to be inserted for the placeholders. + * @return The formatted setting name. + */ + public String insert(String... arguments) { + return String.format(this.getScopedKey(), (Object[]) arguments); + } + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index 12ae777f3f8..bb68152eeba 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -425,6 +425,14 @@ Whether Harvesting (OAI) service is enabled * Convert shibboleth AJP attributes from ISO-8859-1 to UTF-8 */ ShibAttributeCharacterSetConversionEnabled, + /** + *Return the last or first value of an array of affiliation names + */ + ShibAffiliationOrder, + /** + *Split the affiliation array on given string, default ";" + */ + ShibAffiliationSeparator, /** * Validate physical files for all the datafiles in the dataset when publishing */ @@ -449,6 +457,31 @@ Whether Harvesting (OAI) service is enabled * when the Distributor field (citation metadatablock) is set (true) */ ExportInstallationAsDistributorOnlyWhenNotSet, + + /** + * Basic Globus Token for Globus Application + */ + GlobusBasicToken, + /** + * GlobusEndpoint is Globus endpoint for Globus application + */ + GlobusEndpoint, + /** + * Comma separated list of Globus enabled stores + */ + GlobusStores, + /** Globus App URL + * + */ + GlobusAppUrl, + /** Globus Polling Interval how long in seconds Dataverse waits between checks on Globus upload status checks + * + */ + GlobusPollingInterval, + /**Enable single-file download/transfers for Globus + * + */ + GlobusSingleFileTransfer, /** * Optional external executables to run on the metadata for dataverses * and datasets being published; as an extra validation step, to @@ -522,7 +555,11 @@ Whether Harvesting (OAI) service is enabled * would also work) of never muted notifications that cannot be turned off by the users. AlwaysMuted setting overrides * Nevermuted setting warning is logged. */ - NeverMuted + NeverMuted, + /** + * LDN Inbox Allowed Hosts - a comma separated list of IP addresses allowed to submit messages to the inbox + */ + LDNMessageHosts ; @Override diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSource.java b/src/main/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSource.java index fbdbd982085..407f39ce0f9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSource.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSource.java @@ -1,31 +1,37 @@ package edu.harvard.iq.dataverse.settings.source; +import edu.harvard.iq.dataverse.settings.JvmSettings; import org.eclipse.microprofile.config.ConfigProvider; import org.eclipse.microprofile.config.spi.ConfigSource; import java.io.IOException; +import java.io.InputStream; import java.net.URL; +import java.util.ArrayList; import java.util.HashMap; import java.util.HashSet; +import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Properties; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.logging.Level; import java.util.logging.Logger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; /** * Enable using an old name for a new config name. * Usages will be logged and this source will ALWAYS stand back if the new name is used anywhere. - * - * By using a DbSettingConfigSource value (dataverse.settings.fromdb.XXX) as old name, we can - * alias a new name to an old db setting, enabling backward compatibility. */ public final class AliasConfigSource implements ConfigSource { private static final Logger logger = Logger.getLogger(AliasConfigSource.class.getName()); + private static final String ALIASES_PROP_FILE = "META-INF/microprofile-aliases.properties"; - private final ConcurrentHashMap aliases = new ConcurrentHashMap<>(); - private final String ALIASES_PROP_FILE = "META-INF/microprofile-aliases.properties"; + private final ConcurrentHashMap> aliases = new ConcurrentHashMap<>(); + private final ConcurrentHashMap> varArgAliases = new ConcurrentHashMap<>(); public AliasConfigSource() { try { @@ -34,33 +40,63 @@ public AliasConfigSource() { // store in our aliases map importAliases(aliasProps); } catch (IOException e) { - logger.info("Could not read from "+ALIASES_PROP_FILE+". Skipping MPCONFIG alias setup."); + // Usually it's an anti-pattern to catch the exception here, but skipping the file + // should be fine here, as it's optional. + logger.log(Level.INFO, "Could not read from "+ALIASES_PROP_FILE+". Skipping MPCONFIG alias setup.", e); } + + // Store all old names from JvmSettings + importJvmSettings(JvmSettings.getAliasedSettings()); + } + + private void importJvmSettings(List aliasedSettings) { + // First add all simple aliases not containing placeholders + aliasedSettings.stream() + .filter(s -> ! s.needsVarArgs()) + .forEach(setting -> aliases.put(setting.getScopedKey(), setting.getOldNames())); + + // Aliases with placeholders need to be compiled into a regex + aliasedSettings.stream() + .filter(JvmSettings::needsVarArgs) + .forEach(setting -> varArgAliases.put(setting.getPatternizedKey(), setting.getOldNames())); } - Properties readAliases(String filePath) throws IOException { + + private Properties readAliases(String filePath) throws IOException { // get resource from classpath ClassLoader classLoader = this.getClass().getClassLoader(); URL aliasesResource = classLoader.getResource(filePath); + + // Prevent errors if file not found or could not be loaded + if (aliasesResource == null) { + throw new IOException("Could not find or load, class loader returned null"); + } // load properties from file resource (parsing included) Properties aliasProps = new Properties(); - try { - aliasProps.load(aliasesResource.openStream()); - } catch (NullPointerException e) { - throw new IOException(e.getMessage()); + try (InputStream propStream = aliasesResource.openStream()) { + aliasProps.load(propStream); } return aliasProps; } - void importAliases(Properties aliasProps) { - aliasProps.forEach((key, value) -> aliases.put(key.toString(), value.toString())); + private void importAliases(Properties aliasProps) { + aliasProps.forEach((key, value) -> aliases.put(key.toString(), List.of(value.toString()))); + } + + // Has visibility "package" to be usable from test class! + void addAlias(String newName, List oldNames) { + this.aliases.put(newName, oldNames); + } + + // Has visibility "package" to be usable from test class! + void addAlias(Pattern newNamePattern, List oldNamePatterns) { + this.varArgAliases.put(newNamePattern, oldNamePatterns); } @Override public Map getProperties() { - // No, just no. We're not going to drop a list of stuff. We're only - // dealiasing on getValue(); + // No, just no. We're not going to drop a list of stuff. We're only de-aliasing on calls to getValue() return new HashMap<>(); } @@ -79,16 +115,63 @@ public int getOrdinal() { @Override public String getValue(String key) { - String value = null; + + // If the key is null or not starting with the prefix ("dataverse"), we are not going to jump through loops, + // avoiding computation overhead + if (key == null || ! key.startsWith(JvmSettings.PREFIX.getScopedKey())) { + return null; + } + + List oldNames = new ArrayList<>(); + + // Retrieve simple cases without placeholders if (this.aliases.containsKey(key)) { - String oldKey = this.aliases.get(key); - value = ConfigProvider.getConfig().getOptionalValue(oldKey, String.class).orElse(null); + oldNames.addAll(this.aliases.get(key)); + // Or try with regex patterns + } else { + // Seek for the given key within all the patterns for placeholder containing settings, + // returning a Matcher to extract the variable arguments as regex match groups. + Optional foundMatcher = varArgAliases.keySet().stream() + .map(pattern -> pattern.matcher(key)) + .filter(Matcher::matches) + .findFirst(); - if (value != null) { - logger.warning("Detected deprecated config option '"+oldKey+"' in use. Please update your config to use '"+key+"'."); + // Extract the matched groups and construct all old setting names with them + if (foundMatcher.isPresent()) { + Matcher matcher = foundMatcher.get(); + + List varArgs = new ArrayList<>(); + for (int i = 1; i <= matcher.groupCount(); i++) { + varArgs.add(matcher.group(i)); + } + Object[] args = varArgs.toArray(); + + this.varArgAliases + .get(matcher.pattern()) + .forEach(oldNamePattern -> oldNames.add(String.format(oldNamePattern, args))); + } + } + + // Return the first non-empty result + // NOTE: When there are multiple old names in use, they would conflict anyway. Upon deletion of one of the + // old settings the other becomes visible and triggers the warning again. There might even be different + // old settings in different sources, which might conflict, too (see ordinal value). + // NOTE: As the default is an empty oldNames array, loop will only be executed if anything was found before. + for (String oldName : oldNames) { + Optional value = ConfigProvider.getConfig().getOptionalValue(oldName, String.class); + + if (value.isPresent()) { + logger.log( + Level.WARNING, + "Detected deprecated config option {0} in use. Please update your config to use {1}.", + new String[]{oldName, key} + ); + return value.get(); } } - return value; + + // Sane default: nothing found. + return null; } @Override diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigHelper.java b/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigHelper.java deleted file mode 100644 index 7b9783dee06..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigHelper.java +++ /dev/null @@ -1,27 +0,0 @@ -package edu.harvard.iq.dataverse.settings.source; - -import edu.harvard.iq.dataverse.settings.SettingsServiceBean; - -import javax.annotation.PostConstruct; -import javax.ejb.EJB; -import javax.ejb.Singleton; -import javax.ejb.Startup; - -/** - * This is a small helper bean for the MPCONFIG DbSettingConfigSource. - * As it is a singleton and built at application start (=deployment), it will inject the (stateless) - * settings service into the MPCONFIG POJO once it's ready. - * - * MPCONFIG requires it's sources to be POJOs. No direct dependency injection possible. - */ -@Singleton -@Startup -public class DbSettingConfigHelper { - @EJB - SettingsServiceBean settingsSvc; - - @PostConstruct - public void injectService() { - DbSettingConfigSource.injectSettingsService(settingsSvc); - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSource.java b/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSource.java deleted file mode 100644 index 838cd415819..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSource.java +++ /dev/null @@ -1,83 +0,0 @@ -package edu.harvard.iq.dataverse.settings.source; - -import edu.harvard.iq.dataverse.settings.Setting; -import edu.harvard.iq.dataverse.settings.SettingsServiceBean; -import org.eclipse.microprofile.config.spi.ConfigSource; - -import java.time.Duration; -import java.time.Instant; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.logging.Logger; - -/** - * A caching wrapper around SettingServiceBean to provide database settings to MicroProfile Config API. - * Please be aware that this class relies on dependency injection during the application startup. - * Values will not be available before and a severe message will be logged to allow monitoring (potential race conditions) - * The settings will be cached for at least one minute, avoiding unnecessary database calls. - */ -public class DbSettingConfigSource implements ConfigSource { - - private static final Logger logger = Logger.getLogger(DbSettingConfigSource.class.getCanonicalName()); - private static final ConcurrentHashMap properties = new ConcurrentHashMap<>(); - private static Instant lastUpdate; - private static SettingsServiceBean settingsSvc; - public static final String PREFIX = "dataverse.settings.fromdb"; - - /** - * Let the SettingsServiceBean be injected by DbSettingConfigHelper with PostConstruct - * @param injected - */ - public static void injectSettingsService(SettingsServiceBean injected) { - settingsSvc = injected; - updateProperties(); - } - - /** - * Retrieve settings from the database via service and update cache. - */ - public static void updateProperties() { - // skip if the service has not been injected yet - if (settingsSvc == null) { - return; - } - properties.clear(); - Set dbSettings = settingsSvc.listAll(); - dbSettings.forEach(s -> properties.put(PREFIX+"."+s.getName().substring(1) + (s.getLang() == null ? "" : "."+s.getLang()), s.getContent())); - lastUpdate = Instant.now(); - } - - @Override - public Map getProperties() { - // if the cache is at least XX number of seconds old, update before serving data. - if (lastUpdate == null || Instant.now().minus(Duration.ofSeconds(60)).isAfter(lastUpdate)) { - updateProperties(); - } - return properties; - } - - @Override - public Set getPropertyNames() { - return getProperties().keySet(); - } - - @Override - public int getOrdinal() { - return 50; - } - - @Override - public String getValue(String key) { - // log usages for which this has been designed, but not yet ready to serve... - if (settingsSvc == null && key.startsWith(PREFIX)) { - logger.severe("MPCONFIG DbSettingConfigSource not ready yet, but requested for '"+key+"'."); - } - return getProperties().getOrDefault(key, null); - } - - @Override - public String getName() { - return "DataverseDB"; - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/spi/DbSettingConfigSourceProvider.java b/src/main/java/edu/harvard/iq/dataverse/settings/spi/DbSettingConfigSourceProvider.java deleted file mode 100644 index 856a2c64a01..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/settings/spi/DbSettingConfigSourceProvider.java +++ /dev/null @@ -1,14 +0,0 @@ -package edu.harvard.iq.dataverse.settings.spi; - -import edu.harvard.iq.dataverse.settings.source.DbSettingConfigSource; -import org.eclipse.microprofile.config.spi.ConfigSource; -import org.eclipse.microprofile.config.spi.ConfigSourceProvider; - -import java.util.Arrays; - -public class DbSettingConfigSourceProvider implements ConfigSourceProvider { - @Override - public Iterable getConfigSources(ClassLoader forClassLoader) { - return Arrays.asList(new DbSettingConfigSource()); - } -} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/util/FileTypeDetection.java b/src/main/java/edu/harvard/iq/dataverse/util/FileTypeDetection.java deleted file mode 100644 index 52515c00524..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/util/FileTypeDetection.java +++ /dev/null @@ -1,12 +0,0 @@ -package edu.harvard.iq.dataverse.util; - -import java.io.File; -import java.io.IOException; - -public class FileTypeDetection { - - public static String determineFileType(File file) throws IOException { - return FileUtil.determineFileType(file, file.getName()); - } - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java index 67c1c464cb9..339de904f9e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java @@ -32,6 +32,7 @@ import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter; import edu.harvard.iq.dataverse.dataaccess.S3AccessIO; import edu.harvard.iq.dataverse.dataset.DatasetThumbnail; +import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.datasetutility.FileExceedsMaxSizeException; import static edu.harvard.iq.dataverse.datasetutility.FileSizeChecker.bytesToHumanReadable; import edu.harvard.iq.dataverse.ingest.IngestReport; @@ -545,6 +546,9 @@ public static String determineFileType(File f, String fileName) throws IOExcepti } } + if(fileType==null) { + fileType = MIME_TYPE_UNDETERMINED_DEFAULT; + } logger.fine("returning fileType "+fileType); return fileType; } @@ -1185,7 +1189,7 @@ public static CreateDataFileResult createDataFiles(DatasetVersion version, Input } // end createDataFiles - private static boolean useRecognizedType(String suppliedContentType, String recognizedType) { + public static boolean useRecognizedType(String suppliedContentType, String recognizedType) { // is it any better than the type that was supplied to us, // if any? // This is not as trivial a task as one might expect... @@ -1415,7 +1419,7 @@ public static void generateS3PackageStorageIdentifier(DataFile dataFile) { String driverId = dataFile.getOwner().getEffectiveStorageDriverId(); String bucketName = System.getProperty("dataverse.files." + driverId + ".bucket-name"); - String storageId = driverId + "://" + bucketName + ":" + dataFile.getFileMetadata().getLabel(); + String storageId = driverId + DataAccess.SEPARATOR + bucketName + ":" + dataFile.getFileMetadata().getLabel(); dataFile.setStorageIdentifier(storageId); } @@ -1534,7 +1538,7 @@ private static Boolean popupDueToStateOrTerms(DatasetVersion datasetVersion) { } // 1. License and Terms of Use: if (datasetVersion.getTermsOfUseAndAccess() != null) { - License license = datasetVersion.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(datasetVersion); if ((license == null && StringUtils.isNotBlank(datasetVersion.getTermsOfUseAndAccess().getTermsOfUse())) || (license != null && !license.isDefault())) { logger.fine("Popup required because of license or terms of use."); @@ -1617,32 +1621,33 @@ public static String getPublicDownloadUrl(String dataverseSiteUrl, String persis */ public static String getFileDownloadUrlPath(String downloadType, Long fileId, boolean gbRecordsWritten, Long fileMetadataId) { String fileDownloadUrl = "/api/access/datafile/" + fileId; - if (downloadType != null && downloadType.equals("bundle")) { - if (fileMetadataId == null) { - fileDownloadUrl = "/api/access/datafile/bundle/" + fileId; - } else { - fileDownloadUrl = "/api/access/datafile/bundle/" + fileId + "?fileMetadataId=" + fileMetadataId; - } - } - if (downloadType != null && downloadType.equals("original")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=original"; - } - if (downloadType != null && downloadType.equals("RData")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=RData"; - } - if (downloadType != null && downloadType.equals("var")) { - if (fileMetadataId == null) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata"; - } else { - fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata?fileMetadataId=" + fileMetadataId; + if (downloadType != null) { + switch(downloadType) { + case "original": + case"RData": + case "tab": + case "GlobusTransfer": + fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=" + downloadType; + break; + case "bundle": + if (fileMetadataId == null) { + fileDownloadUrl = "/api/access/datafile/bundle/" + fileId; + } else { + fileDownloadUrl = "/api/access/datafile/bundle/" + fileId + "?fileMetadataId=" + fileMetadataId; + } + break; + case "var": + if (fileMetadataId == null) { + fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata"; + } else { + fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata?fileMetadataId=" + fileMetadataId; + } + break; + } + } - } - if (downloadType != null && downloadType.equals("tab")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=tab"; - } if (gbRecordsWritten) { - if (downloadType != null && ((downloadType.equals("original") || downloadType.equals("RData") || downloadType.equals("tab")) || - ((downloadType.equals("var") || downloadType.equals("bundle") ) && fileMetadataId != null))) { + if (fileDownloadUrl.contains("?")) { fileDownloadUrl += "&gbrecs=true"; } else { fileDownloadUrl += "?gbrecs=true"; @@ -1780,10 +1785,10 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio StorageIO storage = dataFile.getStorageIO(); InputStream in = null; - + try { storage.open(DataAccessOption.READ_ACCESS); - + if (!dataFile.isTabularData()) { in = storage.getInputStream(); } else { @@ -1838,7 +1843,7 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio } finally { IOUtils.closeQuietly(in); } - // try again: + // try again: if (recalculatedChecksum.equals(dataFile.getChecksumValue())) { fixed = true; try { @@ -1849,10 +1854,11 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio } } } - + if (!fixed) { String info = BundleUtil.getStringFromBundle("dataset.publish.file.validation.error.wrongChecksumValue", Arrays.asList(dataFile.getId().toString())); logger.log(Level.INFO, info); + logger.fine("Expected: " + dataFile.getChecksumValue() +", calculated: " + recalculatedChecksum); throw new IOException(info); } } @@ -1861,7 +1867,7 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio } public static String getStorageIdentifierFromLocation(String location) { - int driverEnd = location.indexOf("://") + 3; + int driverEnd = location.indexOf(DataAccess.SEPARATOR) + DataAccess.SEPARATOR.length(); int bucketEnd = driverEnd + location.substring(driverEnd).indexOf("/"); return location.substring(0,bucketEnd) + ":" + location.substring(location.lastIndexOf("/") + 1); } @@ -1897,7 +1903,7 @@ public static void deleteTempFile(DataFile dataFile, Dataset dataset, IngestServ } } String si = dataFile.getStorageIdentifier(); - if (si.contains("://")) { + if (si.contains(DataAccess.SEPARATOR)) { //Direct upload files will already have a store id in their storageidentifier //but they need to be associated with a dataset for the overall storagelocation to be calculated //so we temporarily set the owner @@ -1916,7 +1922,7 @@ public static void deleteTempFile(DataFile dataFile, Dataset dataset, IngestServ } catch (IOException ioEx) { // safe to ignore - it's just a temp file. logger.warning(ioEx.getMessage()); - if(dataFile.getStorageIdentifier().contains("://")) { + if(dataFile.getStorageIdentifier().contains(DataAccess.SEPARATOR)) { logger.warning("Failed to delete temporary file " + dataFile.getStorageIdentifier()); } else { logger.warning("Failed to delete temporary file " + FileUtil.getFilesTempDirectory() + "/" diff --git a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java index 55c6f4d83d6..e76e2c5696b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java @@ -75,6 +75,38 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti } catch (Exception e) { return BundleUtil.getStringFromBundle("notification.email.import.filesystem.subject", rootDvNameAsList); } + case GLOBUSUPLOADCOMPLETED: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompleted.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompleted.subject", rootDvNameAsList); + } + case GLOBUSDOWNLOADCOMPLETED: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompleted.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompleted.subject", rootDvNameAsList); + } + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompletedWithErrors.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompletedWithErrors.subject", rootDvNameAsList); + } + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompletedWithErrors.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompletedWithErrors.subject", rootDvNameAsList); + } case CHECKSUMIMPORT: return BundleUtil.getStringFromBundle("notification.email.import.checksum.subject", rootDvNameAsList); @@ -86,8 +118,10 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti return BundleUtil.getStringFromBundle("notification.email.ingestCompleted.subject", rootDvNameAsList); case INGESTCOMPLETEDWITHERRORS: return BundleUtil.getStringFromBundle("notification.email.ingestCompletedWithErrors.subject", rootDvNameAsList); + case DATASETMENTIONED: + return BundleUtil.getStringFromBundle("notification.email.datasetWasMentioned.subject", rootDvNameAsList); } return ""; } -} +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java b/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java index 3131afbf010..ef74819f073 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java @@ -7,7 +7,7 @@ import org.apache.commons.text.StringEscapeUtils; import org.jsoup.Jsoup; -import org.jsoup.safety.Whitelist; +import org.jsoup.safety.Safelist; import org.jsoup.parser.Parser; /** @@ -20,8 +20,8 @@ public class MarkupChecker { /** - * Wrapper around Jsoup clean method with the basic White list - * http://jsoup.org/cookbook/cleaning-html/whitelist-sanitizer + * Wrapper around Jsoup clean method with the basic Safe list + * http://jsoup.org/cookbook/cleaning-html/safelist-sanitizer * @param unsafe * @return */ @@ -33,18 +33,18 @@ public static String sanitizeBasicHTML(String unsafe) { // basic includes: a, b, blockquote, br, cite, code, dd, dl, dt, em, i, li, ol, p, pre, q, small, span, strike, strong, sub, sup, u, ul //Whitelist wl = Whitelist.basic().addTags("img", "h1", "h2", "h3", "kbd", "hr", "s", "del"); - Whitelist wl = Whitelist.basicWithImages().addTags("h1", "h2", "h3", "kbd", "hr", "s", "del", "map", "area").addAttributes("img", "usemap") + Safelist sl = Safelist.basicWithImages().addTags("h1", "h2", "h3", "kbd", "hr", "s", "del", "map", "area").addAttributes("img", "usemap") .addAttributes("map", "name").addAttributes("area", "shape", "coords", "href", "title", "alt") .addEnforcedAttribute("a", "target", "_blank"); - return Jsoup.clean(unsafe, wl); + return Jsoup.clean(unsafe, sl); } /** * Strip all HTMl tags * - * http://jsoup.org/apidocs/org/jsoup/safety/Whitelist.html#none%28%29 + * http://jsoup.org/apidocs/org/jsoup/safety/Safelist.html#none * * @param unsafe * @return @@ -55,7 +55,7 @@ public static String stripAllTags(String unsafe) { return null; } - return Parser.unescapeEntities(Jsoup.clean(unsafe, Whitelist.none()), true); + return Parser.unescapeEntities(Jsoup.clean(unsafe, Safelist.none()), true); } diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index bd27405fae5..7abd0d02065 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -422,7 +422,7 @@ public static long getLongLimitFromStringOrDefault(String limitSetting, Long def if (limitSetting != null && !limitSetting.equals("")) { try { - limit = new Long(limitSetting); + limit = Long.valueOf(limitSetting); } catch (NumberFormatException nfe) { limit = null; } @@ -431,12 +431,12 @@ public static long getLongLimitFromStringOrDefault(String limitSetting, Long def return limit != null ? limit : defaultValue; } - static int getIntLimitFromStringOrDefault(String limitSetting, Integer defaultValue) { + public static int getIntLimitFromStringOrDefault(String limitSetting, Integer defaultValue) { Integer limit = null; if (limitSetting != null && !limitSetting.equals("")) { try { - limit = new Integer(limitSetting); + limit = Integer.valueOf(limitSetting); } catch (NumberFormatException nfe) { limit = null; } @@ -579,7 +579,7 @@ public Integer getSearchHighlightFragmentSize() { } return null; } - + public long getTabularIngestSizeLimit() { // This method will return the blanket ingestable size limit, if // set on the system. I.e., the universal limit that applies to all @@ -856,7 +856,14 @@ public enum FileUploadMethods { * Traditional Dataverse file handling, which tends to involve users * uploading and downloading files using a browser or APIs. */ - NATIVE("native/http"); + NATIVE("native/http"), + + /** + * Upload through Globus of large files + */ + + GLOBUS("globus") + ; private final String text; @@ -896,7 +903,9 @@ public enum FileDownloadMethods { * go through Glassfish. */ RSYNC("rsal/rsync"), - NATIVE("native/http"); + NATIVE("native/http"), + GLOBUS("globus") + ; private final String text; private FileDownloadMethods(final String text) { @@ -984,15 +993,19 @@ public boolean isPublicInstall(){ } public boolean isRsyncUpload(){ - return getUploadMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString(), true); } - + + public boolean isGlobusUpload(){ + return getMethodAvailable(FileUploadMethods.GLOBUS.toString(), true); + } + // Controls if HTTP upload is enabled for both GUI and API. public boolean isHTTPUpload(){ - return getUploadMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString(), true); } - public boolean isRsyncOnly(){ + public boolean isRsyncOnly(){ String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); if(downloadMethods == null){ return false; @@ -1005,26 +1018,37 @@ public boolean isRsyncOnly(){ return false; } else { return Arrays.asList(uploadMethods.toLowerCase().split("\\s*,\\s*")).size() == 1 && uploadMethods.toLowerCase().equals(SystemConfig.FileUploadMethods.RSYNC.toString()); - } + } } public boolean isRsyncDownload() { - String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); - return downloadMethods !=null && downloadMethods.toLowerCase().contains(SystemConfig.FileDownloadMethods.RSYNC.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString(), false); } public boolean isHTTPDownload() { - String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); - logger.warning("Download Methods:" + downloadMethods); - return downloadMethods !=null && downloadMethods.toLowerCase().contains(SystemConfig.FileDownloadMethods.NATIVE.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString(), false); + } + + public boolean isGlobusDownload() { + return getMethodAvailable(FileUploadMethods.GLOBUS.toString(), false); } - private Boolean getUploadMethodAvailable(String method){ - String uploadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.UploadMethods); - if (uploadMethods==null){ + public boolean isGlobusFileDownload() { + return (isGlobusDownload() && settingsService.isTrueForKey(SettingsServiceBean.Key.GlobusSingleFileTransfer, false)); + } + + public List getGlobusStoresList() { + String globusStores = settingsService.getValueForKey(SettingsServiceBean.Key.GlobusStores, ""); + return Arrays.asList(globusStores.split("\\s*,\\s*")); + } + + private Boolean getMethodAvailable(String method, boolean upload) { + String methods = settingsService.getValueForKey( + upload ? SettingsServiceBean.Key.UploadMethods : SettingsServiceBean.Key.DownloadMethods); + if (methods == null) { return false; } else { - return Arrays.asList(uploadMethods.toLowerCase().split("\\s*,\\s*")).contains(method); + return Arrays.asList(methods.toLowerCase().split("\\s*,\\s*")).contains(method); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java new file mode 100644 index 00000000000..b3d5f9d6b74 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java @@ -0,0 +1,250 @@ +package edu.harvard.iq.dataverse.util; + +import java.util.Arrays; +import java.util.logging.Logger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.FileMetadata; +import edu.harvard.iq.dataverse.GlobalId; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; + +public class URLTokenUtil { + + protected static final Logger logger = Logger.getLogger(URLTokenUtil.class.getCanonicalName()); + protected final DataFile dataFile; + protected final Dataset dataset; + protected final FileMetadata fileMetadata; + protected ApiToken apiToken; + protected String localeCode; + + /** + * File level + * + * @param dataFile Required. + * @param apiToken The apiToken can be null + * @param fileMetadata Required. + * @param localeCode optional. + * + */ + public URLTokenUtil(DataFile dataFile, ApiToken apiToken, FileMetadata fileMetadata, String localeCode) + throws IllegalArgumentException { + if (dataFile == null) { + String error = "A DataFile is required."; + logger.warning("Error in URLTokenUtil constructor: " + error); + throw new IllegalArgumentException(error); + } + if (fileMetadata == null) { + String error = "A FileMetadata is required."; + logger.warning("Error in URLTokenUtil constructor: " + error); + throw new IllegalArgumentException(error); + } + this.dataFile = dataFile; + this.dataset = fileMetadata.getDatasetVersion().getDataset(); + this.fileMetadata = fileMetadata; + this.apiToken = apiToken; + this.localeCode = localeCode; + } + + /** + * Dataset level + * + * @param dataset Required. + * @param apiToken The apiToken can be null + */ + public URLTokenUtil(Dataset dataset, ApiToken apiToken, String localeCode) { + this(dataset, null, apiToken, localeCode); + } + + /** + * Dataset level + * + * @param dataset Required. + * @param datafile Optional. + * @param apiToken Optional The apiToken can be null + * @localeCode Optional + * + */ + public URLTokenUtil(Dataset dataset, DataFile datafile, ApiToken apiToken, String localeCode) { + if (dataset == null) { + String error = "A Dataset is required."; + logger.warning("Error in URLTokenUtil constructor: " + error); + throw new IllegalArgumentException(error); + } + this.dataset = dataset; + this.dataFile = datafile; + this.fileMetadata = null; + this.apiToken = apiToken; + this.localeCode = localeCode; + } + + public DataFile getDataFile() { + return dataFile; + } + + public FileMetadata getFileMetadata() { + return fileMetadata; + } + + public ApiToken getApiToken() { + return apiToken; + } + + public String getLocaleCode() { + return localeCode; + } + + public String getQueryParam(String key, String value) { + String tokenValue = null; + tokenValue = getTokenValue(value); + if (tokenValue != null) { + return key + '=' + tokenValue; + } else { + return null; + } + } + + /** + * Tries to replace all occurrences of {} with the value for the + * corresponding ReservedWord + * + * @param url - the input string in which to replace tokens, normally a url + * @throws IllegalArgumentException if there is no matching ReservedWord or if + * the configuation of this instance doesn't + * have values for this ReservedWord (e.g. + * asking for FILE_PID when using the dataset + * constructor, etc.) + */ + public String replaceTokensWithValues(String url) { + String newUrl = url; + Pattern pattern = Pattern.compile("(\\{.*?\\})"); + Matcher matcher = pattern.matcher(url); + while(matcher.find()) { + String token = matcher.group(1); + ReservedWord reservedWord = ReservedWord.fromString(token); + String tValue = getTokenValue(token); + logger.fine("Replacing " + reservedWord.toString() + " with " + tValue + " in " + newUrl); + newUrl = newUrl.replace(reservedWord.toString(), tValue); + } + return newUrl; + } + + private String getTokenValue(String value) { + ReservedWord reservedWord = ReservedWord.fromString(value); + switch (reservedWord) { + case FILE_ID: + // getDataFile is never null for file tools because of the constructor + return getDataFile().getId().toString(); + case FILE_PID: + GlobalId filePid = getDataFile().getGlobalId(); + if (filePid != null) { + return getDataFile().getGlobalId().asString(); + } + break; + case SITE_URL: + return SystemConfig.getDataverseSiteUrlStatic(); + case API_TOKEN: + String apiTokenString = null; + ApiToken theApiToken = getApiToken(); + if (theApiToken != null) { + apiTokenString = theApiToken.getTokenString(); + } + return apiTokenString; + case DATASET_ID: + return dataset.getId().toString(); + case DATASET_PID: + return dataset.getGlobalId().asString(); + case DATASET_VERSION: + String versionString = null; + if (fileMetadata != null) { // true for file case + versionString = fileMetadata.getDatasetVersion().getFriendlyVersionNumber(); + } else { // Dataset case - return the latest visible version (unless/until the dataset + // case allows specifying a version) + if (getApiToken() != null) { + versionString = dataset.getLatestVersion().getFriendlyVersionNumber(); + } else { + versionString = dataset.getLatestVersionForCopy().getFriendlyVersionNumber(); + } + } + if (("DRAFT").equals(versionString)) { + versionString = ":draft"; // send the token needed in api calls that can be substituted for a numeric + // version. + } + return versionString; + case FILE_METADATA_ID: + if (fileMetadata != null) { // true for file case + return fileMetadata.getId().toString(); + } + case LOCALE_CODE: + return getLocaleCode(); + default: + break; + } + throw new IllegalArgumentException("Cannot replace reserved word: " + value); + } + + public static String getScriptForUrl(String url) { + String msg = BundleUtil.getStringFromBundle("externaltools.enable.browser.popups"); + String script = "const newWin = window.open('" + url + "', target='_blank'); if (!newWin || newWin.closed || typeof newWin.closed == \"undefined\") {alert(\"" + msg + "\");}"; + return script; + } + + public enum ReservedWord { + + // TODO: Research if a format like "{reservedWord}" is easily parse-able or if + // another format would be + // better. The choice of curly braces is somewhat arbitrary, but has been + // observed in documentation for + // various REST APIs. For example, "Variable substitutions will be made when a + // variable is named in {brackets}." + // from https://swagger.io/specification/#fixed-fields-29 but that's for URLs. + FILE_ID("fileId"), FILE_PID("filePid"), SITE_URL("siteUrl"), API_TOKEN("apiToken"), + // datasetId is the database id + DATASET_ID("datasetId"), + // datasetPid is the DOI or Handle + DATASET_PID("datasetPid"), DATASET_VERSION("datasetVersion"), FILE_METADATA_ID("fileMetadataId"), + LOCALE_CODE("localeCode"); + + private final String text; + private final String START = "{"; + private final String END = "}"; + + private ReservedWord(final String text) { + this.text = START + text + END; + } + + /** + * This is a centralized method that enforces that only reserved words are + * allowed to be used by external tools. External tool authors cannot pass their + * own query parameters through Dataverse such as "mode=mode1". + * + * @throws IllegalArgumentException + */ + public static ReservedWord fromString(String text) throws IllegalArgumentException { + if (text != null) { + for (ReservedWord reservedWord : ReservedWord.values()) { + if (text.equals(reservedWord.text)) { + return reservedWord; + } + } + } + // TODO: Consider switching to a more informative message that enumerates the + // valid reserved words. + boolean moreInformativeMessage = false; + if (moreInformativeMessage) { + throw new IllegalArgumentException( + "Unknown reserved word: " + text + ". A reserved word must be one of these values: " + + Arrays.asList(ReservedWord.values()) + "."); + } else { + throw new IllegalArgumentException("Unknown reserved word: " + text); + } + } + + @Override + public String toString() { + return text; + } + } +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/util/UrlSignerUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/UrlSignerUtil.java new file mode 100644 index 00000000000..b11334520e6 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/util/UrlSignerUtil.java @@ -0,0 +1,151 @@ +package edu.harvard.iq.dataverse.util; + +import java.net.URL; +import java.nio.charset.Charset; +import java.util.List; +import java.util.logging.Level; +import java.util.logging.Logger; + +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.http.NameValuePair; +import org.apache.http.client.utils.URLEncodedUtils; +import org.joda.time.LocalDateTime; + +/** + * Simple class to sign/validate URLs. + * + */ +public class UrlSignerUtil { + + private static final Logger logger = Logger.getLogger(UrlSignerUtil.class.getName()); + + /** + * + * @param baseUrl - the URL to sign - cannot contain query params + * "until","user", "method", or "token" + * @param timeout - how many minutes to make the URL valid for (note - time skew + * between the creator and receiver could affect the validation + * @param user - a string representing the user - should be understood by the + * creator/receiver + * @param method - one of the HTTP methods + * @param key - a secret key shared by the creator/receiver. In Dataverse + * this could be an APIKey (when sending URL to a tool that will + * use it to retrieve info from Dataverse) + * @return - the signed URL + */ + public static String signUrl(String baseUrl, Integer timeout, String user, String method, String key) { + StringBuilder signedUrl = new StringBuilder(baseUrl); + + boolean firstParam = true; + if (baseUrl.contains("?")) { + firstParam = false; + } + if (timeout != null) { + LocalDateTime validTime = LocalDateTime.now(); + validTime = validTime.plusMinutes(timeout); + validTime.toString(); + signedUrl.append(firstParam ? "?" : "&").append("until=").append(validTime); + firstParam = false; + } + if (user != null) { + signedUrl.append(firstParam ? "?" : "&").append("user=").append(user); + firstParam = false; + } + if (method != null) { + signedUrl.append(firstParam ? "?" : "&").append("method=").append(method); + firstParam=false; + } + signedUrl.append(firstParam ? "?" : "&").append("token="); + logger.fine("String to sign: " + signedUrl.toString() + ""); + signedUrl.append(DigestUtils.sha512Hex(signedUrl.toString() + key)); + logger.fine("Generated Signed URL: " + signedUrl.toString()); + if (logger.isLoggable(Level.FINE)) { + logger.fine( + "URL signature is " + (isValidUrl(signedUrl.toString(), user, method, key) ? "valid" : "invalid")); + } + return signedUrl.toString(); + } + + /** + * This method will only return true if the URL and parameters except the + * "token" are unchanged from the original/match the values sent to this method, + * and the "token" parameter matches what this method recalculates using the + * shared key THe method also assures that the "until" timestamp is after the + * current time. + * + * @param signedUrl - the signed URL as received from Dataverse + * @param method - an HTTP method. If provided, the method in the URL must + * match + * @param user - a string representing the user, if provided the value must + * match the one in the url + * @param key - the shared secret key to be used in validation + * @return - true if valid, false if not: e.g. the key is not the same as the + * one used to generate the "token" any part of the URL preceding the + * "token" has been altered the method doesn't match (e.g. the server + * has received a POST request and the URL only allows GET) the user + * string doesn't match (e.g. the server knows user A is logged in, but + * the URL is only for user B) the url has expired (was used after the + * until timestamp) + */ + public static boolean isValidUrl(String signedUrl, String user, String method, String key) { + boolean valid = true; + try { + URL url = new URL(signedUrl); + List params = URLEncodedUtils.parse(url.getQuery(), Charset.forName("UTF-8")); + String hash = null; + String dateString = null; + String allowedMethod = null; + String allowedUser = null; + for (NameValuePair nvp : params) { + if (nvp.getName().equals("token")) { + hash = nvp.getValue(); + logger.fine("Hash: " + hash); + } + if (nvp.getName().equals("until")) { + dateString = nvp.getValue(); + logger.fine("Until: " + dateString); + } + if (nvp.getName().equals("method")) { + allowedMethod = nvp.getValue(); + logger.fine("Method: " + allowedMethod); + } + if (nvp.getName().equals("user")) { + allowedUser = nvp.getValue(); + logger.fine("User: " + allowedUser); + } + } + + int index = signedUrl.indexOf(((dateString==null && allowedMethod==null && allowedUser==null) ? "?":"&") + "token="); + // Assuming the token is last - doesn't have to be, but no reason for the URL + // params to be rearranged either, and this should only cause false negatives if + // it does happen + String urlToHash = signedUrl.substring(0, index + 7); + logger.fine("String to hash: " + urlToHash + ""); + String newHash = DigestUtils.sha512Hex(urlToHash + key); + logger.fine("Calculated Hash: " + newHash); + if (!hash.equals(newHash)) { + logger.fine("Hash doesn't match"); + valid = false; + } + if (dateString != null && LocalDateTime.parse(dateString).isBefore(LocalDateTime.now())) { + logger.fine("Url is expired"); + valid = false; + } + if (method != null && !method.equals(allowedMethod)) { + logger.fine("Method doesn't match"); + valid = false; + } + if (user != null && !user.equals(allowedUser)) { + logger.fine("User doesn't match"); + valid = false; + } + } catch (Throwable t) { + // Want to catch anything like null pointers, etc. to force valid=false upon any + // error + logger.warning("Bad URL: " + signedUrl + " : " + t.getMessage()); + valid = false; + } + return valid; + } + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java b/src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java index a295f264d66..a6d85e1addb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java @@ -97,82 +97,11 @@ public JsonObjectBuilder getOREMapBuilder(boolean aggregationOnly) throws Except for (DatasetField field : fields) { if (!field.isEmpty()) { DatasetFieldType dfType = field.getDatasetFieldType(); - if (excludeEmail && DatasetFieldType.FieldType.EMAIL.equals(dfType.getFieldType())) { - continue; - } JsonLDTerm fieldName = dfType.getJsonLDTerm(); - if (fieldName.inNamespace()) { - localContext.putIfAbsent(fieldName.getNamespace().getPrefix(), fieldName.getNamespace().getUrl()); - } else { - localContext.putIfAbsent(fieldName.getLabel(), fieldName.getUrl()); - } - JsonArrayBuilder vals = Json.createArrayBuilder(); - if (!dfType.isCompound()) { - for (String val : field.getValues_nondisplay()) { - if (cvocMap.containsKey(dfType.getId())) { - try { - JsonObject cvocEntry = cvocMap.get(dfType.getId()); - if (cvocEntry.containsKey("retrieval-filtering")) { - JsonObject filtering = cvocEntry.getJsonObject("retrieval-filtering"); - JsonObject context = filtering.getJsonObject("@context"); - for (String prefix : context.keySet()) { - localContext.putIfAbsent(prefix, context.getString(prefix)); - } - vals.add(datasetFieldService.getExternalVocabularyValue(val)); - } else { - vals.add(val); - } - } catch(Exception e) { - logger.warning("Couldn't interpret value for : " + val + " : " + e.getMessage()); - logger.log(Level.FINE, ExceptionUtils.getStackTrace(e)); - vals.add(val); - } - } else { - vals.add(val); - } - } - } else { - // ToDo: Needs to be recursive (as in JsonPrinter?) - for (DatasetFieldCompoundValue dscv : field.getDatasetFieldCompoundValues()) { - // compound values are of different types - JsonObjectBuilder child = Json.createObjectBuilder(); - - for (DatasetField dsf : dscv.getChildDatasetFields()) { - DatasetFieldType dsft = dsf.getDatasetFieldType(); - if (excludeEmail && DatasetFieldType.FieldType.EMAIL.equals(dsft.getFieldType())) { - continue; - } - // which may have multiple values - if (!dsf.isEmpty()) { - // Add context entry - // ToDo - also needs to recurse here? - JsonLDTerm subFieldName = dsft.getJsonLDTerm(); - if (subFieldName.inNamespace()) { - localContext.putIfAbsent(subFieldName.getNamespace().getPrefix(), - subFieldName.getNamespace().getUrl()); - } else { - localContext.putIfAbsent(subFieldName.getLabel(), subFieldName.getUrl()); - } - - List values = dsf.getValues_nondisplay(); - if (values.size() > 1) { - JsonArrayBuilder childVals = Json.createArrayBuilder(); - - for (String val : dsf.getValues_nondisplay()) { - childVals.add(val); - } - child.add(subFieldName.getLabel(), childVals); - } else { - child.add(subFieldName.getLabel(), values.get(0)); - } - } - } - vals.add(child); - } + JsonValue jv = getJsonLDForField(field, excludeEmail, cvocMap, localContext); + if(jv!=null) { + aggBuilder.add(fieldName.getLabel(), jv); } - // Add metadata value to aggregation, suppress array when only one value - JsonArray valArray = vals.build(); - aggBuilder.add(fieldName.getLabel(), (valArray.size() != 1) ? valArray : valArray.get(0)); } } // Add metadata related to the Dataset/DatasetVersion @@ -219,7 +148,7 @@ public JsonObjectBuilder getOREMapBuilder(boolean aggregationOnly) throws Except aggBuilder.add(JsonLDTerm.schemaOrg("isPartOf").getLabel(), getDataverseDescription(dataset.getOwner())); String mdl = dataset.getMetadataLanguage(); - if(!mdl.equals(DvObjectContainer.UNDEFINED_METADATA_LANGUAGE_CODE)) { + if (DvObjectContainer.isMetadataLanguageSet(mdl)) { aggBuilder.add(JsonLDTerm.schemaOrg("inLanguage").getLabel(), mdl); } @@ -237,7 +166,32 @@ public JsonObjectBuilder getOREMapBuilder(boolean aggregationOnly) throws Except } else { addIfNotNull(aggRes, JsonLDTerm.schemaOrg("description"), df.getDescription()); } - addIfNotNull(aggRes, JsonLDTerm.schemaOrg("name"), fmd.getLabel()); // "label" is the filename + String fileName = fmd.getLabel();// "label" is the filename + long fileSize = df.getFilesize(); + String mimeType = df.getContentType(); + String currentIngestedName = null; + boolean ingested=df.getOriginalFileName()!= null || df.getOriginalFileSize()!=null || df.getOriginalFileFormat()!=null; + if(ingested) { + if(df.getOriginalFileName()!=null) { + currentIngestedName= fileName; + fileName = df.getOriginalFileName(); + } else { + logger.warning("Missing Original file name for id: " + df.getId()); + } + if(df.getOriginalFileSize()!=null) { + fileSize = df.getOriginalFileSize(); + } else { + logger.warning("Missing Original file size for id: " + df.getId()); + } + if(df.getOriginalFileFormat()!=null) { + mimeType = df.getOriginalFileFormat(); + } else { + logger.warning("Missing Original file format for id: " + df.getId()); + } + + + } + addIfNotNull(aggRes, JsonLDTerm.schemaOrg("name"), fileName); addIfNotNull(aggRes, JsonLDTerm.restricted, fmd.isRestricted()); addIfNotNull(aggRes, JsonLDTerm.directoryLabel, fmd.getDirectoryLabel()); addIfNotNull(aggRes, JsonLDTerm.schemaOrg("version"), fmd.getVersion()); @@ -260,21 +214,20 @@ public JsonObjectBuilder getOREMapBuilder(boolean aggregationOnly) throws Except if (df.getGlobalId().asString().length() != 0) { fileId = df.getGlobalId().asString(); fileSameAs = SystemConfig.getDataverseSiteUrlStatic() - + "/api/access/datafile/:persistentId?persistentId=" + fileId; + + "/api/access/datafile/:persistentId?persistentId=" + fileId + (ingested ? "&format=original":""); } else { fileId = SystemConfig.getDataverseSiteUrlStatic() + "/file.xhtml?fileId=" + df.getId(); - fileSameAs = SystemConfig.getDataverseSiteUrlStatic() + "/api/access/datafile/" + df.getId(); + fileSameAs = SystemConfig.getDataverseSiteUrlStatic() + "/api/access/datafile/" + df.getId() + (ingested ? "?format=original":""); } aggRes.add("@id", fileId); aggRes.add(JsonLDTerm.schemaOrg("sameAs").getLabel(), fileSameAs); fileArray.add(fileId); aggRes.add("@type", JsonLDTerm.ore("AggregatedResource").getLabel()); - addIfNotNull(aggRes, JsonLDTerm.schemaOrg("fileFormat"), df.getContentType()); - addIfNotNull(aggRes, JsonLDTerm.filesize, df.getFilesize()); + addIfNotNull(aggRes, JsonLDTerm.schemaOrg("fileFormat"), mimeType); + addIfNotNull(aggRes, JsonLDTerm.filesize, fileSize); addIfNotNull(aggRes, JsonLDTerm.storageIdentifier, df.getStorageIdentifier()); - addIfNotNull(aggRes, JsonLDTerm.originalFileFormat, df.getOriginalFileFormat()); - addIfNotNull(aggRes, JsonLDTerm.originalFormatLabel, df.getOriginalFormatLabel()); + addIfNotNull(aggRes, JsonLDTerm.currentIngestedName, currentIngestedName); addIfNotNull(aggRes, JsonLDTerm.UNF, df.getUnf()); addIfNotNull(aggRes, JsonLDTerm.rootDataFileId, df.getRootDataFileId()); addIfNotNull(aggRes, JsonLDTerm.previousDataFileId, df.getPreviousDataFileId()); @@ -403,6 +356,89 @@ private JsonLDTerm getTermFor(String fieldTypeName) { } return null; } + + public static JsonValue getJsonLDForField(DatasetField field, Boolean excludeEmail, Map cvocMap, + Map localContext) { + + DatasetFieldType dfType = field.getDatasetFieldType(); + if (excludeEmail && DatasetFieldType.FieldType.EMAIL.equals(dfType.getFieldType())) { + return null; + } + + JsonLDTerm fieldName = dfType.getJsonLDTerm(); + if (fieldName.inNamespace()) { + localContext.putIfAbsent(fieldName.getNamespace().getPrefix(), fieldName.getNamespace().getUrl()); + } else { + localContext.putIfAbsent(fieldName.getLabel(), fieldName.getUrl()); + } + JsonArrayBuilder vals = Json.createArrayBuilder(); + if (!dfType.isCompound()) { + for (String val : field.getValues_nondisplay()) { + if (cvocMap.containsKey(dfType.getId())) { + try { + JsonObject cvocEntry = cvocMap.get(dfType.getId()); + if (cvocEntry.containsKey("retrieval-filtering")) { + JsonObject filtering = cvocEntry.getJsonObject("retrieval-filtering"); + JsonObject context = filtering.getJsonObject("@context"); + for (String prefix : context.keySet()) { + localContext.putIfAbsent(prefix, context.getString(prefix)); + } + vals.add(datasetFieldService.getExternalVocabularyValue(val)); + } else { + vals.add(val); + } + } catch (Exception e) { + logger.warning("Couldn't interpret value for : " + val + " : " + e.getMessage()); + logger.log(Level.FINE, ExceptionUtils.getStackTrace(e)); + vals.add(val); + } + } else { + vals.add(val); + } + } + } else { + // ToDo: Needs to be recursive (as in JsonPrinter?) + for (DatasetFieldCompoundValue dscv : field.getDatasetFieldCompoundValues()) { + // compound values are of different types + JsonObjectBuilder child = Json.createObjectBuilder(); + + for (DatasetField dsf : dscv.getChildDatasetFields()) { + DatasetFieldType dsft = dsf.getDatasetFieldType(); + if (excludeEmail && DatasetFieldType.FieldType.EMAIL.equals(dsft.getFieldType())) { + continue; + } + // which may have multiple values + if (!dsf.isEmpty()) { + // Add context entry + // ToDo - also needs to recurse here? + JsonLDTerm subFieldName = dsft.getJsonLDTerm(); + if (subFieldName.inNamespace()) { + localContext.putIfAbsent(subFieldName.getNamespace().getPrefix(), + subFieldName.getNamespace().getUrl()); + } else { + localContext.putIfAbsent(subFieldName.getLabel(), subFieldName.getUrl()); + } + + List values = dsf.getValues_nondisplay(); + if (values.size() > 1) { + JsonArrayBuilder childVals = Json.createArrayBuilder(); + + for (String val : dsf.getValues_nondisplay()) { + childVals.add(val); + } + child.add(subFieldName.getLabel(), childVals); + } else { + child.add(subFieldName.getLabel(), values.get(0)); + } + } + } + vals.add(child); + } + } + // Add metadata value to aggregation, suppress array when only one value + JsonArray valArray = vals.build(); + return (valArray.size() != 1) ? valArray : valArray.get(0); + } public static void injectSettingsService(SettingsServiceBean settingsSvc, DatasetFieldServiceBean datasetFieldSvc) { settingsService = settingsSvc; diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java index 465360f84cc..127632bf711 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java @@ -45,7 +45,7 @@ import org.apache.commons.lang3.StringUtils; import com.apicatalog.jsonld.JsonLd; -import com.apicatalog.jsonld.api.JsonLdError; +import com.apicatalog.jsonld.JsonLdError; import com.apicatalog.jsonld.document.JsonDocument; import edu.harvard.iq.dataverse.DatasetVersion.VersionState; diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonLDTerm.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonLDTerm.java index 20aeceda7de..065097709cf 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonLDTerm.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonLDTerm.java @@ -34,8 +34,11 @@ public class JsonLDTerm { public static JsonLDTerm categories = JsonLDTerm.DVCore("categories"); public static JsonLDTerm filesize = JsonLDTerm.DVCore("filesize"); public static JsonLDTerm storageIdentifier = JsonLDTerm.DVCore("storageIdentifier"); + @Deprecated public static JsonLDTerm originalFileFormat = JsonLDTerm.DVCore("originalFileFormat"); + @Deprecated public static JsonLDTerm originalFormatLabel = JsonLDTerm.DVCore("originalFormatLabel"); + public static JsonLDTerm currentIngestedName= JsonLDTerm.DVCore("currentIngestedName"); public static JsonLDTerm UNF = JsonLDTerm.DVCore("UNF"); public static JsonLDTerm rootDataFileId = JsonLDTerm.DVCore("rootDataFileId"); public static JsonLDTerm previousDataFileId = JsonLDTerm.DVCore("previousDataFileId"); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index 91f1ac2cfbc..e088122419d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -1,6 +1,23 @@ package edu.harvard.iq.dataverse.util.json; import edu.harvard.iq.dataverse.*; +import edu.harvard.iq.dataverse.AuxiliaryFile; +import edu.harvard.iq.dataverse.ControlledVocabularyValue; +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.DataFileTag; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetDistributor; +import edu.harvard.iq.dataverse.DatasetFieldType; +import edu.harvard.iq.dataverse.DatasetField; +import edu.harvard.iq.dataverse.DatasetFieldCompoundValue; +import edu.harvard.iq.dataverse.DatasetFieldValue; +import edu.harvard.iq.dataverse.DatasetLock; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseContact; +import edu.harvard.iq.dataverse.DataverseFacet; +import edu.harvard.iq.dataverse.DataverseTheme; +import edu.harvard.iq.dataverse.api.Datasets; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.groups.impl.maildomain.MailDomainGroup; import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUser; @@ -19,6 +36,7 @@ import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.globus.FileDetailsHolder; import edu.harvard.iq.dataverse.privateurl.PrivateUrl; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.DatasetFieldWalker; @@ -60,7 +78,7 @@ public class JsonPrinter { @EJB static DatasetFieldServiceBean datasetFieldService; - + public static void injectSettingsService(SettingsServiceBean ssb, DatasetFieldServiceBean dfsb) { settingsService = ssb; datasetFieldService = dfsb; @@ -314,7 +332,7 @@ public static JsonObjectBuilder json(BuiltinUser user) { } public static JsonObjectBuilder json(Dataset ds) { - return jsonObjectBuilder() + JsonObjectBuilder bld = jsonObjectBuilder() .add("id", ds.getId()) .add("identifier", ds.getIdentifier()) .add("persistentUrl", ds.getPersistentURL()) @@ -322,8 +340,19 @@ public static JsonObjectBuilder json(Dataset ds) { .add("authority", ds.getAuthority()) .add("publisher", BrandingUtil.getInstallationBrandName()) .add("publicationDate", ds.getPublicationDateFormattedYYYYMMDD()) - .add("storageIdentifier", ds.getStorageIdentifier()) - .add("metadataLanguage", ds.getMetadataLanguage()); + .add("storageIdentifier", ds.getStorageIdentifier()); + if (DvObjectContainer.isMetadataLanguageSet(ds.getMetadataLanguage())) { + bld.add("metadataLanguage", ds.getMetadataLanguage()); + } + return bld; + } + + public static JsonObjectBuilder json(FileDetailsHolder ds) { + return Json.createObjectBuilder().add(ds.getStorageID() , + Json.createObjectBuilder() + .add("id", ds.getStorageID() ) + .add("hash", ds.getHash()) + .add("mime",ds.getMime())); } public static JsonObjectBuilder json(DatasetVersion dsv) { @@ -338,7 +367,7 @@ public static JsonObjectBuilder json(DatasetVersion dsv) { .add("UNF", dsv.getUNF()).add("archiveTime", format(dsv.getArchiveTime())) .add("lastUpdateTime", format(dsv.getLastUpdateTime())).add("releaseTime", format(dsv.getReleaseTime())) .add("createTime", format(dsv.getCreateTime())); - License license = dsv.getTermsOfUseAndAccess().getLicense(); + License license = DatasetUtil.getLicense(dsv);; if (license != null) { // Standard license bld.add("license", jsonObjectBuilder() @@ -468,7 +497,7 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie blockBld.add("name", block.getName()); final JsonArrayBuilder fieldsArray = Json.createArrayBuilder(); - Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(false); + Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(false); DatasetFieldWalker.walk(fields, settingsService, cvocMap, new DatasetFieldsToJson(fieldsArray)); blockBld.add("fields", fieldsArray); @@ -684,7 +713,7 @@ public void startField(DatasetField f) { objectStack.peek().add("multiple", typ.isAllowMultiples()); objectStack.peek().add("typeClass", typeClassString(typ)); } - + @Override public void addExpandedValuesArray(DatasetField f) { // Invariant: all values are multiple. Diffrentiation between multiple and single is done at endField. @@ -705,7 +734,7 @@ public void endField(DatasetField f) { f.getDatasetFieldType().isAllowMultiples() ? expandedValues : expandedValues.get(0)); } - + valueArrStack.peek().add(jsonField); } } @@ -721,7 +750,7 @@ public void externalVocabularyValue(DatasetFieldValue dsfv, JsonObject cvocEntry } } } - + @Override public void primitiveValue(DatasetFieldValue dsfv) { if (dsfv.getValue() != null) { diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/InternalWorkflowStepSP.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/InternalWorkflowStepSP.java index ef11d306cd3..d99e0901d3c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/InternalWorkflowStepSP.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/InternalWorkflowStepSP.java @@ -25,6 +25,8 @@ public WorkflowStep getStep(String stepType, Map stepParameters) return new AuthorizedExternalStep(stepParameters); case "archiver": return new ArchivalSubmissionWorkflowStep(stepParameters); + case "ldnannounce": + return new LDNAnnounceDatasetVersionStep(stepParameters); default: throw new IllegalArgumentException("Unsupported step type: '" + stepType + "'."); } diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java new file mode 100644 index 00000000000..3478d9398f0 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -0,0 +1,279 @@ +package edu.harvard.iq.dataverse.workflow.internalspi; + +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetField; +import edu.harvard.iq.dataverse.DatasetFieldType; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.branding.BrandingUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.bagit.OREMap; +import edu.harvard.iq.dataverse.util.json.JsonLDTerm; +import edu.harvard.iq.dataverse.util.json.JsonUtil; +import edu.harvard.iq.dataverse.workflow.WorkflowContext; +import edu.harvard.iq.dataverse.workflow.step.Failure; +import edu.harvard.iq.dataverse.workflow.step.WorkflowStep; +import edu.harvard.iq.dataverse.workflow.step.WorkflowStepResult; +import static edu.harvard.iq.dataverse.workflow.step.WorkflowStepResult.OK; + +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.charset.StandardCharsets; +import java.util.Collection; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.UUID; +import java.util.logging.Level; +import java.util.logging.Logger; +import javax.json.Json; +import javax.json.JsonArray; +import javax.json.JsonArrayBuilder; +import javax.json.JsonObject; +import javax.json.JsonObjectBuilder; +import javax.json.JsonValue; + +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; + +/** + * A workflow step that generates and sends an LDN Announcement message to the + * inbox of a configured target. THe initial use case is for Dataverse to + * anounce new dataset versions to the Harvard DASH preprint repository so that + * a DASH admin can create a backlink for any dataset versions that reference a + * DASH deposit or a paper with a DOI where DASH has a preprint copy. + * + * @author qqmyers + */ + +public class LDNAnnounceDatasetVersionStep implements WorkflowStep { + private static final Logger logger = Logger.getLogger(LDNAnnounceDatasetVersionStep.class.getName()); + private static final String REQUIRED_FIELDS = ":LDNAnnounceRequiredFields"; + private static final String LDN_TARGET = ":LDNTarget"; + private static final String RELATED_PUBLICATION = "publication"; + + JsonLDTerm publicationIDType = null; + JsonLDTerm publicationIDNumber = null; + JsonLDTerm publicationURL = null; + + public LDNAnnounceDatasetVersionStep(Map paramSet) { + new HashMap<>(paramSet); + } + + @Override + public WorkflowStepResult run(WorkflowContext context) { + + JsonObject target = JsonUtil.getJsonObject((String) context.getSettings().get(LDN_TARGET)); + if (target != null) { + String inboxUrl = target.getString("inbox"); + + CloseableHttpClient client = HttpClients.createDefault(); + + // build method + + HttpPost announcement; + try { + announcement = buildAnnouncement(false, context, target); + } catch (URISyntaxException e) { + return new Failure("LDNAnnounceDatasetVersion workflow step failed: unable to parse inbox in :LDNTarget setting."); + } + if(announcement==null) { + logger.info(context.getDataset().getGlobalId().asString() + "does not have metadata required to send LDN message. Nothing sent."); + return OK; + } + // execute + try (CloseableHttpResponse response = client.execute(announcement)) { + int code = response.getStatusLine().getStatusCode(); + if (code >= 200 && code < 300) { + // HTTP OK range + return OK; + } else { + String responseBody = new String(response.getEntity().getContent().readAllBytes(), + StandardCharsets.UTF_8); + ; + return new Failure("Error communicating with " + inboxUrl + ". Server response: " + responseBody + + " (" + response + ")."); + } + + } catch (Exception ex) { + logger.log(Level.SEVERE, "Error communicating with remote server: " + ex.getMessage(), ex); + return new Failure("Error executing request: " + ex.getLocalizedMessage(), + "Cannot communicate with remote server."); + } + } + return new Failure("LDNAnnounceDatasetVersion workflow step failed: :LDNTarget setting missing or invalid."); + } + + @Override + public WorkflowStepResult resume(WorkflowContext context, Map internalData, String externalData) { + throw new UnsupportedOperationException("Not supported yet."); // This class does not need to resume. + } + + @Override + public void rollback(WorkflowContext context, Failure reason) { + throw new UnsupportedOperationException("Not supported yet."); // This class does not need to resume. + } + + HttpPost buildAnnouncement(boolean qb, WorkflowContext ctxt, JsonObject target) throws URISyntaxException { + + // First check that we have what is required + DatasetVersion dv = ctxt.getDataset().getReleasedVersion(); + List dvf = dv.getDatasetFields(); + Map fields = new HashMap(); + String[] requiredFields = ((String) ctxt.getSettings().getOrDefault(REQUIRED_FIELDS, "")).split(",\\s*"); + for (String field : requiredFields) { + fields.put(field, null); + } + Set reqFields = fields.keySet(); + for (DatasetField df : dvf) { + if(!df.isEmpty() && reqFields.contains(df.getDatasetFieldType().getName())) { + fields.put(df.getDatasetFieldType().getName(), df); + } + } + if (fields.containsValue(null)) { + logger.fine("DatasetVersion doesn't contain metadata required to trigger announcement"); + return null; + } + // We do, so construct the json-ld body and method + + Map localContext = new HashMap(); + JsonObjectBuilder coarContext = Json.createObjectBuilder(); + Map emptyCvocMap = new HashMap(); + boolean includeLocalContext = false; + for (Entry entry : fields.entrySet()) { + DatasetField field = entry.getValue(); + DatasetFieldType dft = field.getDatasetFieldType(); + String dfTypeName = entry.getKey(); + JsonValue jv = OREMap.getJsonLDForField(field, false, emptyCvocMap, localContext); + switch (dfTypeName) { + case RELATED_PUBLICATION: + JsonArrayBuilder relArrayBuilder = Json.createArrayBuilder(); + publicationIDType = null; + publicationIDNumber = null; + publicationURL = null; + Collection childTypes = dft.getChildDatasetFieldTypes(); + for (DatasetFieldType cdft : childTypes) { + switch (cdft.getName()) { + case "publicationURL": + publicationURL = cdft.getJsonLDTerm(); + break; + case "publicationIDType": + publicationIDType = cdft.getJsonLDTerm(); + break; + case "publicationIDNumber": + publicationIDNumber = cdft.getJsonLDTerm(); + break; + } + + } + + if (jv != null) { + if (jv instanceof JsonArray) { + JsonArray rels = (JsonArray) jv; + for (JsonObject jo : rels.getValuesAs(JsonObject.class)) { + String id = getBestPubId(jo); + relArrayBuilder.add(Json.createObjectBuilder().add("id", id).add("ietf:cite-as", id) + .add("type", "sorg:ScholaryArticle").build()); + } + } + + else { // JsonObject + String id = getBestPubId((JsonObject) jv); + relArrayBuilder.add(Json.createObjectBuilder().add("id", id).add("ietf:cite-as", id) + .add("type", "sorg:ScholaryArticle").build()); + } + } + coarContext.add("IsSupplementTo", relArrayBuilder); + break; + default: + if (jv != null) { + includeLocalContext = true; + coarContext.add(dft.getJsonLDTerm().getLabel(), jv); + } + + } + } + dvf.get(0).getDatasetFieldType().getName(); + JsonObjectBuilder job = Json.createObjectBuilder(); + JsonArrayBuilder context = Json.createArrayBuilder().add("https://purl.org/coar/notify") + .add("https://www.w3.org/ns/activitystreams"); + if (includeLocalContext && !localContext.isEmpty()) { + JsonObjectBuilder contextBuilder = Json.createObjectBuilder(); + for (Entry e : localContext.entrySet()) { + contextBuilder.add(e.getKey(), e.getValue()); + } + context.add(contextBuilder); + } + job.add("@context", context); + job.add("id", "urn:uuid:" + UUID.randomUUID().toString()); + job.add("actor", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) + .add("name", BrandingUtil.getInstallationBrandName()).add("type", "Service")); + job.add("context", coarContext); + Dataset d = ctxt.getDataset(); + job.add("object", + Json.createObjectBuilder().add("id", d.getLocalURL()) + .add("ietf:cite-as", d.getGlobalId().toURL().toExternalForm()) + .add("sorg:name", d.getDisplayName()).add("type", "sorg:Dataset")); + job.add("origin", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) + .add("inbox", SystemConfig.getDataverseSiteUrlStatic() + "/api/inbox").add("type", "Service")); + job.add("target", target); + job.add("type", Json.createArrayBuilder().add("Announce").add("coar-notify:ReleaseAction")); + + HttpPost annPost = new HttpPost(); + annPost.setURI(new URI(target.getString("inbox"))); + String body = JsonUtil.prettyPrint(job.build()); + logger.fine("Body: " + body); + annPost.setEntity(new StringEntity(JsonUtil.prettyPrint(body), "utf-8")); + annPost.setHeader("Content-Type", "application/ld+json"); + return annPost; + } + + private String getBestPubId(JsonObject jo) { + String id = null; + if (jo.containsKey(publicationURL.getLabel())) { + id = jo.getString(publicationURL.getLabel()); + } else if (jo.containsKey(publicationIDType.getLabel())) { + if ((jo.containsKey(publicationIDNumber.getLabel()))) { + String number = jo.getString(publicationIDNumber.getLabel()); + + switch (jo.getString(publicationIDType.getLabel())) { + case "doi": + if (number.startsWith("https://doi.org/")) { + id = number; + } else if (number.startsWith("doi:")) { + id = "https://doi.org/" + number.substring(4); + } + + break; + case "DASH-URN": + if (number.startsWith("http")) { + id = number; + } + break; + } + } + } + return id; + } + + String process(String template, Map values) { + String curValue = template; + for (Map.Entry ent : values.entrySet()) { + String val = ent.getValue(); + if (val == null) { + val = ""; + } + String varRef = "${" + ent.getKey() + "}"; + while (curValue.contains(varRef)) { + curValue = curValue.replace(varRef, val); + } + } + + return curValue; + } + +} diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index cfaeab739ef..26a20c40cb3 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -216,6 +216,7 @@ notification.publishFailedPidReg={0} in {1} could not be published due to a fail notification.workflowFailed=An external workflow run on {0} in {1} has failed. Check your email and/or view the Dataset page which may have additional details. Contact support if this continues to happen. notification.workflowSucceeded=An external workflow run on {0} in {1} has succeeded. Check your email and/or view the Dataset page which may have additional details. notification.statusUpdated=The status of dataset {0} has been updated to {1}. +notification.datasetMentioned=Announcement Received: Newly released {0} {2} {3} Dataset {4}. notification.ingestCompleted=Dataset {1} has one or more tabular files that completed the tabular ingest process and are available in archival formats. notification.ingestCompletedWithErrors=Dataset {1} has one or more tabular files that are available but are not supported for tabular ingest. @@ -231,8 +232,16 @@ notification.access.revoked.datafile=You have been removed from a role in {0}. notification.checksumfail=One or more files in your upload failed checksum validation for dataset {1}. Please re-run the upload script. If the problem persists, please contact support. notification.ingest.completed=Your Dataset {2} has one or more tabular files that completed the tabular ingest process. These files will be available for download in their original formats and other formats for enhanced archival purposes after you publish the dataset. The archival .tab files are displayed in the file table. Please see the guides for more information about ingest and support for tabular files. notification.ingest.completedwitherrors=Your Dataset {2} has one or more tabular files that have been uploaded successfully but are not supported for tabular ingest. After you publish the dataset, these files will not have additional archival features. Please see the guides for more information about ingest and support for tabular files.

Files with incomplete ingest:{5} -notification.mail.import.filesystem=Dataset {2} ({0}/dataset.xhtml?persistentId={1}) has been successfully uploaded and verified. -notification.import.filesystem=Dataset {1} has been successfully uploaded and verified. +notification.mail.import.filesystem=Globus transfer to Dataset {2} ({0}/dataset.xhtml?persistentId={1}) was successful. File(s) have been uploaded and verified. +notification.mail.globus.upload.completed=Globus transfer to Dataset {2} was successful. File(s) have been uploaded and verified.

{3}
+notification.mail.globus.download.completed=Globus transfer of file(s) from the dataset {2} was successful.

{3}
+notification.mail.globus.upload.completedWithErrors=Globus transfer to Dataset {2} is complete with errors.

{3}
+notification.mail.globus.download.completedWithErrors=Globus transfer from the dataset {2} is complete with errors.

{3}
+notification.import.filesystem=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. +notification.globus.upload.completed=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. +notification.globus.download.completed=Globus transfer from the dataset {1} was successful. +notification.globus.upload.completedWithErrors=Globus transfer to Dataset {1} is complete with errors. +notification.globus.download.completedWithErrors=Globus transfer from the dataset {1} is complete with errors. notification.import.checksum={1}, dataset had file checksums added via a batch job. removeNotification=Remove Notification @@ -265,6 +274,7 @@ notification.typeDescription.WORKFLOW_SUCCESS=External workflow run has succeede notification.typeDescription.WORKFLOW_FAILURE=External workflow run has failed notification.typeDescription.STATUSUPDATED=Status of dataset has been updated notification.typeDescription.DATASETCREATED=Dataset was created by user +notification.typeDescription.DATASETMENTIONED=Dataset was referenced in remote system groupAndRoles.manageTips=Here is where you can access and manage all the groups you belong to, and the roles you have been assigned. user.message.signup.label=Create Account @@ -780,6 +790,12 @@ contact.delegation.default_personal=Dataverse Installation Admin notification.email.info.unavailable=Unavailable notification.email.apiTokenGenerated=Hello {0} {1},\n\nAPI Token has been generated. Please keep it secure as you would do with a password. notification.email.apiTokenGenerated.subject=API Token was generated +notification.email.datasetWasMentioned=Hello {0},

The {1} has just been notified that the {2}, {4}, {5} "{8}" in this repository. +notification.email.datasetWasMentioned.subject={0}: A Dataset Relationship has been reported! +notification.email.globus.uploadCompleted.subject={0}: Files uploaded successfully via Globus and verified +notification.email.globus.downloadCompleted.subject={0}: Files downloaded successfully via Globus +notification.email.globus.uploadCompletedWithErrors.subject={0}: Uploaded files via Globus with errors +notification.email.globus.downloadCompletedWithErrors.subject={0}: Downloaded files via Globus with errors # dataverse.xhtml dataverse.name=Dataverse Name @@ -1536,6 +1552,9 @@ dataset.message.submit.remind.draft=When ready for sharing, please submit it dataset.message.publish.remind.draft.filePage=When ready for sharing, please go to the dataset page to publish it so that others can see these changes. dataset.message.submit.remind.draft.filePage=When ready for sharing, please go to the dataset page to submit it for review. dataset.message.publishSuccess=This dataset has been published. +dataset.message.publishGlobusFailure.details=Could not publish Globus data. +dataset.message.publishGlobusFailure=Error with publishing data. +dataset.message.GlobusError=Cannot go to Globus. dataset.message.only.authenticatedUsers=Only authenticated users may release Datasets. dataset.message.deleteSuccess=This dataset has been deleted. dataset.message.bulkFileUpdateSuccess=The selected files have been updated. @@ -1552,8 +1571,8 @@ dataset.message.deleteFailure=This dataset draft could not be deleted. dataset.message.deaccessionFailure=This dataset could not be deaccessioned. dataset.message.createFailure=The dataset could not be created. dataset.message.termsFailure=The dataset terms could not be updated. -dataset.message.label.fileAccess=File Access -dataset.message.publicInstall=Files are stored on a publicly accessible storage server. +dataset.message.label.fileAccess=Publicly-accessible storage +dataset.message.publicInstall=Files in this dataset may be readable outside Dataverse, restricted and embargoed access are disabled dataset.metadata.publicationDate=Publication Date dataset.metadata.publicationDate.tip=The publication date of a Dataset. dataset.metadata.citationDate=Citation Date @@ -1645,6 +1664,13 @@ file.fromHTTP=Upload with HTTP via your browser file.fromDropbox=Upload from Dropbox file.fromDropbox.tip=Select files from Dropbox. file.fromRsync=Upload with rsync + SSH via Data Capture Module (DCM) +file.fromGlobus.tip=Upload files via Globus transfer. This method is recommended for large file transfers. (Using it will cancel any other types of uploads in progress on this page.) +file.fromGlobusAfterCreate.tip=File upload via Globus transfer will be enabled after this dataset is created. +file.fromGlobus=Upload with Globus +file.finishGlobus=Globus Transfer has finished +file.downloadFromGlobus=Download through Globus +file.globus.transfer=Globus Transfer +file.globus.of=of: file.api.httpDisabled=File upload via HTTP is not available for this installation of Dataverse. file.api.alreadyHasPackageFile=File upload via HTTP disabled since this dataset already contains a package file. file.replace.original=Original File @@ -1690,6 +1716,7 @@ file.download.header=Download file.download.subset.header=Download Data Subset file.preview=Preview: file.fileName=File Name +file.sizeNotAvailable=Size not available file.type.tabularData=Tabular Data file.originalChecksumType=Original File {0} file.checksum.exists.tip=A file with this checksum already exists in the dataset. @@ -1715,6 +1742,8 @@ file.rsyncUpload.httpUploadDisabledDueToRsyncFileExisting=HTTP upload is disable file.rsyncUpload.httpUploadDisabledDueToRsyncFileExistingAndPublished=HTTP upload is disabled for this dataset because you have already uploaded files via rsync and published the dataset. file.rsyncUpload.rsyncUploadDisabledDueFileUploadedViaHttp=Upload with rsync + SSH is disabled for this dataset because you have already uploaded files via HTTP. If you would like to switch to rsync upload, then you must first remove all uploaded files from this dataset. Once this dataset is published, the chosen upload method is permanently locked in. file.rsyncUpload.rsyncUploadDisabledDueFileUploadedViaHttpAndPublished=Upload with rsync + SSH is disabled for this dataset because you have already uploaded files via HTTP and published the dataset. +file.globusUpload.inProgressMessage.summary=Globus Transfer in Progress +file.globusUpload.inProgressMessage.details=This dataset is locked while the data files are being transferred and verified. Large transfers may take significant time. You can check transfer status at https://app.globus.org/activity. file.metaData.checksum.copy=Click to copy file.metaData.dataFile.dataTab.unf=UNF file.metaData.dataFile.dataTab.variables=Variables @@ -1971,6 +2000,8 @@ file.results.btn.sort.option.type=Type file.compute.fileAccessDenied=This file is restricted and you may not compute on it because you have not been granted access. file.configure.Button=Configure +file.remotelyStored=This file is stored remotely - click for more info + file.auxfiles.download.header=Download Auxiliary Files # These types correspond to the AuxiliaryFile.Type enum. file.auxfiles.types.DP=Differentially Private Statistics @@ -2474,6 +2505,10 @@ template.delete.error=The dataset template cannot be deleted. template.update=Template data updated template.update.error=Template update failed template.makeDefault.error=The dataset template cannot be made default. +template.instructions.label=Custom Instructions: +template.instructions.label.tip=Click to Edit +template.instructions.empty.label=(None - click to add) + page.copy=Copy of #RolePermissionFragment.java @@ -2730,17 +2765,6 @@ jsonparser.error.parsing.number=Error parsing number: {0} #ConfigureFragmentBean.java configurefragmentbean.apiTokenGenerated=API Token will be generated. Please keep it secure as you would do with a password. -#FacetCategory - staticSearchFields -staticSearchFields.dvCategory=Dataverse Category -staticSearchFields.metadataSource=Metadata Source -staticSearchFields.publicationDate=Publication Year -staticSearchFields.fileTypeGroupFacet=File Type -staticSearchFields.dvObjectType=Type -staticSearchFields.fileTag=File Tag -staticSearchFields.fileAccess=Access -staticSearchFields.publicationStatus=Publication Status -staticSearchFields.subject_ss=Subject - #dataverse category - Facet Labels Researcher=Researcher Research\u0020Project=Research Project @@ -2821,3 +2845,5 @@ publishDatasetCommand.pidNotReserved=Cannot publish dataset because its persiste # APIs api.errors.invalidApiToken=Invalid API token. +api.ldninbox.citation.alert={0},

The {1} has just been notified that the {2}, {3}, cites "{6}" in this repository. +api.ldninbox.citation.subject={0}: A Dataset Citation has been reported! diff --git a/src/main/java/propertyFiles/astrophysics.properties b/src/main/java/propertyFiles/astrophysics.properties index be81ccdc883..a49b8b66510 100644 --- a/src/main/java/propertyFiles/astrophysics.properties +++ b/src/main/java/propertyFiles/astrophysics.properties @@ -1,5 +1,6 @@ metadatablock.name=astrophysics metadatablock.displayName=Astronomy and Astrophysics Metadata +metadatablock.displayFacet=Astronomy and Astrophysics datasetfieldtype.astroType.title=Type datasetfieldtype.astroFacility.title=Facility datasetfieldtype.astroInstrument.title=Instrument diff --git a/src/main/java/propertyFiles/biomedical.properties b/src/main/java/propertyFiles/biomedical.properties index 723a4ac2f40..1bffed2ee03 100644 --- a/src/main/java/propertyFiles/biomedical.properties +++ b/src/main/java/propertyFiles/biomedical.properties @@ -1,5 +1,6 @@ metadatablock.name=biomedical metadatablock.displayName=Life Sciences Metadata +metadatablock.displayFacet=Life Sciences datasetfieldtype.studyDesignType.title=Design Type datasetfieldtype.studyOtherDesignType.title=Other Design Type datasetfieldtype.studyFactorType.title=Factor Type diff --git a/src/main/java/propertyFiles/citation.properties b/src/main/java/propertyFiles/citation.properties index bdcc48b5bf1..668542c92be 100644 --- a/src/main/java/propertyFiles/citation.properties +++ b/src/main/java/propertyFiles/citation.properties @@ -1,5 +1,6 @@ metadatablock.name=citation metadatablock.displayName=Citation Metadata +metadatablock.displayFacet=Citation datasetfieldtype.title.title=Title datasetfieldtype.subtitle.title=Subtitle datasetfieldtype.alternativeTitle.title=Alternative Title @@ -265,6 +266,7 @@ controlledvocabulary.publicationIDType.purl=purl controlledvocabulary.publicationIDType.upc=upc controlledvocabulary.publicationIDType.url=url controlledvocabulary.publicationIDType.urn=urn +controlledvocabulary.publicationIDType.dash-nrs=DASH-NRS controlledvocabulary.contributorType.data_collector=Data Collector controlledvocabulary.contributorType.data_curator=Data Curator controlledvocabulary.contributorType.data_manager=Data Manager diff --git a/src/main/java/propertyFiles/customARCS.properties b/src/main/java/propertyFiles/customARCS.properties index e6665b94e64..8a19405208a 100644 --- a/src/main/java/propertyFiles/customARCS.properties +++ b/src/main/java/propertyFiles/customARCS.properties @@ -1,5 +1,6 @@ metadatablock.name=customARCS metadatablock.displayName=Alliance for Research on Corporate Sustainability Metadata +metadatablock.displayFacet=Alliance for Research on Corporate Sustainability datasetfieldtype.ARCS1.title=1) Were any of these data sets a) purchased, b) obtained through licensed databases, or c) provided by an organization under a nondisclosure or other agreement? datasetfieldtype.ARCS2.title=2) If you responded Yes to Q1, have you ensured that sharing the data does not violate terms of the agreement? If you responded No to Q1, please enter N/A here. datasetfieldtype.ARCS3.title=3) Do any of these data sets include individual-level data (either collected or pre-existing in the dataset) that might make them subject to U.S. or international human subjects considerations? diff --git a/src/main/java/propertyFiles/customCHIA.properties b/src/main/java/propertyFiles/customCHIA.properties index 0b05e388cee..0d59493da96 100644 --- a/src/main/java/propertyFiles/customCHIA.properties +++ b/src/main/java/propertyFiles/customCHIA.properties @@ -1,5 +1,6 @@ metadatablock.name=customCHIA metadatablock.displayName=CHIA Metadata +metadatablock.displayFacet=CHIA datasetfieldtype.sourceCHIA.title=Source datasetfieldtype.datesAdditionalInformationCHIA.title=Dates - Additional Information datasetfieldtype.variablesCHIA.title=Variables diff --git a/src/main/java/propertyFiles/customDigaai.properties b/src/main/java/propertyFiles/customDigaai.properties index 85d7df1f2b7..10bb8f23786 100644 --- a/src/main/java/propertyFiles/customDigaai.properties +++ b/src/main/java/propertyFiles/customDigaai.properties @@ -1,5 +1,6 @@ metadatablock.name=customDigaai metadatablock.displayName=Digaai Metadata +metadatablock.displayFacet=Digaai datasetfieldtype.titulo.title=Título datasetfieldtype.numero.title=Número datasetfieldtype.datadePublicao.title=Data de Publicação @@ -52,4 +53,4 @@ controlledvocabulary.titulo.tc_brazil=TC Brazil controlledvocabulary.titulo.texas_magazine=Texas Magazine controlledvocabulary.titulo.the_brazilian_journal=The Brazilian Journal controlledvocabulary.titulo.today_magazine=Today Magazine -controlledvocabulary.titulo.viver_magazine=Viver Magazine \ No newline at end of file +controlledvocabulary.titulo.viver_magazine=Viver Magazine diff --git a/src/main/java/propertyFiles/customGSD.properties b/src/main/java/propertyFiles/customGSD.properties index 15f118c73c4..40dc0328053 100644 --- a/src/main/java/propertyFiles/customGSD.properties +++ b/src/main/java/propertyFiles/customGSD.properties @@ -1,5 +1,6 @@ metadatablock.name=customGSD metadatablock.displayName=Graduate School of Design Metadata +metadatablock.displayFacet=Graduate School of Design datasetfieldtype.gsdStudentName.title=Student Name datasetfieldtype.gsdStudentProgram.title=Student's Program of Study datasetfieldtype.gsdCourseName.title=Course Name diff --git a/src/main/java/propertyFiles/customMRA.properties b/src/main/java/propertyFiles/customMRA.properties index 8d905d266f0..5a702b980cc 100644 --- a/src/main/java/propertyFiles/customMRA.properties +++ b/src/main/java/propertyFiles/customMRA.properties @@ -1,5 +1,6 @@ metadatablock.name=customMRA metadatablock.displayName=MRA Metadata +metadatablock.displayFacet=MRA datasetfieldtype.mraCollection.title=Murray Research Archive Collection datasetfieldtype.mraCollection.description=Browse the Murray Research Archive collection with the following terms. datasetfieldtype.mraCollection.watermark= diff --git a/src/main/java/propertyFiles/customPSI.properties b/src/main/java/propertyFiles/customPSI.properties index e72e4e50222..a88b7409c5a 100644 --- a/src/main/java/propertyFiles/customPSI.properties +++ b/src/main/java/propertyFiles/customPSI.properties @@ -1,5 +1,6 @@ metadatablock.name=customPSI metadatablock.displayName=PSI Metadata +metadatablock.displayFacet=PSI datasetfieldtype.psiBehavior.title=Behavior datasetfieldtype.psiDonor.title=Donor datasetfieldtype.psiHealthArea.title=Health Area diff --git a/src/main/java/propertyFiles/customPSRI.properties b/src/main/java/propertyFiles/customPSRI.properties index 61370bb9fd1..9e76b412bd8 100644 --- a/src/main/java/propertyFiles/customPSRI.properties +++ b/src/main/java/propertyFiles/customPSRI.properties @@ -1,5 +1,6 @@ metadatablock.name=customPSRI metadatablock.displayName=Political Science Replication Initiative Metadata +metadatablock.displayFacet=Political Science Replication Initiative datasetfieldtype.PSRI1.title=Are the original data publicly available? datasetfieldtype.PSRI2.title=Is the original code available? datasetfieldtype.PSRI3.title=Where are the original data archived (name and url)? diff --git a/src/main/java/propertyFiles/custom_hbgdki.properties b/src/main/java/propertyFiles/custom_hbgdki.properties index 087c706d014..2386b5d00a2 100644 --- a/src/main/java/propertyFiles/custom_hbgdki.properties +++ b/src/main/java/propertyFiles/custom_hbgdki.properties @@ -1,5 +1,6 @@ metadatablock.name=custom_hbgdki metadatablock.displayName=HBGDki Custom Metadata +metadatablock.displayFacet=HBGDki datasetfieldtype.hbgdkiStudyName.title=Name of Study datasetfieldtype.hbgdkiStudyRegistry.title=Study Registry datasetfieldtype.hbgdkiStudyRegistryType.title=ID Type diff --git a/src/main/java/propertyFiles/geospatial.properties b/src/main/java/propertyFiles/geospatial.properties index e47982377cb..04db8d3d05f 100644 --- a/src/main/java/propertyFiles/geospatial.properties +++ b/src/main/java/propertyFiles/geospatial.properties @@ -1,5 +1,6 @@ metadatablock.name=geospatial metadatablock.displayName=Geospatial Metadata +metadatablock.displayFacet=Geospatial datasetfieldtype.geographicCoverage.title=Geographic Coverage datasetfieldtype.country.title=Country / Nation datasetfieldtype.state.title=State / Province @@ -281,4 +282,4 @@ controlledvocabulary.country.western_sahara=Western Sahara controlledvocabulary.country.yemen=Yemen controlledvocabulary.country.zambia=Zambia controlledvocabulary.country.zimbabwe=Zimbabwe -controlledvocabulary.country.aland_islands=Åland Islands \ No newline at end of file +controlledvocabulary.country.aland_islands=Åland Islands diff --git a/src/main/java/propertyFiles/journal.properties b/src/main/java/propertyFiles/journal.properties index e17a9bd6d89..753b5895f0a 100644 --- a/src/main/java/propertyFiles/journal.properties +++ b/src/main/java/propertyFiles/journal.properties @@ -1,5 +1,6 @@ metadatablock.name=journal metadatablock.displayName=Journal Metadata +metadatablock.displayFacet=Journal datasetfieldtype.journalVolumeIssue.title=Journal datasetfieldtype.journalVolume.title=Volume datasetfieldtype.journalIssue.title=Issue diff --git a/src/main/java/propertyFiles/socialscience.properties b/src/main/java/propertyFiles/socialscience.properties index 91e73fa78b9..3698b32573f 100644 --- a/src/main/java/propertyFiles/socialscience.properties +++ b/src/main/java/propertyFiles/socialscience.properties @@ -1,5 +1,6 @@ metadatablock.name=socialscience metadatablock.displayName=Social Science and Humanities Metadata +metadatablock.displayFacet=Social Science and Humanities datasetfieldtype.unitOfAnalysis.title=Unit of Analysis datasetfieldtype.universe.title=Universe datasetfieldtype.timeMethod.title=Time Method diff --git a/src/main/java/propertyFiles/staticSearchFields.properties b/src/main/java/propertyFiles/staticSearchFields.properties new file mode 100644 index 00000000000..ab03de64f23 --- /dev/null +++ b/src/main/java/propertyFiles/staticSearchFields.properties @@ -0,0 +1,11 @@ +#FacetCategory - staticSearchFields +staticSearchFields.metadata_type_ss=Dataset Feature +staticSearchFields.dvCategory=Dataverse Category +staticSearchFields.metadataSource=Metadata Source +staticSearchFields.publicationDate=Publication Year +staticSearchFields.fileTypeGroupFacet=File Type +staticSearchFields.dvObjectType=Type +staticSearchFields.fileTag=File Tag +staticSearchFields.fileAccess=Access +staticSearchFields.publicationStatus=Publication Status +staticSearchFields.subject_ss=Subject \ No newline at end of file diff --git a/src/main/resources/META-INF/microprofile-config.properties b/src/main/resources/META-INF/microprofile-config.properties index 9e5d126d305..16298d83118 100644 --- a/src/main/resources/META-INF/microprofile-config.properties +++ b/src/main/resources/META-INF/microprofile-config.properties @@ -1 +1,10 @@ -# Entries use key=value +# GENERAL +# Will be replaced by Maven property in /target via filtering (see ) +dataverse.version=${project.version} +dataverse.build= + +# DATABASE +dataverse.db.host=localhost +dataverse.db.port=5432 +dataverse.db.user=dataverse +dataverse.db.name=dataverse diff --git a/src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSourceProvider b/src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSourceProvider index f2e23ca1b4e..796f03d7ce3 100644 --- a/src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSourceProvider +++ b/src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSourceProvider @@ -1,2 +1 @@ edu.harvard.iq.dataverse.settings.spi.AliasConfigSourceProvider -edu.harvard.iq.dataverse.settings.spi.DbSettingConfigSourceProvider diff --git a/src/main/resources/db/migration/V5.11.1.3__hdc-3b.sql b/src/main/resources/db/migration/V5.11.1.3__hdc-3b.sql new file mode 100644 index 00000000000..af8143a97d6 --- /dev/null +++ b/src/main/resources/db/migration/V5.11.1.3__hdc-3b.sql @@ -0,0 +1 @@ +ALTER TABLE usernotification ADD COLUMN IF NOT EXISTS additionalinfo VARCHAR; diff --git a/src/main/resources/db/migration/V5.11.1.4__hdc-3b2-template-instructions.sql b/src/main/resources/db/migration/V5.11.1.4__hdc-3b2-template-instructions.sql new file mode 100644 index 00000000000..df1d3068159 --- /dev/null +++ b/src/main/resources/db/migration/V5.11.1.4__hdc-3b2-template-instructions.sql @@ -0,0 +1,14 @@ +ALTER TABLE template ADD COLUMN IF NOT EXISTS instructions TEXT; + +ALTER TABLE dataset ADD COLUMN IF NOT EXISTS template_id BIGINT; + +DO $$ +BEGIN + + BEGIN + ALTER TABLE dataset ADD CONSTRAINT fx_dataset_template_id FOREIGN KEY (template_id) REFERENCES template(id); + EXCEPTION + WHEN duplicate_object THEN RAISE NOTICE 'Table constraint fk_dataset_template_id already exists'; + END; + +END $$; diff --git a/src/main/resources/db/migration/V5.11.1.5__8536-metadata-block-facet.sql b/src/main/resources/db/migration/V5.11.1.5__8536-metadata-block-facet.sql new file mode 100644 index 00000000000..47435004b6d --- /dev/null +++ b/src/main/resources/db/migration/V5.11.1.5__8536-metadata-block-facet.sql @@ -0,0 +1,11 @@ +ALTER TABLE dataverse + ADD COLUMN IF NOT EXISTS metadatablockfacetroot BOOLEAN; + +UPDATE dataverse SET metadatablockfacetroot = false; + +CREATE TABLE IF NOT EXISTS dataversemetadatablockfacet ( + id SERIAL NOT NULL, + dataverse_id BIGINT NOT NULL, + metadatablock_id BIGINT NOT NULL, + PRIMARY KEY (ID) +); diff --git a/src/main/resources/db/migration/V5.11.1.6__storageconstraint.sql b/src/main/resources/db/migration/V5.11.1.6__storageconstraint.sql new file mode 100644 index 00000000000..c2629213e98 --- /dev/null +++ b/src/main/resources/db/migration/V5.11.1.6__storageconstraint.sql @@ -0,0 +1,10 @@ +DO $$ +BEGIN + + BEGIN + ALTER TABLE dvobject ADD CONSTRAINT chk_dvobject_storageidentifier check (strpos(storageidentifier,'..') = 0); + EXCEPTION + WHEN duplicate_object THEN RAISE NOTICE 'Table constraint chk_dvobject_storageidentifier already exists'; + END; + +END $$; diff --git a/src/main/webapp/dataset-license-terms.xhtml b/src/main/webapp/dataset-license-terms.xhtml index 1a190064a42..1cbf297bf89 100644 --- a/src/main/webapp/dataset-license-terms.xhtml +++ b/src/main/webapp/dataset-license-terms.xhtml @@ -242,13 +242,13 @@ or !empty termsOfUseAndAccess.contactForAccess or !empty termsOfUseAndAccess.sizeOfCollection or !empty termsOfUseAndAccess.studyCompletion)}">
- +  
-
+
-
+
-
+
@@ -292,7 +297,7 @@
-
+
@@ -442,7 +447,7 @@ -
  • +
  • @@ -452,7 +457,7 @@
    • -
  • +
  • - + + diff --git a/src/main/webapp/file-download-button-fragment.xhtml b/src/main/webapp/file-download-button-fragment.xhtml index 53dc1bc11b1..ac1ec525b44 100644 --- a/src/main/webapp/file-download-button-fragment.xhtml +++ b/src/main/webapp/file-download-button-fragment.xhtml @@ -56,6 +56,33 @@
  • #{bundle['file.accessBtn.header.download']}
    • + + + +
  • + + + + #{bundle['file.globus.of']} #{fileMetadata.dataFile.friendlyType == 'Unknown' ? bundle['file.download.filetype.unknown'] : fileMetadata.dataFile.friendlyType} + + + + + GT: #{fileMetadata.dataFile.friendlyType == 'Unknown' ? bundle['file.download.filetype.unknown'] : fileMetadata.dataFile.friendlyType} + +
    • + +
  • +
  • #{bundle['file.accessRequested']}  - +
    + #{fileMetadata.dataFile.storageIO.remoteStoreName} + #{fileMetadata.dataFile.storageIO.remoteStoreName} +
    diff --git a/src/main/webapp/globus.xhtml b/src/main/webapp/globus.xhtml new file mode 100644 index 00000000000..f4eebd4babf --- /dev/null +++ b/src/main/webapp/globus.xhtml @@ -0,0 +1,30 @@ + + + + + + + + + + + + + + + + + + + diff --git a/src/main/webapp/manage-templates.xhtml b/src/main/webapp/manage-templates.xhtml index 1af6760534f..c9841ace8e8 100644 --- a/src/main/webapp/manage-templates.xhtml +++ b/src/main/webapp/manage-templates.xhtml @@ -178,10 +178,12 @@ + - + +
    diff --git a/src/main/webapp/metadataFragment.xhtml b/src/main/webapp/metadataFragment.xhtml index 00a3ffef96c..324fd2e0b84 100755 --- a/src/main/webapp/metadataFragment.xhtml +++ b/src/main/webapp/metadataFragment.xhtml @@ -76,6 +76,12 @@ + +
    + + +
    +     @@ -109,6 +115,13 @@

    + +
    + + +
    +     + @@ -198,7 +211,7 @@
    - +
    - +
    + #{dsf.datasetFieldType.localeTitle} - -
    + + +
    + + + + +
    +     + +
    +
    +
    +
    @@ -268,17 +294,31 @@
    +
    -
    +
    +
    #{dsf.datasetFieldType.localeTitle} - + -
    +
    + +
    + + + + + + + +
    +
    +

    diff --git a/src/main/webapp/resources/css/structure.css b/src/main/webapp/resources/css/structure.css index 86016f91eeb..c184c46cee9 100644 --- a/src/main/webapp/resources/css/structure.css +++ b/src/main/webapp/resources/css/structure.css @@ -99,6 +99,7 @@ td.col-select-width, th.col-select-width {width:36px;} .no-margin-top {margin-top:0 !important;} .no-margin-bottom {margin-bottom:0 !important;} .margin-sides {margin-left:1em; margin-right:1em;} +.margin-right {margin-right:.5em;} .padding-12 {padding: 12px;} .padding-6 {padding: 6px;} .padding-none {padding: 0 !important;} @@ -772,6 +773,17 @@ div[id$="filesTable"] thead[id$="filesTable_head"] th.ui-selection-column .ui-ch /* Non standard for webkit */ word-break: break-word; } +/*Remote Store Branding*/ +.remote-info { + width: fit-content; + margin-left: auto; + margin-right: 10px; + display: block; + padding:5px; +} +.remote-info > a { + color:white; +} /* REQUEST ACCESS DOWNLOAD OPTION LINK */ div[id$="requestPanel"].iq-dropdown-list-item {display:list-item !important;} @@ -870,6 +882,8 @@ div.panel-body.read-terms{max-height:220px; overflow-y:scroll; width:100%; backg /* UPLOAD FILES */ #dragdropMsg {padding:20px;font-size:1.3em;color:#808080;text-align:center;} .dropin-btn-status.ui-icon {background: url("https://www.dropbox.com/static/images/widgets/dbx-saver-status.png") no-repeat;} +.globus-btn.ui-icon {background: url("https://docs.globus.org/images/home/transfer.png") no-repeat;background-size:contain;display:inline-block;} + /* VERSIONS */ div[id$="versionsTable"] th.col-select-width * {display:none;} @@ -1116,3 +1130,8 @@ span.label-default { background-color: #757575 } .login-container h1 { font-size:30px; } #embargoInputs label { font-weight: normal; } + +/*Template Instructions Editing*/ +.ui-inplace .ui-inputfield { + width:60% +} diff --git a/src/main/webapp/template.xhtml b/src/main/webapp/template.xhtml index 97bce47e332..280d5ed05b9 100644 --- a/src/main/webapp/template.xhtml +++ b/src/main/webapp/template.xhtml @@ -47,13 +47,13 @@ - + @@ -63,7 +63,8 @@ - + + @@ -96,6 +97,7 @@ + @@ -105,7 +107,8 @@ - + + diff --git a/src/test/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacetTest.java b/src/test/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacetTest.java new file mode 100644 index 00000000000..7ae2d26a113 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/DataverseMetadataBlockFacetTest.java @@ -0,0 +1,45 @@ +package edu.harvard.iq.dataverse; + +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Test; + +/** + * + * @author adaybujeda + */ +public class DataverseMetadataBlockFacetTest { + + @Test + public void equals_should_be_based_on_id() { + Long sameId = MocksFactory.nextId(); + DataverseMetadataBlockFacet target1 = new DataverseMetadataBlockFacet(); + target1.setId(sameId); + target1.setDataverse(new Dataverse()); + target1.setMetadataBlock(new MetadataBlock()); + + DataverseMetadataBlockFacet target2 = new DataverseMetadataBlockFacet(); + target2.setId(sameId); + target2.setDataverse(new Dataverse()); + target2.setMetadataBlock(new MetadataBlock()); + + MatcherAssert.assertThat(target1.equals(target2), Matchers.is(true)); + + + Dataverse sameDataverse = new Dataverse(); + MetadataBlock sameMetadataBlock = new MetadataBlock(); + target1 = new DataverseMetadataBlockFacet(); + target1.setId(MocksFactory.nextId()); + target1.setDataverse(sameDataverse); + target1.setMetadataBlock(sameMetadataBlock); + + target2 = new DataverseMetadataBlockFacet(); + target2.setId(MocksFactory.nextId()); + target2.setDataverse(sameDataverse); + target2.setMetadataBlock(sameMetadataBlock); + + MatcherAssert.assertThat(target1.equals(target2), Matchers.is(false)); + } + +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/DataverseTest.java b/src/test/java/edu/harvard/iq/dataverse/DataverseTest.java new file mode 100644 index 00000000000..cb0561dd0f4 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/DataverseTest.java @@ -0,0 +1,65 @@ +package edu.harvard.iq.dataverse; + +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.Arrays; +import java.util.List; + +/** + * + * @author adaybujeda + */ +public class DataverseTest { + + private Dataverse OWNER; + private List OWNER_METADATABLOCKFACETS; + + @Before + public void beforeEachTest() { + OWNER = new Dataverse(); + OWNER.setId(MocksFactory.nextId()); + OWNER.setMetadataBlockRoot(true); + + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + OWNER_METADATABLOCKFACETS = Arrays.asList(metadataBlockFacet); + OWNER.setMetadataBlockFacets(OWNER_METADATABLOCKFACETS); + } + + @Test + public void getMetadataBlockFacets_should_return_internal_metadatablockfacets_when_metadatablockfacetroot_is_true() { + Dataverse target = new Dataverse(); + target.setId(MocksFactory.nextId()); + target.setMetadataBlockFacetRoot(true); + target.setOwner(OWNER); + + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + List internalMetadataBlockFacets = Arrays.asList(metadataBlockFacet); + target.setMetadataBlockFacets(internalMetadataBlockFacets); + List result = target.getMetadataBlockFacets(); + + MatcherAssert.assertThat(result, Matchers.is(internalMetadataBlockFacets)); + } + + @Test + public void getMetadataBlockFacets_should_return_owner_metadatablockfacets_when_metadatablockfacetroot_is_false() { + Dataverse target = new Dataverse(); + target.setId(MocksFactory.nextId()); + target.setMetadataBlockFacetRoot(false); + target.setOwner(OWNER); + + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + List internalMetadataBlockFacets = Arrays.asList(metadataBlockFacet); + target.setMetadataBlockFacets(internalMetadataBlockFacets); + List result = target.getMetadataBlockFacets(); + + MatcherAssert.assertThat(result, Matchers.is(OWNER_METADATABLOCKFACETS)); + } + +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/MetadataBlockTest.java b/src/test/java/edu/harvard/iq/dataverse/MetadataBlockTest.java new file mode 100644 index 00000000000..85aaa37bb30 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/MetadataBlockTest.java @@ -0,0 +1,79 @@ +package edu.harvard.iq.dataverse; + +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.UUID; + +public class MetadataBlockTest { + + private static final String PROPERTIES_FILE_NAME = "metadataBlockTest"; + + @Test + public void equals_should_be_based_on_id_only() { + Long id = MocksFactory.nextId(); + MetadataBlock metadataBlock1 = new MetadataBlock(); + metadataBlock1.setId(id); + metadataBlock1.setName(UUID.randomUUID().toString()); + MetadataBlock metadataBlock2 = new MetadataBlock(); + metadataBlock2.setId(id); + metadataBlock1.setName(UUID.randomUUID().toString()); + + MatcherAssert.assertThat(metadataBlock1.equals(metadataBlock2), Matchers.is(true)); + + metadataBlock1 = new MetadataBlock(); + metadataBlock1.setId(MocksFactory.nextId()); + metadataBlock1.setName("EQUAL"); + metadataBlock2 = new MetadataBlock(); + metadataBlock2.setId(MocksFactory.nextId()); + metadataBlock1.setName("EQUAL"); + + MatcherAssert.assertThat(metadataBlock1.equals(metadataBlock2), Matchers.is(false)); + } + + @Test + public void getLocaleDisplayName_should_default_value_from_displayName_when_bundle_not_found() { + MetadataBlock target = Mockito.spy(new MetadataBlock()); + target.setName(UUID.randomUUID().toString()); + target.setDisplayName(UUID.randomUUID().toString()); + + //Value when no resource file found with metadata block name + MatcherAssert.assertThat(target.getLocaleDisplayName(), Matchers.is(target.getDisplayName())); + Mockito.verify(target).getLocaleValue("metadatablock.displayName"); + } + + @Test + public void getLocaleDisplayName_should_get_value_from_properties_based_on_name() { + MetadataBlock target = Mockito.spy(new MetadataBlock()); + target.setName(PROPERTIES_FILE_NAME); + target.setDisplayName(UUID.randomUUID().toString()); + + // Values is coming from the metadataBlockTest.properties file + MatcherAssert.assertThat(target.getLocaleDisplayName(), Matchers.is("property_value_for_displayName")); + Mockito.verify(target).getLocaleValue("metadatablock.displayName"); + } + + @Test + public void getLocaleDisplayFacet_should_default_value_from_displayName_when_bundle_not_found() { + MetadataBlock target = Mockito.spy(new MetadataBlock()); + target.setName(UUID.randomUUID().toString()); + target.setDisplayName(UUID.randomUUID().toString()); + + MatcherAssert.assertThat(target.getLocaleDisplayFacet(), Matchers.is(target.getDisplayName())); + Mockito.verify(target).getLocaleValue("metadatablock.displayFacet"); + } + + @Test + public void getLocaleDisplayFacet_should_get_value_from_properties_based_on_name() { + MetadataBlock target = Mockito.spy(new MetadataBlock()); + target.setName(PROPERTIES_FILE_NAME); + target.setDisplayName(UUID.randomUUID().toString()); + + // Values is coming from the metadataBlockTest.properties file + MatcherAssert.assertThat(target.getLocaleDisplayFacet(), Matchers.is("property_value_for_displayFacet")); + Mockito.verify(target).getLocaleValue("metadatablock.displayFacet"); + } +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java index c34ee2dd4bf..cf06fd9937b 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java @@ -761,7 +761,8 @@ public void testLoadMetadataBlock_NoErrorPath() { Map>> data = JsonPath.from(body).getMap("data"); assertEquals(1, data.size()); List> addedElements = data.get("added"); - assertEquals(321, addedElements.size()); + //Note -test depends on the number of elements in the production citation block, so any changes to the # of elements there can break this test + assertEquals(322, addedElements.size()); Map statistics = new HashMap<>(); for (Map unit : addedElements) { @@ -777,7 +778,7 @@ public void testLoadMetadataBlock_NoErrorPath() { assertEquals(3, statistics.size()); assertEquals(1, (int) statistics.get("MetadataBlock")); assertEquals(78, (int) statistics.get("DatasetField")); - assertEquals(242, (int) statistics.get("Controlled Vocabulary")); + assertEquals(243, (int) statistics.get("Controlled Vocabulary")); } @Test diff --git a/src/test/java/edu/harvard/iq/dataverse/api/BagIT.java b/src/test/java/edu/harvard/iq/dataverse/api/BagIT.java new file mode 100644 index 00000000000..4ac76ac846d --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/api/BagIT.java @@ -0,0 +1,78 @@ +package edu.harvard.iq.dataverse.api; + +import com.jayway.restassured.RestAssured; +import com.jayway.restassured.response.Response; +import edu.harvard.iq.dataverse.engine.command.impl.LocalSubmitToArchiveCommand; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import static javax.ws.rs.core.Response.Status.CREATED; +import static javax.ws.rs.core.Response.Status.OK; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +public class BagIT { + + @BeforeClass + public static void setUpClass() { + + RestAssured.baseURI = UtilIT.getRestAssuredBaseUri(); + + Response setArchiverClassName = UtilIT.setSetting(SettingsServiceBean.Key.ArchiverClassName, LocalSubmitToArchiveCommand.class.getCanonicalName()); + setArchiverClassName.then().assertThat() + .statusCode(OK.getStatusCode()); + + Response setArchiverSettings = UtilIT.setSetting(SettingsServiceBean.Key.ArchiverSettings, ":BagItLocalPath, :BagGeneratorThreads"); + setArchiverSettings.then().assertThat() + .statusCode(OK.getStatusCode()); + + Response setBagItLocalPath = UtilIT.setSetting(":BagItLocalPath", "/tmp"); + setBagItLocalPath.then().assertThat() + .statusCode(OK.getStatusCode()); + + } + + @Test + public void testBagItExport() { + + Response createUser = UtilIT.createRandomUser(); + createUser.then().assertThat().statusCode(OK.getStatusCode()); + String username = UtilIT.getUsernameFromResponse(createUser); + String apiToken = UtilIT.getApiTokenFromResponse(createUser); + Response toggleSuperuser = UtilIT.makeSuperUser(username); + toggleSuperuser.then().assertThat() + .statusCode(OK.getStatusCode()); + + Response createDataverse = UtilIT.createRandomDataverse(apiToken); + createDataverse.then().assertThat().statusCode(CREATED.getStatusCode()); + String dataverseAlias = UtilIT.getAliasFromResponse(createDataverse); + Integer dataverseId = UtilIT.getDataverseIdFromResponse(createDataverse); + + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); + createDataset.prettyPrint(); + createDataset.then().assertThat() + .statusCode(CREATED.getStatusCode()); + + String datasetPid = UtilIT.getDatasetPersistentIdFromResponse(createDataset); + + Response publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAlias, apiToken); + publishDataverse.then().assertThat().statusCode(OK.getStatusCode()); + Response publishDataset = UtilIT.publishDatasetViaNativeApi(datasetPid, "major", apiToken); + publishDataset.then().assertThat().statusCode(OK.getStatusCode()); + + Response archiveDataset = UtilIT.archiveDataset(datasetPid, "1.0", apiToken); + archiveDataset.prettyPrint(); + archiveDataset.then().assertThat().statusCode(OK.getStatusCode()); + + } + + @AfterClass + public static void tearDownClass() { + + // Not checking if delete happened. Hopefully, it did. + UtilIT.deleteSetting(SettingsServiceBean.Key.ArchiverClassName); + UtilIT.deleteSetting(SettingsServiceBean.Key.ArchiverSettings); + UtilIT.deleteSetting(":BagItLocalPath"); + + } + +} diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java index 5f25d2eb68d..95c2d27dc5c 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java @@ -486,7 +486,7 @@ public void testDataFileAPIPermissions() { } @Test - public void testImportDDI() throws IOException { + public void testImportDDI() throws IOException, InterruptedException { Response createUser = UtilIT.createRandomUser(); String username = UtilIT.getUsernameFromResponse(createUser); @@ -534,6 +534,12 @@ public void testImportDDI() throws IOException { Response destroyDatasetResponsePidRel = UtilIT.destroyDataset(datasetIdIntPidRel, apiToken); assertEquals(200, destroyDatasetResponsePidRel.getStatusCode()); + // This last dataset we have just imported, let's give it a sec. to finish indexing (?) + // or whatever it is that may still be happening. (Have been seeing intermittent 500 from the next + // destroyDataset() line lately) + + Thread.sleep(1000L); + Integer datasetIdIntRelease = JsonPath.from(importDDIRelease.body().asString()).getInt("data.id"); Response destroyDatasetResponseRelease = UtilIT.destroyDataset(datasetIdIntRelease, apiToken); assertEquals(200, destroyDatasetResponseRelease.getStatusCode()); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DataversesTest.java b/src/test/java/edu/harvard/iq/dataverse/api/DataversesTest.java new file mode 100644 index 00000000000..10113110b66 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/api/DataversesTest.java @@ -0,0 +1,237 @@ +package edu.harvard.iq.dataverse.api; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.DataverseServiceBean; +import edu.harvard.iq.dataverse.EjbDataverseEngine; +import edu.harvard.iq.dataverse.GuestbookResponseServiceBean; +import edu.harvard.iq.dataverse.GuestbookServiceBean; +import edu.harvard.iq.dataverse.MetadataBlock; +import edu.harvard.iq.dataverse.MetadataBlockServiceBean; +import edu.harvard.iq.dataverse.api.datadeposit.SwordServiceBean; +import edu.harvard.iq.dataverse.api.dto.DataverseMetadataBlockFacetDTO; +import edu.harvard.iq.dataverse.api.imports.ImportServiceBean; +import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroupServiceBean; +import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; +import edu.harvard.iq.dataverse.engine.command.impl.ListMetadataBlockFacetsCommand; +import edu.harvard.iq.dataverse.engine.command.impl.UpdateMetadataBlockFacetRootCommand; +import edu.harvard.iq.dataverse.engine.command.impl.UpdateMetadataBlockFacetsCommand; +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.junit.MockitoJUnitRunner; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.core.Response; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.UUID; + +/** + * + * @author adaybujeda + */ +@RunWith(MockitoJUnitRunner.class) +public class DataversesTest { + // From AbstractApiBean class + @Mock + private EjbDataverseEngine engineSvc; + @Mock + private MetadataBlockServiceBean metadataBlockSvc; + @Mock + private PrivateUrlServiceBean privateUrlSvc; + @Mock + private HttpServletRequest httpRequest; + + // From Dataverses class + @Mock + private ExplicitGroupServiceBean explicitGroupSvc; + @Mock + private ImportServiceBean importService; + @Mock + private SettingsServiceBean settingsService; + @Mock + private GuestbookResponseServiceBean guestbookResponseService; + @Mock + private GuestbookServiceBean guestbookService; + @Mock + private DataverseServiceBean dataverseService; + @Mock + private SwordServiceBean swordService; + + @InjectMocks + private Dataverses target; + + private Dataverse VALID_DATAVERSE; + + @Before + public void beforeEachTest() { + VALID_DATAVERSE = new Dataverse(); + VALID_DATAVERSE.setId(MocksFactory.nextId()); + VALID_DATAVERSE.setAlias(UUID.randomUUID().toString()); + VALID_DATAVERSE.setMetadataBlockFacetRoot(true); + + Mockito.lenient().when(dataverseService.findByAlias(VALID_DATAVERSE.getAlias())).thenReturn(VALID_DATAVERSE); + Mockito.lenient().when(httpRequest.getHeader("X-Dataverse-key")).thenReturn(UUID.randomUUID().toString()); + Mockito.lenient().when(privateUrlSvc.getPrivateUrlUserFromToken(Mockito.anyString())).thenReturn(new PrivateUrlUser(0)); + } + + @Test + public void listMetadataBlockFacets_should_return_404_when_dataverse_is_not_found() { + String dataverseAlias = UUID.randomUUID().toString(); + Mockito.when(dataverseService.findByAlias(dataverseAlias)).thenReturn(null); + Response result = target.listMetadataBlockFacets(dataverseAlias); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(404)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void listMetadataBlockFacets_should_return_the_list_of_metadataBlockFacetDTOs() throws Exception{ + MetadataBlock metadataBlock = Mockito.mock(MetadataBlock.class); + Mockito.when(metadataBlock.getName()).thenReturn("test_metadata_block_name"); + Mockito.when(metadataBlock.getLocaleDisplayFacet()).thenReturn("test_metadata_facet_name"); + DataverseMetadataBlockFacet dataverseMetadataBlockFacet = new DataverseMetadataBlockFacet(); + dataverseMetadataBlockFacet.setDataverse(VALID_DATAVERSE); + dataverseMetadataBlockFacet.setMetadataBlock(metadataBlock); + Mockito.when(engineSvc.submit(Mockito.any(ListMetadataBlockFacetsCommand.class))).thenReturn(Arrays.asList(dataverseMetadataBlockFacet)); + + Response response = target.listMetadataBlockFacets(VALID_DATAVERSE.getAlias()); + + MatcherAssert.assertThat(response.getStatus(), Matchers.is(200)); + MatcherAssert.assertThat(response.getEntity(), Matchers.notNullValue()); + DataverseMetadataBlockFacetDTO result = (DataverseMetadataBlockFacetDTO)response.getEntity(); + MatcherAssert.assertThat(result.getDataverseId(), Matchers.is(VALID_DATAVERSE.getId())); + MatcherAssert.assertThat(result.getDataverseAlias(), Matchers.is(VALID_DATAVERSE.getAlias())); + MatcherAssert.assertThat(result.isMetadataBlockFacetRoot(), Matchers.is(VALID_DATAVERSE.isMetadataBlockFacetRoot())); + MatcherAssert.assertThat(result.getMetadataBlocks().size(), Matchers.is(1)); + MatcherAssert.assertThat(result.getMetadataBlocks().get(0).getMetadataBlockName(), Matchers.is("test_metadata_block_name")); + MatcherAssert.assertThat(result.getMetadataBlocks().get(0).getMetadataBlockFacet(), Matchers.is("test_metadata_facet_name")); + + Mockito.verify(engineSvc).submit(Mockito.any(ListMetadataBlockFacetsCommand.class)); + } + + @Test + public void listMetadataBlockFacets_should_return_empty_list_when_metadata_block_facet_is_null() throws Exception{ + Mockito.when(engineSvc.submit(Mockito.any(ListMetadataBlockFacetsCommand.class))).thenReturn(null); + + Response response = target.listMetadataBlockFacets(VALID_DATAVERSE.getAlias()); + + MatcherAssert.assertThat(response.getStatus(), Matchers.is(200)); + DataverseMetadataBlockFacetDTO result = (DataverseMetadataBlockFacetDTO)response.getEntity(); + MatcherAssert.assertThat(result.getDataverseId(), Matchers.is(VALID_DATAVERSE.getId())); + MatcherAssert.assertThat(result.getDataverseAlias(), Matchers.is(VALID_DATAVERSE.getAlias())); + MatcherAssert.assertThat(result.isMetadataBlockFacetRoot(), Matchers.is(VALID_DATAVERSE.isMetadataBlockFacetRoot())); + MatcherAssert.assertThat(result.getMetadataBlocks(), Matchers.is(Collections.emptyList())); + + Mockito.verify(engineSvc).submit(Mockito.any(ListMetadataBlockFacetsCommand.class)); + } + + @Test + public void setMetadataBlockFacets_should_return_404_when_dataverse_is_not_found() { + String dataverseAlias = UUID.randomUUID().toString(); + Mockito.when(dataverseService.findByAlias(dataverseAlias)).thenReturn(null); + Response result = target.setMetadataBlockFacets(dataverseAlias, Collections.emptyList()); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(404)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void setMetadataBlockFacets_should_return_400_when_dataverse_has_metadata_facet_root_set_to_false() { + String dataverseAlias = UUID.randomUUID().toString(); + Dataverse dataverse = Mockito.mock(Dataverse.class); + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(false); + Mockito.when(dataverseService.findByAlias(dataverseAlias)).thenReturn(dataverse); + + Response result = target.setMetadataBlockFacets(dataverseAlias, Collections.emptyList()); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(400)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void setMetadataBlockFacets_should_return_400_when_invalid_metadata_block() { + Mockito.when(metadataBlockSvc.findByName("valid_block")).thenReturn(new MetadataBlock()); + Mockito.when(metadataBlockSvc.findByName("invalid_block")).thenReturn(null); + List metadataBlocks = Arrays.asList("valid_block", "invalid_block"); + Response result = target.setMetadataBlockFacets(VALID_DATAVERSE.getAlias(), metadataBlocks); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(400)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void setMetadataBlockFacets_should_return_200_when_update_is_successful() throws Exception { + MetadataBlock validBlock = new MetadataBlock(); + Mockito.when(metadataBlockSvc.findByName("valid_block")).thenReturn(validBlock); + List metadataBlocks = Arrays.asList("valid_block"); + Response result = target.setMetadataBlockFacets(VALID_DATAVERSE.getAlias(), metadataBlocks); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(200)); + ArgumentCaptor updateCommand = ArgumentCaptor.forClass(UpdateMetadataBlockFacetsCommand.class); + Mockito.verify(engineSvc).submit(updateCommand.capture()); + + MatcherAssert.assertThat(updateCommand.getValue().getEditedDataverse(), Matchers.is(VALID_DATAVERSE)); + MatcherAssert.assertThat(updateCommand.getValue().getMetadataBlockFacets().size(), Matchers.is(1)); + MatcherAssert.assertThat(updateCommand.getValue().getMetadataBlockFacets().get(0).getDataverse(), Matchers.is(VALID_DATAVERSE)); + MatcherAssert.assertThat(updateCommand.getValue().getMetadataBlockFacets().get(0).getMetadataBlock(), Matchers.is(validBlock)); + } + + @Test + public void setMetadataBlockFacets_should_support_empty_metadatablock_list() throws Exception{ + Response result = target.setMetadataBlockFacets(VALID_DATAVERSE.getAlias(), Collections.emptyList()); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(200)); + Mockito.verify(engineSvc).submit(Mockito.any(UpdateMetadataBlockFacetsCommand.class)); + } + + @Test + public void updateMetadataBlockFacetsRoot_should_return_404_when_dataverse_is_not_found() { + String dataverseAlias = UUID.randomUUID().toString(); + Mockito.when(dataverseService.findByAlias(dataverseAlias)).thenReturn(null); + Response result = target.updateMetadataBlockFacetsRoot(dataverseAlias, "true"); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(404)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void updateMetadataBlockFacetsRoot_should_return_400_when_invalid_boolean() throws Exception{ + Response result = target.updateMetadataBlockFacetsRoot(VALID_DATAVERSE.getAlias(), "invalid"); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(400)); + Mockito.verifyNoMoreInteractions(engineSvc); + } + + @Test + public void updateMetadataBlockFacetsRoot_should_return_200_and_make_no_update_when_dataverse_is_found_and_facet_root_has_not_changed() { + // VALID_DATAVERSE.metadataBlockFacetRoot is true + Response result = target.updateMetadataBlockFacetsRoot(VALID_DATAVERSE.getAlias(), "true"); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(200)); + Mockito.verifyZeroInteractions(engineSvc); + } + + @Test + public void updateMetadataBlockFacetsRoot_should_return_200_and_execute_command_when_dataverse_is_found_and_facet_root_has_changed() throws Exception { + // VALID_DATAVERSE.metadataBlockFacetRoot is true + Response result = target.updateMetadataBlockFacetsRoot(VALID_DATAVERSE.getAlias(), "false"); + + MatcherAssert.assertThat(result.getStatus(), Matchers.is(200)); + ArgumentCaptor updateRootCommand = ArgumentCaptor.forClass(UpdateMetadataBlockFacetRootCommand.class); + Mockito.verify(engineSvc).submit(updateRootCommand.capture()); + + MatcherAssert.assertThat(updateRootCommand.getValue().getEditedDataverse(), Matchers.is(VALID_DATAVERSE)); + } +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/api/HarvestingServerIT.java b/src/test/java/edu/harvard/iq/dataverse/api/HarvestingServerIT.java index 635a8a16ec5..7579ab265fd 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/HarvestingServerIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/HarvestingServerIT.java @@ -223,7 +223,13 @@ public void testOaiFunctionality() throws InterruptedException { logger.info("identifier: " + identifier); // Let's try and create an OAI set with the dataset we have just - // created and published: + // created and published: + // - however, publish command is executed asynchronously, i.e. it may + // still be running after we received the OK from the publish API. + // So let's give it a couple of extra seconds to finish, to make sure + // the dataset is published, exported and indexed - because the OAI + // set create API requires all of the above. + Thread.sleep(3000L); String setName = identifier; String setQuery = "dsPersistentId:" + identifier; String apiPath = String.format("/api/harvest/server/oaisets/%s", setName); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/RemoteStoreIT.java b/src/test/java/edu/harvard/iq/dataverse/api/RemoteStoreIT.java new file mode 100644 index 00000000000..ae5bc8b7316 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/api/RemoteStoreIT.java @@ -0,0 +1,72 @@ +package edu.harvard.iq.dataverse.api; + +import com.jayway.restassured.RestAssured; +import com.jayway.restassured.response.Response; +import javax.json.Json; +import javax.json.JsonObjectBuilder; +import static javax.ws.rs.core.Response.Status.CREATED; +import static javax.ws.rs.core.Response.Status.OK; +import org.junit.BeforeClass; +import org.junit.Test; + +public class RemoteStoreIT { + + @BeforeClass + public static void setUp() { + RestAssured.baseURI = UtilIT.getRestAssuredBaseUri(); + } + + @Test + public void testRemoteStore() { + Response createUser = UtilIT.createRandomUser(); + createUser.then().assertThat().statusCode(OK.getStatusCode()); + String apiToken = UtilIT.getApiTokenFromResponse(createUser); + String username = UtilIT.getUsernameFromResponse(createUser); + + UtilIT.makeSuperUser(username).then().assertThat().statusCode(OK.getStatusCode()); + + Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken); + createDataverseResponse.prettyPrint(); + createDataverseResponse.then().assertThat() + .statusCode(CREATED.getStatusCode()); + + String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); + + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); + createDataset.prettyPrint(); + createDataset.then().assertThat() + .statusCode(CREATED.getStatusCode()); + + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + String datasetPid = UtilIT.getDatasetPersistentIdFromResponse(createDataset); + + /** + * Note that you must configure various JVM options for this to work: + * + * -Ddataverse.files.trsa.type=remote + * -Ddataverse.files.trsa.label=trsa + * -Ddataverse.files.trsa.base-url=https://qdr.syr.edu + * -Ddataverse.files.trsa.base-store=file + * + * In practice, most installation will also enable download-redirect + * (below) to prevent the files from being streamed through Dataverse! + * + * -Ddataverse.files.trsa.download-redirect=true + */ + JsonObjectBuilder remoteFileJson = Json.createObjectBuilder() + .add("description", "A remote image.") + .add("storageIdentifier", "trsa://themes/custom/qdr/images/CoreTrustSeal-logo-transparent.png") + .add("checksumType", "MD5") + .add("md5Hash", "509ef88afa907eaf2c17c1c8d8fde77e") + .add("label", "testlogo.png") + .add("fileName", "testlogo.png") + .add("mimeType", "image/png"); + + Response addRemoteFile = UtilIT.addRemoteFile(datasetId.toString(), remoteFileJson.build().toString(), apiToken); + addRemoteFile.prettyPrint(); + addRemoteFile.then().assertThat() + .statusCode(OK.getStatusCode()); + System.out.println("done!"); + } + +} diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index c791ce72f41..716cd1e8d84 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -680,6 +680,16 @@ static Response uploadFileViaNative(String datasetId, String pathToFile, String return requestSpecification.post("/api/datasets/" + datasetId + "/add"); } + static Response addRemoteFile(String datasetId, String jsonAsString, String apiToken) { + RequestSpecification requestSpecification = given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .multiPart("datasetId", datasetId); + if (jsonAsString != null) { + requestSpecification.multiPart("jsonData", jsonAsString); + } + return requestSpecification.post("/api/datasets/" + datasetId + "/add"); + } + static Response uploadAuxFile(Long fileId, String pathToFile, String formatTag, String formatVersion, String mimeType, boolean isPublic, String type, String apiToken) { String nullOrigin = null; return uploadAuxFile(fileId, pathToFile, formatTag, formatVersion, mimeType, isPublic, type, nullOrigin, apiToken); @@ -1869,6 +1879,14 @@ static Response deleteSetting(SettingsServiceBean.Key settingKey) { return response; } + /** + * @param settingKey Include the colon like :BagItLocalPath + */ + static Response deleteSetting(String settingKey) { + Response response = given().when().delete("/api/admin/settings/" + settingKey); + return response; + } + static Response getSetting(SettingsServiceBean.Key settingKey) { Response response = given().when().get("/api/admin/settings/" + settingKey); return response; @@ -1879,6 +1897,14 @@ static Response setSetting(SettingsServiceBean.Key settingKey, String value) { return response; } + /** + * @param settingKey Include the colon like :BagItLocalPath + */ + public static Response setSetting(String settingKey, String value) { + Response response = given().body(value).when().put("/api/admin/settings/" + settingKey); + return response; + } + static Response getRoleAssignmentsOnDataverse(String dataverseAliasOrId, String apiToken) { String url = "/api/dataverses/" + dataverseAliasOrId + "/assignments"; return given() @@ -2917,6 +2943,20 @@ static Response getDatasetVersionArchivalStatus(Integer datasetId, String versio .get("/api/datasets/" + datasetId + "/" + version + "/archivalStatus"); return response; } + + static Response archiveDataset(String idOrPersistentIdOfDataset, String version, String apiToken) { + String idInPath = idOrPersistentIdOfDataset; + String optionalQueryParam = ""; + if (!NumberUtils.isNumber(idOrPersistentIdOfDataset)) { + idInPath = ":persistentId"; + optionalQueryParam = "?persistentId=" + idOrPersistentIdOfDataset; + } + Response response = given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .post("/api/admin/submitDatasetVersionToArchive/" + idInPath + "/" + version + optionalQueryParam); + return response; + } + static Response setDatasetVersionArchivalStatus(Integer datasetId, String version, String apiToken, String status, String message) { Response response = given() .header(API_TOKEN_HTTP_HEADER, apiToken).contentType("application/json; charset=utf-8").body("{\"status\":\"" + status + "\", \"message\":\"" + message + "\"}") diff --git a/src/test/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIOTest.java b/src/test/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIOTest.java index 286fe8a6595..95621dd8750 100644 --- a/src/test/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIOTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIOTest.java @@ -9,6 +9,8 @@ import edu.harvard.iq.dataverse.Dataset; import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.mocks.MocksFactory; +import edu.harvard.iq.dataverse.util.FileUtil; + import java.io.BufferedReader; import java.io.BufferedWriter; import java.io.ByteArrayInputStream; @@ -24,6 +26,9 @@ import org.junit.After; import org.junit.Test; import static org.junit.Assert.*; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + import org.junit.Before; /** @@ -285,4 +290,31 @@ public void testGetAuxFileAsInputStream() throws IOException { } assertEquals("This is a test string\n", sb.toString()); } + + @Test + public void testFileIdentifierFormats() throws IOException { + System.setProperty("dataverse.files.filetest.type", "file"); + System.setProperty("dataverse.files.filetest.label", "Ftest"); + System.setProperty("dataverse.files.filetest.directory", "/tmp/mydata"); + + FileUtil.generateStorageIdentifier(); + assertTrue(DataAccess.isValidDirectStorageIdentifier("filetest://" + FileUtil.generateStorageIdentifier())); + //The tests here don't use a valid identifier string + assertFalse(DataAccess.isValidDirectStorageIdentifier(dataFile.getStorageIdentifier())); + //bad store id + String defaultType = System.getProperty("dataverse.files.file.type"); + //Assure file isn't a defined store before test and reset afterwards if it was + System.clearProperty("dataverse.files.file.type"); + assertFalse(DataAccess.isValidDirectStorageIdentifier("file://" + FileUtil.generateStorageIdentifier())); + if(defaultType!=null) { + System.out.println("dataverse.files.file.type reset to " + defaultType); + System.setProperty("dataverse.files.file.type", defaultType); + } + //breakout + assertFalse(DataAccess.isValidDirectStorageIdentifier("filetest://../" + FileUtil.generateStorageIdentifier())); + + System.clearProperty("dataverse.files.filetest.type"); + System.clearProperty("dataverse.files.filetest.label"); + System.clearProperty("dataverse.files.filetest.directory"); + } } diff --git a/src/test/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIOTest.java b/src/test/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIOTest.java new file mode 100644 index 00000000000..f66b3306dda --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIOTest.java @@ -0,0 +1,124 @@ +/* + * Copyright 2018 Forschungszentrum Jülich GmbH + * SPDX-License-Identifier: Apache 2.0 + */ +package edu.harvard.iq.dataverse.dataaccess; + +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.GlobalId; +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import edu.harvard.iq.dataverse.util.UrlSignerUtil; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import static org.junit.jupiter.api.Assertions.*; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; +import java.io.IOException; +import java.nio.file.Paths; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.STRICT_STUBS) +public class RemoteOverlayAccessIOTest { + + @Mock + + private Dataset dataset; + private DataFile datafile; + private DataFile badDatafile; + private String baseStoreId="182ad2bda2f-c3508e719076"; + private String logoPath = "images/dataverse_project_logo.svg"; + private String pid = "10.5072/F2/ABCDEF"; + + @BeforeEach + public void setUp() { + System.setProperty("dataverse.files.test.type", "remote"); + System.setProperty("dataverse.files.test.label", "testOverlay"); + System.setProperty("dataverse.files.test.base-url", "https://demo.dataverse.org/resources"); + System.setProperty("dataverse.files.test.base-store", "file"); + System.setProperty("dataverse.files.test.download-redirect", "true"); + System.setProperty("dataverse.files.test.remote-store-name", "DemoDataCorp"); + System.setProperty("dataverse.files.test.secret-key", "12345"); // Real keys should be much longer, more random + System.setProperty("dataverse.files.file.type", "file"); + System.setProperty("dataverse.files.file.label", "default"); + datafile = MocksFactory.makeDataFile(); + dataset = MocksFactory.makeDataset(); + dataset.setGlobalId(GlobalId.parse("doi:" + pid).get()); + datafile.setOwner(dataset); + datafile.setStorageIdentifier("test://" + baseStoreId + "//" + logoPath); + + badDatafile = MocksFactory.makeDataFile(); + badDatafile.setOwner(dataset); + badDatafile.setStorageIdentifier("test://" + baseStoreId + "//../.." + logoPath); + } + + @AfterEach + public void tearDown() { + System.clearProperty("dataverse.files.test.type"); + System.clearProperty("dataverse.files.test.label"); + System.clearProperty("dataverse.files.test.base-url"); + System.clearProperty("dataverse.files.test.base-store"); + System.clearProperty("dataverse.files.test.download-redirect"); + System.clearProperty("dataverse.files.test.label"); + System.clearProperty("dataverse.files.test.remote-store-name"); + System.clearProperty("dataverse.files.test.secret-key"); + System.clearProperty("dataverse.files.file.type"); + System.clearProperty("dataverse.files.file.label"); + } + + @Test + void testRemoteOverlayFiles() throws IOException { + // We can read the storageIdentifier and get the driver + assertTrue(datafile.getStorageIdentifier() + .startsWith(DataAccess.getStorageDriverFromIdentifier(datafile.getStorageIdentifier()))); + // We can get the driver type from it's ID + assertTrue(DataAccess.getDriverType("test").equals(System.getProperty("dataverse.files.test.type"))); + // When we get a StorageIO for the file, it is the right type + StorageIO storageIO = DataAccess.getStorageIO(datafile); + assertTrue(storageIO instanceof RemoteOverlayAccessIO); + // When we use it, we can get properties like the remote store name + RemoteOverlayAccessIO remoteIO = (RemoteOverlayAccessIO) storageIO; + assertTrue(remoteIO.getRemoteStoreName().equals(System.getProperty("dataverse.files.test.remote-store-name"))); + // And can get a temporary download URL for the main file + String signedURL = remoteIO.generateTemporaryDownloadUrl(null, null, null); + // And the URL starts with the right stuff + assertTrue(signedURL.startsWith(System.getProperty("dataverse.files.test.base-url") + "/" + logoPath)); + // And the signature is valid + assertTrue( + UrlSignerUtil.isValidUrl(signedURL, null, null, System.getProperty("dataverse.files.test.secret-key"))); + // And we get an unsigned URL with the right stuff with no key + System.clearProperty("dataverse.files.test.secret-key"); + String unsignedURL = remoteIO.generateTemporaryDownloadUrl(null, null, null); + assertTrue(unsignedURL.equals(System.getProperty("dataverse.files.test.base-url") + "/" + logoPath)); + // Once we've opened, we can get the file size (only works if the HEAD call to + // the file URL works + remoteIO.open(DataAccessOption.READ_ACCESS); + assertTrue(remoteIO.getSize() > 0); + // If we ask for the path for an aux file, it is correct + System.out.println(Paths + .get(System.getProperty("dataverse.files.file.directory", "/tmp/files"), pid, baseStoreId + ".auxobject").toString()); + System.out.println(remoteIO.getAuxObjectAsPath("auxobject").toString()); + assertTrue(Paths + .get(System.getProperty("dataverse.files.file.directory", "/tmp/files"), pid, baseStoreId + ".auxobject") + .equals(remoteIO.getAuxObjectAsPath("auxobject"))); + IOException thrown = assertThrows(IOException.class, () -> DataAccess.getStorageIO(badDatafile), + "Expected getStorageIO() to throw, but it didn't"); + // 'test' is the driverId in the IOException messages + assertTrue(thrown.getMessage().contains("test")); + + } + + @Test + void testRemoteOverlayIdentifierFormats() throws IOException { + + assertTrue(DataAccess.isValidDirectStorageIdentifier(datafile.getStorageIdentifier())); + assertFalse(DataAccess.isValidDirectStorageIdentifier(badDatafile.getStorageIdentifier())); + assertFalse(DataAccess.isValidDirectStorageIdentifier(datafile.getStorageIdentifier().replace("test", "bad"))); + } + +} diff --git a/src/test/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIOTest.java b/src/test/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIOTest.java index 1f118a0ea68..0e8e8500a1e 100644 --- a/src/test/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIOTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIOTest.java @@ -9,6 +9,9 @@ import edu.harvard.iq.dataverse.Dataset; import edu.harvard.iq.dataverse.api.UtilIT; import edu.harvard.iq.dataverse.mocks.MocksFactory; +import edu.harvard.iq.dataverse.util.FileUtil; + +import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @@ -30,7 +33,7 @@ public class S3AccessIOTest { @Mock private AmazonS3 s3client; - private S3AccessIO dataSetAccess; + private StorageIO dataSetAccess; private S3AccessIO dataFileAccess; private Dataset dataSet; private DataFile dataFile; @@ -42,11 +45,21 @@ public void setup() throws IOException { dataSet = MocksFactory.makeDataset(); dataFile.setOwner(dataSet); dataFileId = UtilIT.getRandomIdentifier(); - dataFile.setStorageIdentifier("s3://bucket:"+dataFileId); - dataSetAccess = new S3AccessIO<>(dataSet, null, s3client, "s3"); - dataFileAccess = new S3AccessIO<>(dataFile, null, s3client, "s3"); + System.setProperty("dataverse.files.s3test.type", "s3"); + System.setProperty("dataverse.files.s3test.label", "S3test"); + System.setProperty("dataverse.files.s3test.bucket-name", "thebucket"); + + dataFile.setStorageIdentifier("s3test://thebucket:"+dataFileId); + dataSetAccess = new S3AccessIO<>(dataSet, null, s3client, "s3test"); + dataFileAccess = new S3AccessIO<>(dataFile, null, s3client, "s3test"); } + @AfterEach + public void tearDown() { + System.clearProperty("dataverse.files.s3test.type"); + System.clearProperty("dataverse.files.s3test.label"); + System.clearProperty("dataverse.files.s3test.bucket-name"); + } /* createTempFile getStorageLocation @@ -99,7 +112,7 @@ void keyNullstorageIdInvalid_getMainFileKey() throws IOException { @Test void default_getUrlExpirationMinutes() { // given - System.clearProperty("dataverse.files.s3.url-expiration-minutes"); + System.clearProperty("dataverse.files.s3test.url-expiration-minutes"); // when & then assertEquals(60, dataFileAccess.getUrlExpirationMinutes()); } @@ -107,7 +120,7 @@ void default_getUrlExpirationMinutes() { @Test void validSetting_getUrlExpirationMinutes() { // given - System.setProperty("dataverse.files.s3.url-expiration-minutes", "120"); + System.setProperty("dataverse.files.s3test.url-expiration-minutes", "120"); // when & then assertEquals(120, dataFileAccess.getUrlExpirationMinutes()); } @@ -115,9 +128,20 @@ void validSetting_getUrlExpirationMinutes() { @Test void invalidSetting_getUrlExpirationMinutes() { // given - System.setProperty("dataverse.files.s3.url-expiration-minutes", "NaN"); + System.setProperty("dataverse.files.s3test.url-expiration-minutes", "NaN"); // when & then assertEquals(60, dataFileAccess.getUrlExpirationMinutes()); } + @Test + void testS3IdentifierFormats() throws IOException { + assertTrue(DataAccess.isValidDirectStorageIdentifier("s3test://thebucket:" + FileUtil.generateStorageIdentifier())); + //The tests here don't use a valid identifier string + assertFalse(DataAccess.isValidDirectStorageIdentifier(dataFile.getStorageIdentifier())); + //bad store id + assertFalse(DataAccess.isValidDirectStorageIdentifier("s3://thebucket:" + FileUtil.generateStorageIdentifier())); + //bad bucket + assertFalse(DataAccess.isValidDirectStorageIdentifier("s3test://bucket:" + FileUtil.generateStorageIdentifier())); + } + } diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlocksCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlocksCommandTest.java new file mode 100644 index 00000000000..520c91f47ff --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/ListMetadataBlocksCommandTest.java @@ -0,0 +1,66 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.List; + +/** + * + * @author adaybujeda + */ +public class ListMetadataBlocksCommandTest { + + private DataverseRequest dataverseRequest; + private Dataverse dataverse; + private DataverseMetadataBlockFacet metadataBlockFacet; + + @Before + public void beforeEachTest() { + dataverseRequest = Mockito.mock(DataverseRequest.class); + dataverse = Mockito.mock(Dataverse.class); + metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + Mockito.when(dataverse.getMetadataBlockFacets()).thenReturn(Arrays.asList(metadataBlockFacet)); + } + + @Test + public void execute_should_return_dataverse_metadata_block_facets() throws CommandException { + ListMetadataBlockFacetsCommand target = new ListMetadataBlockFacetsCommand(dataverseRequest, dataverse); + + List result = target.execute(Mockito.mock(CommandContext.class)); + + MatcherAssert.assertThat(result.size(), Matchers.is(1)); + MatcherAssert.assertThat(result.get(0), Matchers.is(metadataBlockFacet)); + } + + + @Test + public void getRequiredPermissions_should_return_empty_for_all_when_dataverse_is_released() { + Mockito.when(dataverse.isReleased()).thenReturn(true); + ListMetadataBlockFacetsCommand target = new ListMetadataBlockFacetsCommand(dataverseRequest, dataverse); + + MatcherAssert.assertThat(target.getRequiredPermissions().get(""), Matchers.emptyCollectionOf(Permission.class)); + } + + @Test + public void getRequiredPermissions_should_return_ViewUnpublishedDataverse_for_all_when_dataverse_is_not_released() { + Mockito.when(dataverse.isReleased()).thenReturn(false); + ListMetadataBlockFacetsCommand target = new ListMetadataBlockFacetsCommand(dataverseRequest, dataverse); + + MatcherAssert.assertThat(target.getRequiredPermissions().get("").size(), Matchers.is(1)); + MatcherAssert.assertThat(target.getRequiredPermissions().get(""), Matchers.hasItems(Permission.ViewUnpublishedDataverse)); + } + +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommandTest.java new file mode 100644 index 00000000000..711e7881af5 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetRootCommandTest.java @@ -0,0 +1,111 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.MetadataBlock; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * + * @author adaybujeda + */ +public class UpdateMetadataBlockFacetRootCommandTest { + + private DataverseRequest dataverseRequest; + private Dataverse dataverse; + + @Before + public void beforeEachTest() { + dataverseRequest = Mockito.mock(DataverseRequest.class); + dataverse = Mockito.mock(Dataverse.class); + } + + @Test + public void should_not_update_dataverse_when_root_value_does_not_change() throws CommandException { + boolean metadataBlockFacetRoot = true; + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(metadataBlockFacetRoot); + UpdateMetadataBlockFacetRootCommand target = new UpdateMetadataBlockFacetRootCommand(dataverseRequest, dataverse, metadataBlockFacetRoot); + + CommandContext context = Mockito.mock(CommandContext.class, Mockito.RETURNS_DEEP_STUBS); + target.execute(context); + + Mockito.verify(dataverse).isMetadataBlockFacetRoot(); + Mockito.verifyNoMoreInteractions(dataverse); + Mockito.verifyZeroInteractions(context.dataverses()); + } + + @Test + public void should_set_metadataBlockFacetRoot_and_update_metadata_block_facets_to_empty_list_when_root_value_changes_to_false() throws CommandException { + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(true); + UpdateMetadataBlockFacetRootCommand target = new UpdateMetadataBlockFacetRootCommand(dataverseRequest, dataverse, false); + + CommandContext context = Mockito.mock(CommandContext.class, Mockito.RETURNS_DEEP_STUBS); + target.execute(context); + + Mockito.verify(dataverse).isMetadataBlockFacetRoot(); + Mockito.verify(dataverse).setMetadataBlockFacetRoot(false); + Mockito.verify(dataverse).setMetadataBlockFacets(Collections.emptyList()); + Mockito.verifyNoMoreInteractions(dataverse); + Mockito.verify(context.dataverses()).save(dataverse); + } + + @Test + public void should_set_metadataBlockFacetRoot_and_update_metadata_block_facets_to_parent_list_when_root_value_changes_to_true() throws CommandException { + Dataverse parentDataverse = Mockito.mock(Dataverse.class); + DataverseMetadataBlockFacet blockFacet1 = new DataverseMetadataBlockFacet(); + MetadataBlock block1 = Mockito.mock(MetadataBlock.class); + blockFacet1.setDataverse(parentDataverse); + blockFacet1.setMetadataBlock(block1); + DataverseMetadataBlockFacet blockFacet2 = new DataverseMetadataBlockFacet(); + MetadataBlock block2 = Mockito.mock(MetadataBlock.class); + blockFacet2.setDataverse(parentDataverse); + blockFacet2.setMetadataBlock(block2); + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(false); + Mockito.when(dataverse.getMetadataBlockFacets()).thenReturn(Arrays.asList(blockFacet1, blockFacet2)); + UpdateMetadataBlockFacetRootCommand target = new UpdateMetadataBlockFacetRootCommand(dataverseRequest, dataverse, true); + + CommandContext context = Mockito.mock(CommandContext.class, Mockito.RETURNS_DEEP_STUBS); + target.execute(context); + + Mockito.verify(dataverse).isMetadataBlockFacetRoot(); + Mockito.verify(dataverse).getMetadataBlockFacets(); + Mockito.verify(dataverse).setMetadataBlockFacetRoot(true); + + ArgumentCaptor> createdBlockFacets = ArgumentCaptor.forClass(List.class); + Mockito.verify(dataverse).setMetadataBlockFacets(createdBlockFacets.capture()); + MatcherAssert.assertThat(createdBlockFacets.getValue().size(), Matchers.is(2)); + MatcherAssert.assertThat(createdBlockFacets.getValue().get(0).getDataverse(), Matchers.is(dataverse)); + MatcherAssert.assertThat(createdBlockFacets.getValue().get(0).getMetadataBlock(), Matchers.is(block1)); + MatcherAssert.assertThat(createdBlockFacets.getValue().get(1).getDataverse(), Matchers.is(dataverse)); + MatcherAssert.assertThat(createdBlockFacets.getValue().get(1).getMetadataBlock(), Matchers.is(block2)); + Mockito.verifyNoMoreInteractions(dataverse); + Mockito.verify(context.dataverses()).save(dataverse); + } + + @Test + public void getEditedDataverse_should_return_set_dataverse() { + UpdateMetadataBlockFacetRootCommand target = new UpdateMetadataBlockFacetRootCommand(dataverseRequest, dataverse, true); + + MatcherAssert.assertThat(target.getEditedDataverse(), Matchers.is(dataverse)); + } + + @Test + public void getMetadataBlockFacetRoot_should_return_set_metadata_block_facet_root() { + UpdateMetadataBlockFacetRootCommand target = new UpdateMetadataBlockFacetRootCommand(dataverseRequest, dataverse, true); + + MatcherAssert.assertThat(target.getMetadataBlockFacetRoot(), Matchers.is(true)); + } + +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommandTest.java new file mode 100644 index 00000000000..2d64de80f3d --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateMetadataBlockFacetsCommandTest.java @@ -0,0 +1,78 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; +import edu.harvard.iq.dataverse.mocks.MocksFactory; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * + * @author adaybujeda + */ +public class UpdateMetadataBlockFacetsCommandTest { + + private DataverseRequest dataverseRequest; + private Dataverse dataverse; + + @Before + public void beforeEachTest() { + dataverseRequest = Mockito.mock(DataverseRequest.class); + dataverse = Mockito.mock(Dataverse.class); + } + + @Test(expected = IllegalCommandException.class) + public void should_throw_IllegalCommandException_when_dataverse_is_not_metadata_facet_root() throws CommandException { + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(false); + + UpdateMetadataBlockFacetsCommand target = new UpdateMetadataBlockFacetsCommand(dataverseRequest, dataverse, Collections.emptyList()); + + CommandContext context = Mockito.mock(CommandContext.class, Mockito.RETURNS_DEEP_STUBS); + target.execute(context); + } + + @Test + public void should_update_facets() throws CommandException { + Mockito.when(dataverse.isMetadataBlockFacetRoot()).thenReturn(true); + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + List metadataBlockFacets = Arrays.asList(metadataBlockFacet); + + UpdateMetadataBlockFacetsCommand target = new UpdateMetadataBlockFacetsCommand(dataverseRequest, dataverse, metadataBlockFacets); + + CommandContext context = Mockito.mock(CommandContext.class, Mockito.RETURNS_DEEP_STUBS); + target.execute(context); + + Mockito.verify(dataverse).setMetadataBlockFacets(metadataBlockFacets); + Mockito.verify(context.dataverses()).save(dataverse); + } + + @Test + public void getEditedDataverse_should_return_set_dataverse() { + UpdateMetadataBlockFacetsCommand target = new UpdateMetadataBlockFacetsCommand(dataverseRequest, dataverse, Collections.emptyList()); + + MatcherAssert.assertThat(target.getEditedDataverse(), Matchers.is(dataverse)); + } + + @Test + public void getMetadataBlockFacets_should_return_set_metadata_block_facets() { + DataverseMetadataBlockFacet metadataBlockFacet = new DataverseMetadataBlockFacet(); + metadataBlockFacet.setId(MocksFactory.nextId()); + List metadataBlockFacets = Arrays.asList(metadataBlockFacet); + UpdateMetadataBlockFacetsCommand target = new UpdateMetadataBlockFacetsCommand(dataverseRequest, dataverse, metadataBlockFacets); + + MatcherAssert.assertThat(target.getMetadataBlockFacets(), Matchers.is(metadataBlockFacets)); + } + +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/search/SearchIncludeFragmentTest.java b/src/test/java/edu/harvard/iq/dataverse/search/SearchIncludeFragmentTest.java new file mode 100644 index 00000000000..f94da336ca3 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/search/SearchIncludeFragmentTest.java @@ -0,0 +1,126 @@ +package edu.harvard.iq.dataverse.search; + +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; +import edu.harvard.iq.dataverse.MetadataBlock; +import org.hamcrest.MatcherAssert; +import org.hamcrest.Matchers; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; + +public class SearchIncludeFragmentTest { + + + @Test + public void getFriendlyNamesFromFilterQuery_should_return_null_when_null_or_empty_filter_query() { + SearchIncludeFragment target = new SearchIncludeFragment(); + + MatcherAssert.assertThat(target.getFriendlyNamesFromFilterQuery(null), Matchers.nullValue()); + MatcherAssert.assertThat(target.getFriendlyNamesFromFilterQuery(""), Matchers.nullValue()); + } + + @Test + public void getFriendlyNamesFromFilterQuery_should_return_null_when_filter_query_does_not_have_key_value_pair_separated_by_colon() { + SearchIncludeFragment target = new SearchIncludeFragment(); + + MatcherAssert.assertThat(target.getFriendlyNamesFromFilterQuery("key_value_no_colon"), Matchers.nullValue()); + } + + @Test + public void getFriendlyNamesFromFilterQuery_key_should_return_key_when_filter_query_key_does_not_match_any_friendly_names() { + SearchIncludeFragment target = new SearchIncludeFragment(); + + List result = target.getFriendlyNamesFromFilterQuery("key:value"); + MatcherAssert.assertThat(result.get(0), Matchers.is("key")); + } + + @Test + public void getFriendlyNamesFromFilterQuery_key_should_return_friendly_name_when_filter_query_key_does_match_friendly_name() { + SearchIncludeFragment target = new SearchIncludeFragment(); + target.datasetfieldFriendlyNamesBySolrField = Map.of("key", "KeyFriendlyName"); + + List result = target.getFriendlyNamesFromFilterQuery("key:value"); + MatcherAssert.assertThat(result.get(0), Matchers.is("KeyFriendlyName")); + } + + @Test + public void getFriendlyNamesFromFilterQuery_key_should_return_static_friendly_name_when_filter_query_key_does_not_match_friendly_name_but_matches_static_names() { + SearchIncludeFragment target = new SearchIncludeFragment(); + target.staticSolrFieldFriendlyNamesBySolrField = Map.of("key", "staticKeyFriendlyName"); + + List result = target.getFriendlyNamesFromFilterQuery("key:value"); + MatcherAssert.assertThat(result.get(0), Matchers.is("staticKeyFriendlyName")); + } + + @Test + public void getFriendlyNamesFromFilterQuery_value_should_return_metadata_block_facet_label_when_key_is_metadata_types_and_value_matches_metadata_block_name() { + SearchIncludeFragment target = new SearchIncludeFragment(); + Dataverse dataverse = Mockito.mock(Dataverse.class); + MetadataBlock block = Mockito.mock(MetadataBlock.class); + Mockito.when(block.getName()).thenReturn("metadata_block_name"); + Mockito.when(block.getLocaleDisplayFacet()).thenReturn("display_facet"); + DataverseMetadataBlockFacet blockFacet = new DataverseMetadataBlockFacet(); + blockFacet.setMetadataBlock(block); + Mockito.when(dataverse.getMetadataBlockFacets()).thenReturn(Arrays.asList(blockFacet)); + target.setDataverse(dataverse); + + List result = target.getFriendlyNamesFromFilterQuery(String.format("%s:\"metadata_block_name\"", SearchFields.METADATA_TYPES)); + MatcherAssert.assertThat(result.get(1), Matchers.is("display_facet")); + + Mockito.verify(dataverse, Mockito.times(2)).getMetadataBlockFacets(); + Mockito.verify(block).getName(); + Mockito.verify(block).getLocaleDisplayFacet(); + } + + @Test + public void getFriendlyNamesFromFilterQuery_value_should_return_value_when_key_is_metadata_types_and_value_does_not_matches_metadata_block_name() { + SearchIncludeFragment target = new SearchIncludeFragment(); + Dataverse dataverse = Mockito.mock(Dataverse.class); + MetadataBlock block = Mockito.mock(MetadataBlock.class); + Mockito.when(block.getName()).thenReturn("metadata_block_name"); + Mockito.when(block.getLocaleDisplayFacet()).thenReturn("display_facet"); + DataverseMetadataBlockFacet blockFacet = new DataverseMetadataBlockFacet(); + blockFacet.setMetadataBlock(block); + Mockito.when(dataverse.getMetadataBlockFacets()).thenReturn(Arrays.asList(blockFacet)); + target.setDataverse(dataverse); + + List result = target.getFriendlyNamesFromFilterQuery(String.format("%s:\"no_match_block_name\"", SearchFields.METADATA_TYPES)); + MatcherAssert.assertThat(result.get(1), Matchers.is("no_match_block_name")); + + Mockito.verify(dataverse, Mockito.times(2)).getMetadataBlockFacets(); + Mockito.verify(block).getName(); + } + + @Test + public void getFriendlyNamesFromFilterQuery_value_should_return_value_when_key_is_metadata_types_and_dataverse_is_null() { + SearchIncludeFragment target = new SearchIncludeFragment(); + + List result = target.getFriendlyNamesFromFilterQuery(String.format("%s:\"no_dataverse\"", SearchFields.METADATA_TYPES)); + MatcherAssert.assertThat(result.get(1), Matchers.is("no_dataverse")); + } + + @Test + public void getFriendlyNamesFromFilterQuery_value_should_return_value_when_key_is_metadata_types_and_metadata_blocks_are_null() { + SearchIncludeFragment target = new SearchIncludeFragment(); + Dataverse dataverse = Mockito.mock(Dataverse.class); + Mockito.when(dataverse.getMetadataBlockFacets()).thenReturn(null); + target.setDataverse(dataverse); + + List result = target.getFriendlyNamesFromFilterQuery(String.format("%s:\"no_metadata_blocks\"", SearchFields.METADATA_TYPES)); + MatcherAssert.assertThat(result.get(1), Matchers.is("no_metadata_blocks")); + + Mockito.verify(dataverse).getMetadataBlockFacets(); + } + + @Test + public void getFriendlyNamesFromFilterQuery_value_should_remove_quotes_from_beginning_and_end() { + SearchIncludeFragment target = new SearchIncludeFragment(); + + List result = target.getFriendlyNamesFromFilterQuery("key:\"value_\"_with_quotes\""); + MatcherAssert.assertThat(result.get(1), Matchers.is("value_\"_with_quotes")); + } +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/settings/JvmSettingsTest.java b/src/test/java/edu/harvard/iq/dataverse/settings/JvmSettingsTest.java new file mode 100644 index 00000000000..68458f6623c --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/settings/JvmSettingsTest.java @@ -0,0 +1,20 @@ +package edu.harvard.iq.dataverse.settings; + +import edu.harvard.iq.dataverse.util.testing.JvmSetting; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class JvmSettingsTest { + @Test + @JvmSetting(key = JvmSettings.VERSION, value = "foobar") + void lookupSetting() { + assertEquals("foobar", JvmSettings.VERSION.lookup()); + assertEquals("foobar", JvmSettings.VERSION.lookupOptional().orElse("")); + } + + /* + * TODO: add more tests here for features like old names, patterned settings etc when adding + * these in other pull requests adding new settings making use of these features. + */ +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSourceTest.java b/src/test/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSourceTest.java index 36c4d99f743..621423b3121 100644 --- a/src/test/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSourceTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSourceTest.java @@ -1,41 +1,95 @@ package edu.harvard.iq.dataverse.settings.source; +import edu.harvard.iq.dataverse.util.testing.SystemProperty; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; -import java.io.IOException; -import java.util.Properties; +import java.util.List; +import java.util.regex.Pattern; -import static org.junit.jupiter.api.Assertions.*; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; class AliasConfigSourceTest { - AliasConfigSource source = new AliasConfigSource(); + static AliasConfigSource source = new AliasConfigSource(); + + static final String VALUE = "test1234"; + + static final String SIMPLE_NEW = "dataverse.test.foobar"; + static final String SIMPLE_OLD = "dataverse.former.hello"; + static final String SIMPLE_MULTI_NEW = "dataverse.test.multi"; + static final String SIMPLE_MULTI_OLD = "former.2"; + + static final Pattern PATTERNED_NEW_PATTERN = Pattern.compile("dataverse\\.(.+?)\\.next\\.test"); + static final String PATTERNED_NEW_NAME = "dataverse.single.next.test"; + static final String PATTERNED_OLD_PATTERN = "dataverse.%s.test.foobar"; + static final String PATTERNED_OLD_NAME = "dataverse.single.test.foobar"; + + static final Pattern DUAL_PATTERNED_NEW_PATTERN = Pattern.compile("dataverse\\.(.+?)\\.two\\.(.+?)\\.foobar"); + static final String DUAL_PATTERNED_NEW_NAME = "dataverse.single.two.test.foobar"; + static final String DUAL_PATTERNED_OLD_PATTERN = "dataverse.%s.test.%s.foobar"; + static final String DUAL_PATTERNED_OLD_NAME = "dataverse.single.test.test.foobar"; + + static final Pattern PATTERNED_MULTI_NEW_PATTERN = Pattern.compile("dataverse\\.(.+?)\\.next\\.test2"); + static final String PATTERNED_MULTI_NEW_NAME = "dataverse.multi.next.test2"; + static final String PATTERNED_MULTI_OLD_PATTERN = "dataverse.%s.test.foobar2"; + static final String PATTERNED_MULTI_OLD_NAME = "dataverse.multi.test.foobar2"; + + static final Pattern DUALSINGLE_PATTERNED_NEW_PATTERN = Pattern.compile("dataverse\\.(.+?)\\.(.+?)\\.test2"); + static final String DUALSINGLE_PATTERNED_NEW_NAME = "dataverse.multi.next.test2"; + static final String DUALSINGLE_PATTERNED_OLD_PATTERN = "dataverse.%s.test.foobar2"; + static final String DUALSINGLE_PATTERNED_OLD_NAME = "dataverse.multi.test.foobar2"; + + @BeforeAll + static void setUp() { + source.addAlias(SIMPLE_NEW, List.of(SIMPLE_OLD)); + source.addAlias(SIMPLE_MULTI_NEW, List.of("former.1", SIMPLE_MULTI_OLD, "former.3")); + source.addAlias(PATTERNED_NEW_PATTERN, List.of(PATTERNED_OLD_PATTERN)); + source.addAlias(DUAL_PATTERNED_NEW_PATTERN, List.of(DUAL_PATTERNED_OLD_PATTERN)); + source.addAlias(PATTERNED_MULTI_NEW_PATTERN, List.of("dataverse.%s.test1.foobar1", PATTERNED_MULTI_OLD_PATTERN, "dataverse.test.%s.test")); + source.addAlias(DUALSINGLE_PATTERNED_NEW_PATTERN, List.of(DUALSINGLE_PATTERNED_OLD_PATTERN)); + } + + @Test + void testNullIfNotInScope() { + assertNull(source.getValue(null)); + assertNull(source.getValue("test.out.of.scope")); + } + + @Test + @SystemProperty(key = SIMPLE_OLD, value = VALUE) + void testSimpleAlias() { + assertEquals(VALUE, source.getValue(SIMPLE_NEW)); + } + + @Test + @SystemProperty(key = SIMPLE_MULTI_OLD, value = VALUE) + void testSimpleMultipleAlias() { + assertEquals(VALUE, source.getValue(SIMPLE_MULTI_NEW)); + } + + @Test + @SystemProperty(key = PATTERNED_OLD_NAME, value = VALUE) + void testPatternedAlias() { + assertEquals(VALUE, source.getValue(PATTERNED_NEW_NAME)); + } + + @Test + @SystemProperty(key = DUAL_PATTERNED_OLD_NAME, value = VALUE) + void testDualPatternedAlias() { + assertEquals(VALUE, source.getValue(DUAL_PATTERNED_NEW_NAME)); + } @Test - void getValue() { - // given - System.setProperty("dataverse.hello.foobar", "test"); - Properties aliases = new Properties(); - aliases.setProperty("dataverse.goodbye.foobar", "dataverse.hello.foobar"); - - // when - source.importAliases(aliases); - - // then - assertEquals("test", source.getValue("dataverse.goodbye.foobar")); + @SystemProperty(key = PATTERNED_MULTI_OLD_NAME, value = VALUE) + void testPatternedMultipleAlias() { + assertEquals(VALUE, source.getValue(PATTERNED_MULTI_NEW_NAME)); } @Test - void readImportTestAliasesFromFile() throws IOException { - // given - System.setProperty("dataverse.old.example", "test"); - String filePath = "test-microprofile-aliases.properties"; - - // when - Properties aliases = source.readAliases(filePath); - source.importAliases(aliases); - - // then - assertEquals("test", source.getValue("dataverse.new.example")); + @SystemProperty(key = DUALSINGLE_PATTERNED_OLD_NAME, value = VALUE) + void testDualSinglePatternedAlias() { + assertEquals(VALUE, source.getValue(DUALSINGLE_PATTERNED_NEW_NAME)); } } \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSourceTest.java b/src/test/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSourceTest.java deleted file mode 100644 index 9ceca24aadf..00000000000 --- a/src/test/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigSourceTest.java +++ /dev/null @@ -1,48 +0,0 @@ -package edu.harvard.iq.dataverse.settings.source; - -import edu.harvard.iq.dataverse.settings.Setting; -import edu.harvard.iq.dataverse.settings.SettingsServiceBean; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.MethodOrderer.OrderAnnotation; -import org.junit.jupiter.api.Order; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.TestMethodOrder; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.Mockito; -import org.mockito.junit.jupiter.MockitoExtension; - -import java.util.Arrays; -import java.util.HashSet; -import java.util.Set; - -import static org.junit.jupiter.api.Assertions.*; - -@ExtendWith(MockitoExtension.class) -@TestMethodOrder(OrderAnnotation.class) -class DbSettingConfigSourceTest { - - DbSettingConfigSource dbSource = new DbSettingConfigSource(); - @Mock - SettingsServiceBean settingsSvc; - - @Test - @Order(1) - void testEmptyIfNoSettingsService() { - assertEquals(null, dbSource.getValue("foobar")); - assertDoesNotThrow(DbSettingConfigSource::updateProperties); - } - - @Test - @Order(2) - void testDataRetrieval() { - Set settings = new HashSet<>(Arrays.asList(new Setting(":FooBar", "hello"), new Setting(":FooBarI18N", "de", "hallo"))); - Mockito.when(settingsSvc.listAll()).thenReturn(settings); - - DbSettingConfigSource.injectSettingsService(settingsSvc); - - assertEquals("hello", dbSource.getValue("dataverse.settings.fromdb.FooBar")); - assertEquals("hallo", dbSource.getValue("dataverse.settings.fromdb.FooBarI18N.de")); - } - -} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/util/FileTypeDetectionTest.java b/src/test/java/edu/harvard/iq/dataverse/util/FileTypeDetectionTest.java deleted file mode 100644 index 5d2b9b4d56a..00000000000 --- a/src/test/java/edu/harvard/iq/dataverse/util/FileTypeDetectionTest.java +++ /dev/null @@ -1,42 +0,0 @@ -package edu.harvard.iq.dataverse.util; - -import java.io.File; -import java.io.IOException; -import java.util.logging.Level; -import java.util.logging.Logger; -import org.apache.commons.io.FileUtils; -import org.junit.AfterClass; -import static org.junit.Assert.assertEquals; -import org.junit.BeforeClass; -import org.junit.Test; - -public class FileTypeDetectionTest { - - static String baseDirForConfigFiles = "/tmp"; - - @BeforeClass - public static void setUpClass() { - System.setProperty("com.sun.aas.instanceRoot", baseDirForConfigFiles); - String testFile1Src = "conf/jhove/jhove.conf"; - String testFile1Tmp = baseDirForConfigFiles + "/config/jhove.conf"; - try { - FileUtils.copyFile(new File(testFile1Src), new File(testFile1Tmp)); - } catch (IOException ex) { - Logger.getLogger(JhoveFileTypeTest.class.getName()).log(Level.SEVERE, null, ex); - } - } - - @AfterClass - public static void tearDownClass() { - // SiteMapUtilTest relies on com.sun.aas.instanceRoot being null. - System.clearProperty("com.sun.aas.instanceRoot"); - } - - @Test - public void testDetermineFileTypeJupyterNoteboook() throws Exception { - File file = new File("src/test/java/edu/harvard/iq/dataverse/util/irc-metrics.ipynb"); - // https://jupyter.readthedocs.io/en/latest/reference/mimetype.html - assertEquals("application/x-ipynb+json", FileTypeDetection.determineFileType(file)); - } - -} diff --git a/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java index 226c677ed0f..01fb8aad6cf 100644 --- a/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java @@ -315,6 +315,18 @@ public void testDetermineFileTypeByExtension() { fail("File does not exist: " + file.toPath().toString()); } } + + @Test + public void testDetermineFileTypeFromName() { + //Verify that name of the local file isn't used in determining the type (as we often use *.tmp when the real name has a different extension) + try { + File file = File.createTempFile("empty", "png"); + assertEquals("text/plain", FileUtil.determineFileType(file, "something.txt")); + } catch (IOException ex) { + Logger.getLogger(FileUtilTest.class.getName()).log(Level.SEVERE, null, ex); + } + + } @Test public void testDetermineFileTypeByName() { @@ -329,6 +341,18 @@ public void testDetermineFileTypeByName() { fail("File does not exist: " + file.toPath().toString()); } } + + @Test + public void testDetermineFileTypeFromNameLocalFile() { + //Verify that name of the local file isn't used in determining the type (as we often use *.tmp when the real name has a different extension) + try { + File file = File.createTempFile("empty", "png"); + assertEquals("text/plain", FileUtil.determineFileType(file, "something.txt")); + } catch (IOException ex) { + Logger.getLogger(FileUtilTest.class.getName()).log(Level.SEVERE, null, ex); + } + + } // isThumbnailSuppported() has been moved from DataFileService to FileUtil: /** diff --git a/src/test/java/edu/harvard/iq/dataverse/util/UrlSignerUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/util/UrlSignerUtilTest.java new file mode 100644 index 00000000000..2b9d507758f --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/UrlSignerUtilTest.java @@ -0,0 +1,50 @@ +package edu.harvard.iq.dataverse.util; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.logging.Level; +import java.util.logging.Logger; + +import org.junit.Test; + +public class UrlSignerUtilTest { + + @Test + public void testSignAndValidate() { + + final String url = "http://localhost:8080/api/test1"; + final String get = "GET"; + final String post = "POST"; + + final String user1 = "Alice"; + final String user2 = "Bob"; + final int tooQuickTimeout = -1; + final int longTimeout = 1000; + final String key = "abracadabara open sesame"; + final String badkey = "abracadabara open says me"; + + Logger.getLogger(UrlSignerUtil.class.getName()).setLevel(Level.FINE); + + String signedUrl1 = UrlSignerUtil.signUrl(url, longTimeout, user1, get, key); + assertTrue(UrlSignerUtil.isValidUrl(signedUrl1, user1, get, key)); + assertTrue(UrlSignerUtil.isValidUrl(signedUrl1, user1, null, key)); + assertTrue(UrlSignerUtil.isValidUrl(signedUrl1, null, get, key)); + + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1, null, get, badkey)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1, user2, get, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1, user1, post, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1.replace(user1, user2), user1, get, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1.replace(user1, user2), user2, get, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl1.replace(user1, user2), null, get, key)); + + String signedUrl2 = UrlSignerUtil.signUrl(url, null, null, null, key); + assertTrue(UrlSignerUtil.isValidUrl(signedUrl2, null, null, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl2, null, post, key)); + assertFalse(UrlSignerUtil.isValidUrl(signedUrl2, user1, null, key)); + + String signedUrl3 = UrlSignerUtil.signUrl(url, tooQuickTimeout, user1, get, key); + + assertFalse(UrlSignerUtil.isValidUrl(signedUrl3, user1, get, key)); + } +} diff --git a/src/test/java/edu/harvard/iq/dataverse/util/UrlTokenUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/util/UrlTokenUtilTest.java new file mode 100644 index 00000000000..ffc6b813045 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/UrlTokenUtilTest.java @@ -0,0 +1,50 @@ +package edu.harvard.iq.dataverse.util; + +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.FileMetadata; +import edu.harvard.iq.dataverse.GlobalId; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.when; + +import java.util.ArrayList; +import java.util.List; + +import org.junit.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +public class UrlTokenUtilTest { + + @Test + public void testGetToolUrlWithOptionalQueryParameters() { + + DataFile dataFile = new DataFile(); + dataFile.setId(42l); + FileMetadata fmd = new FileMetadata(); + DatasetVersion dv = new DatasetVersion(); + Dataset ds = new Dataset(); + ds.setId(50L); + ds.setGlobalId(new GlobalId("doi:10.5072/FK2ABCDEF")); + dv.setDataset(ds); + fmd.setDatasetVersion(dv); + List fmdl = new ArrayList(); + fmdl.add(fmd); + dataFile.setFileMetadatas(fmdl); + ApiToken apiToken = new ApiToken(); + apiToken.setTokenString("7196b5ce-f200-4286-8809-03ffdbc255d7"); + URLTokenUtil urlTokenUtil = new URLTokenUtil(dataFile, apiToken, fmd, "en"); + assertEquals("en", urlTokenUtil.replaceTokensWithValues("{localeCode}")); + assertEquals("42 test en", urlTokenUtil.replaceTokensWithValues("{fileId} test {localeCode}")); + assertEquals("42 test en", urlTokenUtil.replaceTokensWithValues("{fileId} test {localeCode}")); + + assertEquals("https://librascholar.org/api/files/42/metadata?key=" + apiToken.getTokenString(), urlTokenUtil.replaceTokensWithValues("{siteUrl}/api/files/{fileId}/metadata?key={apiToken}")); + + URLTokenUtil urlTokenUtil2 = new URLTokenUtil(ds, apiToken, "en"); + assertEquals("https://librascholar.org/api/datasets/50?key=" + apiToken.getTokenString(), urlTokenUtil2.replaceTokensWithValues("{siteUrl}/api/datasets/{datasetId}?key={apiToken}")); + assertEquals("https://librascholar.org/api/datasets/:persistentId/?persistentId=doi:10.5072/FK2ABCDEF&key=" + apiToken.getTokenString(), urlTokenUtil2.replaceTokensWithValues("{siteUrl}/api/datasets/:persistentId/?persistentId={datasetPid}&key={apiToken}")); + } +} diff --git a/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSetting.java b/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSetting.java new file mode 100644 index 00000000000..f54cadaf253 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSetting.java @@ -0,0 +1,64 @@ +package edu.harvard.iq.dataverse.util.testing; + +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.parallel.ResourceAccessMode; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.api.parallel.Resources; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Inherited; +import java.lang.annotation.Repeatable; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + + +/** + * {@code @SetJvmSetting} is a JUnit Jupiter extension to set the value of a + * JVM setting (internally a system property) for a test execution. + * + *

    The key and value of the JVM setting to be set must be specified via + * {@link #key()} and {@link #value()}. After the annotated method has been + * executed, the initial default value is restored.

    + * + *

    {@code SetJvmSetting} can be used on the method and on the class level. + * It is repeatable and inherited from higher-level containers. If a class is + * annotated, the configured property will be set before every test inside that + * class. Any method level configurations will override the class level + * configurations.

    + * + * Parallel execution of tests using this extension is prohibited by using + * resource locking provided by JUnit5 - system properties are a global state, + * these tests NEED to be done serial. + */ +@Retention(RetentionPolicy.RUNTIME) +@Target({ ElementType.METHOD, ElementType.TYPE }) +@Inherited +@Repeatable(JvmSetting.JvmSettings.class) +@ExtendWith(JvmSettingExtension.class) +@ResourceLock(value = Resources.SYSTEM_PROPERTIES, mode = ResourceAccessMode.READ_WRITE) +public @interface JvmSetting { + + /** + * The key of the system property to be set. + */ + edu.harvard.iq.dataverse.settings.JvmSettings key(); + + /** + * The value of the system property to be set. + */ + String value(); + + String[] varArgs() default {}; + + /** + * Containing annotation of repeatable {@code @SetSystemProperty}. + */ + @Retention(RetentionPolicy.RUNTIME) + @Target({ ElementType.METHOD, ElementType.TYPE }) + @Inherited + @ExtendWith(JvmSettingExtension.class) + @interface JvmSettings { + JvmSetting[] value(); + } +} diff --git a/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSettingExtension.java b/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSettingExtension.java new file mode 100644 index 00000000000..56e87589139 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/testing/JvmSettingExtension.java @@ -0,0 +1,75 @@ +package edu.harvard.iq.dataverse.util.testing; + +import edu.harvard.iq.dataverse.settings.JvmSettings; +import org.junit.jupiter.api.extension.AfterTestExecutionCallback; +import org.junit.jupiter.api.extension.BeforeTestExecutionCallback; +import org.junit.jupiter.api.extension.ExtensionContext; + +public class JvmSettingExtension implements BeforeTestExecutionCallback, AfterTestExecutionCallback { + + private ExtensionContext.Store getStore(ExtensionContext context) { + return context.getStore(ExtensionContext.Namespace.create(getClass(), context.getRequiredTestClass(), context.getRequiredTestMethod())); + } + + @Override + public void beforeTestExecution(ExtensionContext extensionContext) throws Exception { + extensionContext.getTestMethod().ifPresent(method -> { + JvmSetting[] settings = method.getAnnotationsByType(JvmSetting.class); + for (JvmSetting setting : settings) { + // get the setting name (might need var args substitution) + String settingName = getSettingName(setting); + + // get the setting ... + String oldSetting = System.getProperty(settingName); + + // if present - store in context to restore later + if (oldSetting != null) { + getStore(extensionContext).put(settingName, oldSetting); + } + + // set to new value + System.setProperty(settingName, setting.value()); + } + }); + } + + @Override + public void afterTestExecution(ExtensionContext extensionContext) throws Exception { + extensionContext.getTestMethod().ifPresent(method -> { + JvmSetting[] settings = method.getAnnotationsByType(JvmSetting.class); + for (JvmSetting setting : settings) { + // get the setting name (might need var args substitution) + String settingName = getSettingName(setting); + + // get a stored setting from context + String oldSetting = getStore(extensionContext).remove(settingName, String.class); + + // if present before, restore + if (oldSetting != null) { + System.setProperty(settingName, oldSetting); + // if NOT present before, delete + } else { + System.clearProperty(settingName); + } + } + }); + } + + private String getSettingName(JvmSetting setting) { + JvmSettings target = setting.key(); + + if (target.needsVarArgs()) { + String[] variableArguments = setting.varArgs(); + + if (variableArguments == null || variableArguments.length != target.numberOfVarArgs()) { + throw new IllegalArgumentException("You must provide " + target.numberOfVarArgs() + + " variable arguments via varArgs = {...} for setting " + target + + " (\"" + target.getScopedKey() + "\")"); + } + + return target.insert(variableArguments); + } + + return target.getScopedKey(); + } +} diff --git a/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemProperty.java b/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemProperty.java new file mode 100644 index 00000000000..2770831296f --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemProperty.java @@ -0,0 +1,60 @@ +package edu.harvard.iq.dataverse.util.testing; + +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.parallel.ResourceAccessMode; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.api.parallel.Resources; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Inherited; +import java.lang.annotation.Repeatable; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * {@code @SystemProperty} is a JUnit Jupiter extension to set the value of an + * arbitrary system property for a test execution. + * + *

    The key and value of the property to be set must be specified via + * {@link #key()} and {@link #value()}. After the annotated method has been + * executed, the initial default value is restored.

    + * + *

    {@code SetJvmSetting} can be used on the method and on the class level. + * It is repeatable and inherited from higher-level containers. If a class is + * annotated, the configured property will be set before every test inside that + * class. Any method level configurations will override the class level + * configurations.

    + * + * Parallel execution of tests using this extension is prohibited by using + * resource locking provided by JUnit5 - system properties are a global state, + * these tests NEED to be done serial. + */ +@Retention(RetentionPolicy.RUNTIME) +@Target({ ElementType.METHOD, ElementType.TYPE }) +@Inherited +@Repeatable(SystemProperty.SystemProperties.class) +@ExtendWith(SystemPropertyExtension.class) +@ResourceLock(value = Resources.SYSTEM_PROPERTIES, mode = ResourceAccessMode.READ_WRITE) +public @interface SystemProperty { + /** + * The key of the system property to be set. + */ + String key(); + + /** + * The value of the system property to be set. + */ + String value(); + + /** + * Containing annotation of repeatable {@code @SystemProperty}. + */ + @Retention(RetentionPolicy.RUNTIME) + @Target({ ElementType.METHOD, ElementType.TYPE }) + @Inherited + @ExtendWith(SystemPropertyExtension.class) + @interface SystemProperties { + SystemProperty[] value(); + } +} \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemPropertyExtension.java b/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemPropertyExtension.java new file mode 100644 index 00000000000..146b7c63713 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/util/testing/SystemPropertyExtension.java @@ -0,0 +1,56 @@ +package edu.harvard.iq.dataverse.util.testing; + +import org.junit.jupiter.api.extension.AfterTestExecutionCallback; +import org.junit.jupiter.api.extension.BeforeTestExecutionCallback; +import org.junit.jupiter.api.extension.ExtensionContext; + +public class SystemPropertyExtension implements BeforeTestExecutionCallback, AfterTestExecutionCallback { + + private ExtensionContext.Store getStore(ExtensionContext context) { + return context.getStore(ExtensionContext.Namespace.create(getClass(), context.getRequiredTestClass(), context.getRequiredTestMethod())); + } + + @Override + public void beforeTestExecution(ExtensionContext extensionContext) throws Exception { + extensionContext.getTestMethod().ifPresent(method -> { + SystemProperty[] settings = method.getAnnotationsByType(SystemProperty.class); + for (SystemProperty setting : settings) { + // get the property name + String settingName = setting.key(); + + // get the setting ... + String oldSetting = System.getProperty(settingName); + + // if present - store in context to restore later + if (oldSetting != null) { + getStore(extensionContext).put(settingName, oldSetting); + } + + // set to new value + System.setProperty(settingName, setting.value()); + } + }); + } + + @Override + public void afterTestExecution(ExtensionContext extensionContext) throws Exception { + extensionContext.getTestMethod().ifPresent(method -> { + SystemProperty[] settings = method.getAnnotationsByType(SystemProperty.class); + for (SystemProperty setting : settings) { + /// get the property name + String settingName = setting.key(); + + // get a stored setting from context + String oldSetting = getStore(extensionContext).remove(settingName, String.class); + + // if present before, restore + if (oldSetting != null) { + System.setProperty(settingName, oldSetting); + // if NOT present before, delete + } else { + System.clearProperty(settingName); + } + } + }); + } +} diff --git a/src/test/resources/propertyFiles/metadataBlockTest.properties b/src/test/resources/propertyFiles/metadataBlockTest.properties new file mode 100644 index 00000000000..25ec317c14f --- /dev/null +++ b/src/test/resources/propertyFiles/metadataBlockTest.properties @@ -0,0 +1,2 @@ +metadatablock.displayName=property_value_for_displayName +metadatablock.displayFacet=property_value_for_displayFacet \ No newline at end of file diff --git a/tests/integration-tests.txt b/tests/integration-tests.txt index 71ba38e0aae..85b37c79835 100644 --- a/tests/integration-tests.txt +++ b/tests/integration-tests.txt @@ -1 +1 @@ -DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT +DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT,BagIT