Skip to content

Commit

Permalink
Merge branch 'develop' into 10291-collection_thumbnail IQSS#10291
Browse files Browse the repository at this point in the history
  • Loading branch information
pdurbin committed Sep 4, 2024
2 parents 33550da + d4b9260 commit 27f682f
Show file tree
Hide file tree
Showing 26 changed files with 299 additions and 142 deletions.
3 changes: 3 additions & 0 deletions doc/release-notes/10744-ro-crate-docs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## RO-Crate Support (Metadata Export)

Dataverse now supports [RO-Crate](https://www.researchobject.org/ro-crate/) in the sense that dataset metadata can be exported in that format. This functionality is not available out of the box but you can enable one or more RO-Crate exporters from the [list of external exporters](https://preview.guides.gdcc.io/en/develop/installation/advanced.html#inventory-of-external-exporters). See also #10744.
11 changes: 11 additions & 0 deletions doc/release-notes/10797-update-current-version-bug-fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
A significant bug in the superuser-only "Update-Current-Version" publication was found and fixed in this release. If the Update-Current-Version option was used when changes were made to the dataset Terms (rather than to dataset metadata), or if the PID provider service was down/returned an error, the update would fail and render the dataset unusable and require restoration from a backup. The fix in this release allows the update to succeed in both of these cases and redesigns the functionality such that any unknown issues should not make the dataset unusable (i.e. the error would be reported and the dataset would remain in its current state with the last-published version as it was and changes still in the draft version.)

Users of earlier Dataverse releases are encouraged to alert their superusers to this issue. Those who wish to disable this functionality have two options:
* Change the dataset.updateRelease entry in the Bundle.properties file (or local language version) to "Do Not Use" or similar (doesn't disable but alerts superusers to the issue), or
* Edit the dataset.xhtml file to remove the lines

<c:if test="#{dataverseSession.user.isSuperuser()}">
<f:selectItem rendered="#" itemLabel="#{bundle['dataset.updateRelease']}" itemValue="3" />
</c:if>

, delete the contents of the generated and osgi-cache directories in the Dataverse Payara domain, and restart the Payara server.
1 change: 1 addition & 0 deletions doc/release-notes/10800-add-dataverse-request-json-fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed the "addDataverse" API endpoint (/dataverses/{id} POST) expected request JSON structure to parse facetIds as described in the docs.
6 changes: 6 additions & 0 deletions doc/release-notes/10819-publish-thumbnail-bug.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
The initial release of the Dataverse v6.3 introduced a bug where publishing would break the dataset thumbnail, which in turn broke the rendering of the parent Collection ("dataverse") page. This problem was fixed in the PR 10820.

This bug fix will prevent this from happening in the future, but does not fix any existing broken links. To restore any broken thumbnails caused by this bug, you can call the http://localhost:8080/api/admin/clearThumbnailFailureFlag API, which will attempt to clear the flag on all files (regardless of whether caused by this bug or some other problem with the file) or the http://localhost:8080/api/admin/clearThumbnailFailureFlag/id to clear the flag for individual files. Calling the former, batch API is recommended.

Additionally, the same PR made it possible to turn off the feature that automatically selects of one of the image datafiles to serve as the thumbnail of the parent dataset. An admin can turn it off by raising the feature flag `<jvm-options>-Ddataverse.feature.disable-dataset-thumbnail-autoselect=true</jvm-options>`. When the feature is disabled, a user can still manually pick a thumbnail image, or upload a dedicated thumbnail image.

2 changes: 0 additions & 2 deletions doc/sphinx-guides/source/container/running/demo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,6 @@ Some JVM options can be configured as environment variables. For example, you ca
We are in the process of making more JVM options configurable as environment variables. Look for the term "MicroProfile Config" in under :doc:`/installation/config` in the Installation Guide to know if you can use them this way.

Please note that for a few environment variables (the ones that start with ``%ct`` in :download:`microprofile-config.properties <../../../../../src/main/resources/META-INF/microprofile-config.properties>`), you have to prepend ``_CT_`` to make, for example, ``_CT_DATAVERSE_SITEURL``. We are working on a fix for this in https://github.com/IQSS/dataverse/issues/10285.

There is a final way to configure JVM options that we plan to deprecate once all JVM options have been converted to MicroProfile Config. Look for "magic trick" under "tunables" at :doc:`../app-image` for more information.

Database Settings
Expand Down
12 changes: 12 additions & 0 deletions doc/sphinx-guides/source/contributor/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,18 @@ If the page is written in Markdown (.md), use this form:
:depth: 3
```

### Links

Getting links right with .rst files can be tricky.

#### Custom Titles

You can use a custom title when linking to a document like this:

:doc:`Custom title </api/intro>`

See also <https://docs.readthedocs.io/en/stable/guides/cross-referencing-with-sphinx.html#the-doc-role>

### Images

A good documentation is just like a website enhanced and upgraded by adding high quality and self-explanatory images. Often images depict a lot of written text in a simple manner. Within our Sphinx docs, you can add them in two ways: a) add a PNG image directly and include or b) use inline description languages like GraphViz (current only option).
Expand Down
22 changes: 17 additions & 5 deletions doc/sphinx-guides/source/developers/version-control.rst
Original file line number Diff line number Diff line change
Expand Up @@ -137,17 +137,29 @@ Make a Pull Request
~~~~~~~~~~~~~~~~~~~

Make a pull request to get approval to merge your changes into the develop branch.
If the pull request notes indicate that release notes are necessary, the workflow can then verify the existence of a corresponding file and respond with a 'thank you!' message. On the other hand, if no release notes are detected, the contributor can be gently reminded of their absence. Please see :doc:`making-releases` for guidance on writing release notes.
Note that once a pull request is created, we'll remove the corresponding issue from our kanban board so that we're only tracking one card.

Feedback on the pull request template we use is welcome! Here's an example of a pull request for issue #3827: https://github.com/IQSS/dataverse/pull/3827
Feedback on the pull request template we use is welcome!

Here's an example of a pull request for issue #9729: https://github.com/IQSS/dataverse/pull/10474

Replace Issue with Pull Request
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the pull request closes an issue that has been prioritized, someone from the core team will do the following:

- Move the open issue to the "Done" column of the `project board`_. We do this to track only one card, the pull request, on the project board. Merging the pull request will close the issue because we use the "closes #1234" `keyword <https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue>`_ .
- Copy all labels from the issue to the pull request with the exception of the "size" label.
- Add a size label to the pull request that reflects the amount of review and QA time needed.
- Move the pull request to the "Ready for Review" column.

.. _project board: https://github.com/orgs/IQSS/projects/34

Make Sure Your Pull Request Has Been Advanced to Code Review
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://github.com/orgs/IQSS/projects/34.
Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column on the `project board`_.

Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for various ways to reach out to developers who have enough access to the GitHub repo to move your issue and pull request to the "Code Review" column.
Look at :ref:`getting-help-developers` for various ways to reach out to developers who have enough access to the GitHub repo to move your issue and pull request to the "Code Review" column.

Summary of Git commands
~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
5 changes: 4 additions & 1 deletion doc/sphinx-guides/source/installation/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,10 @@ Use the :ref:`dataverse.spi.exporters.directory` configuration option to specify
Inventory of External Exporters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For a list of external exporters, see the README at https://github.com/gdcc/dataverse-exporters
For a list of external exporters, see the README at https://github.com/gdcc/dataverse-exporters. To highlight a few:

- Croissant
- RO-Crate

Developing New Exporters
^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
5 changes: 4 additions & 1 deletion doc/sphinx-guides/source/user/dataset-management.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,10 @@ Once a dataset has been published, its metadata can be exported in a variety of
- OpenAIRE
- Schema.org JSON-LD

Additional formats can be enabled. See :ref:`inventory-of-external-exporters` in the Installation Guide.
Additional formats can be enabled. See :ref:`inventory-of-external-exporters` in the Installation Guide. To highlight a few:

- Croissant
- RO-Crate

Each of these metadata exports contains the metadata of the most recently published version of the dataset.

Expand Down
2 changes: 1 addition & 1 deletion docker/compose/demo/compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ services:
restart: on-failure
user: payara
environment:
_CT_DATAVERSE_SITEURL: "https://demo.example.org"
DATAVERSE_SITEURL: "https://demo.example.org"
DATAVERSE_DB_HOST: postgres
DATAVERSE_DB_PASSWORD: secret
DATAVERSE_DB_USER: dataverse
Expand Down
5 changes: 5 additions & 0 deletions src/main/java/edu/harvard/iq/dataverse/Dataset.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import edu.harvard.iq.dataverse.license.License;
import edu.harvard.iq.dataverse.makedatacount.DatasetExternalCitations;
import edu.harvard.iq.dataverse.makedatacount.DatasetMetrics;
import edu.harvard.iq.dataverse.settings.FeatureFlags;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.sql.Timestamp;
Expand Down Expand Up @@ -206,6 +207,10 @@ public Dataset(boolean isHarvested) {
StorageUse storageUse = new StorageUse(this);
this.setStorageUse(storageUse);
}

if (FeatureFlags.DISABLE_DATASET_THUMBNAIL_AUTOSELECT.enabled()) {
this.setUseGenericThumbnail(true);
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -851,6 +851,7 @@ public Dataset setNonDatasetFileAsThumbnail(Dataset dataset, InputStream inputSt
}
dataset = DatasetUtil.persistDatasetLogoToStorageAndCreateThumbnails(dataset, inputStream);
dataset.setThumbnailFile(null);
dataset.setUseGenericThumbnail(false);
return merge(dataset);
}

Expand Down
140 changes: 71 additions & 69 deletions src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import static edu.harvard.iq.dataverse.batch.jobs.importer.filesystem.FileRecordJobListener.SEP;
import edu.harvard.iq.dataverse.batch.util.LoggingUtil;
import edu.harvard.iq.dataverse.search.SolrSearchResult;
import edu.harvard.iq.dataverse.settings.FeatureFlags;
import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
import edu.harvard.iq.dataverse.util.BundleUtil;
import edu.harvard.iq.dataverse.util.MarkupChecker;
Expand Down Expand Up @@ -807,100 +808,101 @@ public Long getThumbnailByVersionId(Long versionId) {
return null;
}

Long thumbnailFileId;

// First, let's see if there are thumbnails that have already been
// generated:
try {
thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id "
+ "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o "
+ "WHERE dv.id = " + versionId + " "
+ "AND df.id = o.id "
+ "AND fm.datasetversion_id = dv.id "
+ "AND fm.datafile_id = df.id "
+ "AND df.restricted = false "
+ "AND df.embargo_id is null "
+ "AND df.retention_id is null "
+ "AND o.previewImageAvailable = true "
+ "ORDER BY df.id LIMIT 1;").getSingleResult();
} catch (Exception ex) {
thumbnailFileId = null;
}

if (thumbnailFileId != null) {
logger.fine("DatasetVersionService,getThumbnailByVersionid(): found already generated thumbnail for version " + versionId + ": " + thumbnailFileId);
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
}

if (!systemConfig.isThumbnailGenerationDisabledForImages()) {
// OK, let's try and generate an image thumbnail!
long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitImage();
if (!FeatureFlags.DISABLE_DATASET_THUMBNAIL_AUTOSELECT.enabled()) {
Long thumbnailFileId;

// First, let's see if there are thumbnails that have already been
// generated:
try {
thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id "
+ "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o "
+ "WHERE dv.id = " + versionId + " "
+ "AND df.id = o.id "
+ "AND fm.datasetversion_id = dv.id "
+ "AND fm.datafile_id = df.id "
+ "AND o.previewimagefail = false "
+ "AND df.restricted = false "
+ "AND df.embargo_id is null "
+ "AND df.retention_id is null "
+ "AND df.contenttype LIKE 'image/%' "
+ "AND NOT df.contenttype = 'image/fits' "
+ "AND df.filesize < " + imageThumbnailSizeLimit + " "
+ "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult();
+ "AND o.previewImageAvailable = true "
+ "ORDER BY df.id LIMIT 1;").getSingleResult();
} catch (Exception ex) {
thumbnailFileId = null;
}

if (thumbnailFileId != null) {
logger.fine("obtained file id: " + thumbnailFileId);
DataFile thumbnailFile = datafileService.find(thumbnailFileId);
if (thumbnailFile != null) {
if (datafileService.isThumbnailAvailable(thumbnailFile)) {
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
logger.fine("DatasetVersionService,getThumbnailByVersionid(): found already generated thumbnail for version " + versionId + ": " + thumbnailFileId);
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
}

if (!systemConfig.isThumbnailGenerationDisabledForImages()) {
// OK, let's try and generate an image thumbnail!
long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitImage();

try {
thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id "
+ "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o "
+ "WHERE dv.id = " + versionId + " "
+ "AND df.id = o.id "
+ "AND fm.datasetversion_id = dv.id "
+ "AND fm.datafile_id = df.id "
+ "AND o.previewimagefail = false "
+ "AND df.restricted = false "
+ "AND df.embargo_id is null "
+ "AND df.retention_id is null "
+ "AND df.contenttype LIKE 'image/%' "
+ "AND NOT df.contenttype = 'image/fits' "
+ "AND df.filesize < " + imageThumbnailSizeLimit + " "
+ "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult();
} catch (Exception ex) {
thumbnailFileId = null;
}

if (thumbnailFileId != null) {
logger.fine("obtained file id: " + thumbnailFileId);
DataFile thumbnailFile = datafileService.find(thumbnailFileId);
if (thumbnailFile != null) {
if (datafileService.isThumbnailAvailable(thumbnailFile)) {
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
}
}
}
}
}

// And if that didn't work, try the same thing for PDFs:
if (!systemConfig.isThumbnailGenerationDisabledForPDF()) {
// OK, let's try and generate an image thumbnail!
long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitPDF();
try {
thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id "
+ "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o "
+ "WHERE dv.id = " + versionId + " "
+ "AND df.id = o.id "
+ "AND fm.datasetversion_id = dv.id "
+ "AND fm.datafile_id = df.id "
+ "AND o.previewimagefail = false "
+ "AND df.restricted = false "
+ "AND df.embargo_id is null "
+ "AND df.retention_id is null "
+ "AND df.contenttype = 'application/pdf' "
+ "AND df.filesize < " + imageThumbnailSizeLimit + " "
+ "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult();
} catch (Exception ex) {
thumbnailFileId = null;
}
// And if that didn't work, try the same thing for PDFs:
if (!systemConfig.isThumbnailGenerationDisabledForPDF()) {
// OK, let's try and generate an image thumbnail!
long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitPDF();
try {
thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id "
+ "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o "
+ "WHERE dv.id = " + versionId + " "
+ "AND df.id = o.id "
+ "AND fm.datasetversion_id = dv.id "
+ "AND fm.datafile_id = df.id "
+ "AND o.previewimagefail = false "
+ "AND df.restricted = false "
+ "AND df.embargo_id is null "
+ "AND df.retention_id is null "
+ "AND df.contenttype = 'application/pdf' "
+ "AND df.filesize < " + imageThumbnailSizeLimit + " "
+ "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult();
} catch (Exception ex) {
thumbnailFileId = null;
}

if (thumbnailFileId != null) {
DataFile thumbnailFile = datafileService.find(thumbnailFileId);
if (thumbnailFile != null) {
if (datafileService.isThumbnailAvailable(thumbnailFile)) {
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
if (thumbnailFileId != null) {
DataFile thumbnailFile = datafileService.find(thumbnailFileId);
if (thumbnailFile != null) {
if (datafileService.isThumbnailAvailable(thumbnailFile)) {
assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId);
return thumbnailFileId;
}
}
}
}
}

return null;
}

Expand Down
Loading

0 comments on commit 27f682f

Please sign in to comment.