fix S3 direct upload NPE and NetCDF/HDF5 and keep geospatial metadata extraction

[pdurbin]

What this PR does / why we need it:

The fileMetadataExtractableFromNetcdf method added in PR #9541 breaks S3 direct upload. This PR fixes it.

It turns out a number of features don't work when S3 direct upload is enabled. This PR enumerates them.

Finally a config setting was added: dataverse.netcdf.geo-extract-s3-direct-upload.

Which issue(s) this PR closes:

• closes fileMetadataExtractableFromNetcdf causing regression for S3 direct upload #9601
• related to Direct S3 Upload: Does not detect non-browser supplied file types, locally detected, such as .dta #6762

Special notes for your reviewer:

As if (driverType.equals("tmp")) in IngestServiceBean.saveAndAddFilesToDataset() is executing, note that if S3 direct upload is enabled, driverType is "s3". However, if S3 direct upload is disabled (but S3 is still enabled), driverType is "tmp". (I found this confusing. I'd be happy to add a comment in the code about this.)

This means that methods like extractMetadataNcml() and code to generate thumbnails is not executed when S3 direct upload is enabled.

The behavior above is not changed by this pull request, but it did lead me to set expectations better in the guides.

Suggestions on how to test this:

Primary bug:

• enable S3 direct upload
• upload any file, should work
• upload a NetCDF or HDF5 file, expect no NcML file (always) and no geospatial metadata extraction (unless you enable the new config setting: dataverse.netcdf.geo-extract-s3-direct-upload)

To test FITS extraction:

• enable S3 direct upload
• enable astro block
• upload a FITS file (expect no metadata extraction and the following error in the log: "fits is a filename/extension Dataverse doesn't know about. Consider adding it to the MimeTypeDetectionByFileExtension.properties file.")

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No.

Is there a release notes update needed for this change?:

Yes, I updated the previous release note to mention the new setting.

Additional documentation:

I expanded our docs on what doesn't work during S3 direct upload, including NetCDF/HDF5 metadata extraction. I also documented the new setting.

[github-actions]

📦 Pushed preview application image as

ghcr.io/gdcc/dataverse:9601-keep-features

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[kcondon]

Having some trouble getting this to work. Tried env var, tried jvm option, tried .properties file.
Will try a known good setting to make sure I am configuring it correctly.

[kcondon]

@pdurbin I was unable to get this working using a jvm-option:
-Ddataverse.netcdf.geo-extract-s3-direct-upload=true

I also tried an env var, DATAVERSE_NETCDF_GEO_EXTRACT_S3_DIRECT_UPLOAD and no luck.

[poikilotherm]

Can we please start having an "ingest" scope where we collect these? (Netcdf being a subscope of it)

[pdurbin]

Maybe? The timing is poor because I was away for two weeks and now you're away until August.

What are you suggesting, specifically?

Instead of...

dataverse.netcdf.geo-extract-s3-direct-upload

... you'd like to see...

dataverse.ingest.netcdf.geo-extract-s3-direct-upload?

I'm ok with it if @landreev and/or @scolapasta are ok with it. And @sekmiller did the review. And @qqmyers does a lot of storage work. They might want to weigh in.

For the record, I looked for ingest stuff in our config page and we do already have these:

• dataverse.files.<id>.ingestsizelimit
• :TabularIngestSizeLimit

Personally, I've been referring to this feature as "metadata extraction" (from a file) rather than ingest. I'm not sure where we draw the line on what ingest is.

I'll plug having a glossary, like I always do! 😜

• User Guide - Create a Glossary of Terms #4274

[coveralls]

coverage: 20.373% (+0.003%) from 20.37% when pulling bbea224 on 9601-keep-features into f24c7de on develop.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:9601-keep-features

ghcr.io/gdcc/configbaker:9601-keep-features

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[pdurbin]

I removed the "waiting" label 'cuz I'm back, baby! 😜

First, I merged the latest from develop.

Then I retested, this time on my dev1 server using JVM options rather than on my laptop using Docker and environment variables (both should work).

Here's the file I'm using to test: https://github.com/IQSS/dataverse/blob/develop/src/test/resources/netcdf/ICOADS_R3.0.0_1662-10.nc

The geospatial block must be enabled for the bounding box to be populated.

By using <jvm-options>-Ddataverse.netcdf.geo-extract-s3-direct-upload=true</jvm-options> in domain.xml, it was populated, like this:

If you remove that JVM option and keep the setup the same (s3 direct upload enabled), the bounding box should not be populated.

For reference, here are some of my JVM options. I configured a new collection to use this "s3" store:

<jvm-options>-Ddataverse.files.s3.type=s3</jvm-options>
<jvm-options>-Ddataverse.files.s3.label=s3</jvm-options>
<jvm-options>-Ddataverse.files.s3.custom-endpoint-url=s3.us-east-2.amazonaws.com</jvm-options>
<jvm-options>-Ddataverse.files.s3.custom-endpoint-region=us-east-2</jvm-options>
<jvm-options>-Ddataverse.files.s3.bucket-name=pdurbin</jvm-options>
<jvm-options>-Ddataverse.files.s3.access-key=REDACTED</jvm-options>
<jvm-options>-Ddataverse.files.s3.secret-key=REDACTED</jvm-options>
<jvm-options>-Ddataverse.files.s3.upload-redirect=true</jvm-options>
<jvm-options>-Ddataverse.netcdf.geo-extract-s3-direct-upload=true</jvm-options>

I'm putting this into "Ready for QA". I'm happy to be co-assigned to work through any problems.

[kcondon]

@pdurbin Thanks for the examples. I misunderstood -I was thinking of the aux file that is generated post upload for netcdf files. Also, the netcdf file may not contain bounding box info so need to have one that does. I have confirmed bounding box metadata is extracted for direct s3 upload. Note that the netcdf aux file is not generated for direct s3 upload.