Validate the physical files when publishing a dataset

[landreev]

This is a proposal/an idea to consider.

Analyzing the issues we've had with users' physical files over the years, most were caused by something that went wrong with the files still being in the Draft/unpublished state.

This is because we only attempt/allow to delete the physical file before the datafile gets published. And that's when something can go wrong - most commonly, an attempted delete or ingest would remove or corrupt the physical file, while preserving the datafile entry in the database. Then the user publishes the version, without realizing the physical file is missing.

This makes me think we should consider going through the files as they are being published, and validating/confirming that they are still present, that the checksums add up, etc.
Somewhat similarly to how we attempt to register global ids for datafiles on publish.

Doing so would prevent most, if not all possible situations when a physical file is missing from a published dataset.

[scolapasta]

@landreev agreed this is a good idea and can reuse the publishing in progress locks that file PIDs use. (it could also conceivably use the publish workflows that @michbarsinai added).File exists and checksum matches seem like the right checks.

[djbrooke]

I moved this to ready. Since this is a case where there would need to be intervention from the installation's support contact, the red alert message to contact support should be presented.

[djbrooke]

• Could take place before we initiate File PID registration (note this would introduce a new lock for those installations not using File PIDs)
• Should we update the messaging on the "Locks in progress..."?
• The person should receive a message to contact the installation support if this check fails
• In future cases such as TRSAs, we may only be able to validate the files that Dataverse manages
• We should get an idea of the performance hit here (we are OK with some hit for the sake of data integrity)

[landreev]

We ran into a problem of creating a RestAssured failure test - specifically, how to create an invalid datafile, with the checksum that doesn't match the content. Since RestAssured tests are run remotely, there's no direct access to the filesystem to mess with the file (delete/overwrite it, etc) after the datafile was created via API.
I've proposed to address it in #6783, since this would be easy to achieve when S3 is used for storage.

[landreev]

@kcondon - The scenario you brought up yesterday - when somebody is publishing a version with nothing but a single typo correction in the metadata (and no new files added) - it really does feel wrong to go through validating every file in the dataset, again.
Should I go ahead and change it to validating new (unpublished) files only?

[scolapasta]

I haven't looked at the details, but (related to what you suggest) only do these checks if/when files change? So even if uploading a new file, maybe we still check them all, but we don't have to do checks if only metadata changes. Or similarly, do check for any major version. (this could still mean only metadata changes, as the user can select that, but a) may make the logic simple, and b) might be a good check to happen in those circumstances anyway.

[kcondon]

@landreev Well, your validation scenario is to confirm that the set of files was not modified in some way due to an errant write operation, such as an unsuccessful delete. I was focusing more on validation of new files so did not consider as deeply what you were trying to detect. I suppose that, however unlikely, any write operation might impact files so reverifying the set makes sense, contingent on performance.

I just wonder whether there might ultimately be a ux impact, as there is now, when updating a dataset with a large number of files but where the current changes are minimal. Now, all file doi metadata entries are rewritten at DataCite, regardless of what has changed and that can take a while with lots of files.

[landreev]

... to confirm that the set of files was not modified in some way due to an errant write operation, such as an unsuccessful delete. ...

We never try to delete the physical file associated with the datafile, once it's published. At the same time, it's not entirely accurate to say that we NEVER touch the physical file once the datafile is published. We have the re-ingest and un-ingest APIs now, for example. Let me think about it some more. Maybe something along the lines of @scolapasta's suggestions - major versions only? - would be a good balance.