8859 update api error msg

[sekmiller]

What this PR does / why we need it: Update api endpoints and commands so that update/publishing on datasets that are out of compliance with the requirement that datasets containing restricted files must either allow access requests or provide terms of access will throw a wrapped response with the reason for the failure is made clear to the API User.

Which issue(s) this PR closes:

Closes #8859 API calls, related to edit and publish fail if request access is false and no Terms of Access (if there are restricted files on the dataset)

Special notes for your reviewer:

Suggestions on how to test this: Test the publish and edit dataset apis with a dataset that has restricted files with request access false and Terms of Access blank (you may have to manipulate the termsofuseandaccess with a sql query update)

Does this PR introduce a user interface change? If mockups are available, please link/include them here: no

Is there a release notes update needed for this change?: no

Additional documentation: none

[coveralls]

Coverage decreased (-0.002%) to 20.097% when pulling 0204e7c on 8859-update-api-error-msg into 454f3f1 on develop.

[pdurbin]

I don't have time to do a full review but I spoke with @sekmiller and he said you definitely have to get the dataset out of compliance first (such as with update termsofuseandaccess set fileaccessrequest = false;).

I started by trying reproduce the problems on develop (1e52809).

You definitely can't create a new draft from a published version. If I run this...

    String pathToJsonFile = "doc/sphinx-guides/source/_static/api/dataset-update-metadata.json";
    Response updateTitle = UtilIT.updateDatasetMetadataViaNative(datasetPid, pathToJsonFile, apiToken);
    updateTitle.prettyPrint();

... I get this error:

{"status":"ERROR","message":"Command edu.harvard.iq.dataverse.engine.command.impl.CreateDatasetVersionCommand@78cf3b61 failed: Exception thrown from bean: javax.validation.ConstraintViolationException: One or more Bean Validation constraints were violated while executing Automatic Bean Validation on callback event: prePersist for class: edu.harvard.iq.dataverse.TermsOfUseAndAccess. Please refer to the embedded constraint violations for details."}

I haven't tested other edits but it sounds like Steve addressed these.

He said he couldn't reproduce the "delete file" bug.

I'm not sure about "add file".

We might want @qqmyers to take a look since he opened the issue and found some of these bugs.

---

Update: I think I got add and delete reversed. Please check with Steve.

Also, I switched to the branch for this PR and the error is better but still a little long and weird:

{"status":"ERROR","message":"Validation Failed: Failed Validation Terms Of Use and Access (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=null ]).java.util.stream.ReferencePipeline$3@64e2381b"}

[qqmyers]

I haven't run anything but I'm not seeing how this change results in a 409 versus 500 error. If it does, great.

Specific Qs after a discussion with @scolapasta :

• Trying to follow the path through code, it looks like the validateOrDie call throws an illegalcommandexception which I think is caught by

  dataverse/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java

  Lines 632 to 633 in 6bd3ec6

  	} catch (IllegalCommandException ex) {
  	throw new WrappedResponse( ex, forbidden(ex.getMessage() ) );

  resulting in a 403 response? (Do we need a new CommandException subtype to bubble up to a 409?)
• It looks like CreateNewDatasetVersion and SubmitForReview are updated. I think UpdateDatasetVersion can be affected too? (I.e. an existing bad version exists - UI would fix it but api won't and will get a failure.) Have the API calls using that command been checked?
• If UpdateDatasetVersion, CuratePublished commands are affected, I think there are places where they are invoked via commandEngine.submit rather than execCommand which skips the Wrapping step and I think results in 500 errors to the user.
• looking at the test, which is not about a bad dataset, but just a call to restrict when allowAccessRequests/ToA doesn't exist, returns a 400/BAD_REQUEST. Is this another case where a 409 is better - the request is OK, it is just the current state of the dataset that is a problem.
• as the work is done, can specific api calls that are changed be listed for testing/QA purposes?

I'm not sure how far to go here, whether some things should be documented as separate issues, etc.

[pdurbin]

@sekmiller and I just talked about how this message when restricting a file (after disallowing "request access") doesn't make much sense:

{"status":"ERROR","message":"Problem saving datafile data.tab: Validation Failed: Failed Validation Terms Of Use and Access (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=1 ]).java.util.stream.ReferencePipeline$3@470fdaa"

Instead it should say something like "In order to restrict a file, you need to either allow 'request access' or set terms of use."

[pdurbin]

On develop as of 9ac2244 I created a dataset with a restricted file and put it out of compliance with this:

update termsofuseandaccess set fileaccessrequest = false;

Then I tried these methods from UtilIT (from testApiErrors in the PR). As expected, all gave cryptic and ugly messages:

• updateDatasetJsonLDMetadata gave {"status":"ERROR","message":"Validation Failed: Failed Validation Terms Of Use and Access (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=null ]).java.util.stream.ReferencePipeline$3@42dd7845"}
• updateDatasetMetadataViaNative gave {"status":"ERROR","message":"Validation Failed: You must enable request access or add terms of access in datasets with restricted files. (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=null ]).java.util.stream.ReferencePipeline$3@12ba18fb"}
• uploadFileViaNative gave {"status":"ERROR","message":"Failed Validation Terms Of Use and Access"}
• deleteFile gave <summary>Could not delete file: edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Validation Failed: You must enable request access or add terms of access in datasets with restricted files. (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=8 ]).java.util.stream.ReferencePipeline$3@3be2382f</summary>
• publishDatasetViaNativeApi gave {"status":"ERROR","message":"Validation Failed: You must enable request access or add terms of access in datasets with restricted files. (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=8 ]).java.util.stream.ReferencePipeline$3@3307f222"}

Now, with this PR (as of 2aeb457), the same operations now yield CONFLICT (409) as the HTTP response status code (with the exception of SWORD, which is not easily changed). However, only one of the five messages (uploadFileViaNative) has been made more clear. Here's the complete list:

• updateDatasetJsonLDMetadata gave {"status":"ERROR","message":"Validation Failed: Failed Validation Terms Of Use and Access (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=null ]).java.util.stream.ReferencePipeline$3@55af3877"}
• updateDatasetMetadataViaNative gave {"status":"ERROR","message":"Validation Failed: Failed Validation Terms Of Use and Access (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=null ]).java.util.stream.ReferencePipeline$3@108076d4"}
• uploadFileViaNative gave {"status":"ERROR","message":"Terms of Use and Access are invalid. You must enable request access or add terms of access in datasets with restricted files."}
• deleteFile gave <summary>Could not delete file: edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Validation Failed: Terms of Use and Access are invalid. You must enable request access or add terms of access in datasets with restricted files. (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=8 ]).java.util.stream.ReferencePipeline$3@59c2cdb5</summary>
• publishDatasetViaNativeApi gave {"status":"ERROR","message":"Validation Failed: Terms of Use and Access are invalid. You must enable request access or add terms of access in datasets with restricted files. (Invalid value:edu.harvard.iq.dataverse.TermsOfUseAndAccess[ id=8 ]).java.util.stream.ReferencePipeline$3@2cce8cb1"}

I assume we want all five of these messages to be made more clear (including removing noise like "ReferencePipeline").

[pdurbin]

Ok as of 9f23b6a all five errors messages above look good and say {"status":"ERROR","message":"Terms of Use and Access are invalid. You must enable request access or add terms of access in datasets with restricted files."}

I've basically already done the QA for this PR so assuming the tests pass in Jenkins I'll merge it.

[pdurbin]

As discussed in PR comments, looking good now. Approving and merging. All tests passed.