Add new APIs in support of rsync

[djbrooke]

We'll need to add some new APIs to support adding files using rsync.

• Checksum validation API
• Upload Mechanisms API (was Upload API)
• Script Request API

[pdurbin]

During the 2016-09-08 SBGrid Sprint Planning meeting ( https://docs.google.com/document/d/1wWSdKUOGA1L7UqFsgF3aOs8_9uyjnVpsPAxk7FObOOI/edit ) this issue was given an effort level of "8". Here are links to the code that was demo'ed the day before:

• Checksum validation API: https://github.com/bmckinney/dataverse-canonical/blob/069fd8d72bfe96ec93b4005550b0c771dee01fc4/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java#L728
• Upload API: ???
• Script Request API: https://github.com/bmckinney/dataverse-canonical/blob/069fd8d72bfe96ec93b4005550b0c771dee01fc4/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java#L828

The biggest unknown for me here is what is meant by the "Upload API". Has it been written already? I'm adding @djbrooke @scolapasta and @bmckinney to help me figure out what the Upload API is.

[pdurbin]

@bmckinney @scolapasta @djbrooke and I discussed this and it turns out that what was really meant was Upload Mechanisms API. This is the "setUploadMechanisms" method, for example at https://github.com/bmckinney/dataverse-canonical/blob/069fd8d72bfe96ec93b4005550b0c771dee01fc4/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java#L775 which is to say that that code has already been written so we should go ahead and make a pull request. I moved this issue to Ready at https://waffle.io/IQSS/dataverse

[pdurbin]

@bmckinney and I discussed this issue this morning at https://dataverseteam.slack.com/archives/dcm-merge/p1477917910000400 and I dragged it to "Development" in https://waffle.io/IQSS/dataverse and will assign it to myself since I plan to work on it this week.

[pdurbin]

@bmckinney and I discussed this issue today and he said I shouldn't use my old code at https://github.com/pdurbin/dataverse/tree/3145-dcm because he had to tweak it. He's going to leave a comment here about where I should look for the latest code. Thanks!

[pdurbin]

I'm picking up work on this again and just pushed some scratch work to pdurbin@172b968

Because pull request #3497 has not yet been merged, I had to steal an update to UserNotification.java and I hard-coded "42" rather than calling jo.start("FileSystemImportJob", props);. @djbrooke and I agree that the plan is to merge pull request #3497 first. Then I'll merge that code into whatever branch I put through QA for this issue. Next I'm considering making a new branch based on the branch behind pull request #3497 (3353-batch-job-import) so see if that makes more sense, so that I don't have to wait for it to be merged to have a complete solution.

[djbrooke]

In the planning meeting today, @pdurbin mentioned that he's blocked by the code in #3353 not currently working. @landreev offered his assistance.

A helpful concept (to me, at least) is that someone suggested thinking of this as an API wrapper around the rsync functionality.

[pdurbin]

I learned in a meeting yesterday with @pameyer @djbrooke and @scolapasta yesterday that @bmckinney is apparently working on this. It sounds like the plan is to copy and paste working code from https://github.com/sbgrid/sbgrid-dataverse/tree/feature/dcm-api (unless I'm mistaken) to make a pull request against the main IQSS repo.

[bmckinney]

The SBGrid feature overlay was made into a Dataverse branch back on 11/4. I just added a couple bug fixes and merged develop.

@pameyer and I have a working DCM at dv.sbgrid.org (basically the demo we showed at the September 2016 meeting). (see: Testing DCM) It includes the UI overlay for DCM (not yet made into a dataverse branch since it hasn't gone through code review, etc.)

Finally, see swagger docs for more details.

[bmckinney]

sorry, didn't mean to close this!

[pdurbin]

@bmckinney ah, good to know that the 3352-dcm-apis branch is in play. Thanks for the reminder. I much prefer working from a branch that's in the main IQSS repo rather than the overlay.

[djbrooke]

Moving out of development. This is dependent on #3353 and will be started on when that code is merged.

[djbrooke]

I'm closing this issue. We'll create new issues, clarify, and re-estimate if we still need these APIs as part of the new approach for handling large datasets.

[pdurbin]

This issue lives on as #3725.