3353 batch job import #3497

bmckinney · 2016-11-29T20:57:10Z

RFI Checklist

Before submitting the pull request, fill out sections (1.) Related Issues and (2.) Pull Request Checklist.

1. Related Issues

List and link to the issues in this Pull Request.

Connects to File Import Batch job in support of rsync #3353 File Import Batch job in support of rsync

2. Pull Request Checklist

Functionality completed as described in FRD
Dependencies, risks, assumptions in FRD addressed
Unit tests completed
Deployment requirements identified (e.g., SQL scripts, indexing)
Documentation completed
All code checkins completed

3. Review Checklist

After the pull request has been submitted, fill out this section.

Code review completed or waived
Testing requirements completed
Usability testing completed or waived
Support testing completed or waived
Merged with develop branch and resolved conflicts

…in favor of system properties

…tions to admin in all cases

…blished dataset tests

…y truncated, json

… the writer class

…test; misc. cleanup

coveralls · 2016-11-29T21:01:32Z

Coverage decreased (-0.1%) to 8.613% when pulling a99f929 on 3353-batch-job-import into 0cefe11 on develop.

coveralls · 2016-11-30T17:26:33Z

Coverage decreased (-0.1%) to 8.612% when pulling 4838dbd on 3353-batch-job-import into f739a75 on develop.

… IpAddress; removes catching EJBException

coveralls · 2016-11-30T17:36:07Z

Coverage decreased (-0.1%) to 8.613% when pulling 45a90ac on 3353-batch-job-import into f739a75 on develop.

…o override checksumType and checksumManifest; API returns NOT_IMPLEMENTED error if mode isn't MERGE; adds a dataset lock during import; adds integration test for DataverseRole.EDITOR

coveralls · 2016-12-07T12:16:43Z

Coverage decreased (-0.1%) to 8.606% when pulling 9de9b24 on 3353-batch-job-import into f739a75 on develop.

… no subdirectory, should be empty)

coveralls · 2016-12-07T13:30:38Z

Coverage decreased (-0.1%) to 8.606% when pulling 16caeb2 on 3353-batch-job-import into f739a75 on develop.

… early on a changing datasetversion

coveralls · 2016-12-08T15:12:23Z

Coverage decreased (-0.1%) to 8.606% when pulling de8a649 on 3353-batch-job-import into f739a75 on develop.

coveralls · 2016-12-08T15:13:57Z

Coverage decreased (-0.1%) to 8.606% when pulling 993ec9e on 3353-batch-job-import into 75a4146 on develop.

coveralls · 2016-12-08T18:57:49Z

Coverage decreased (-0.1%) to 8.599% when pulling 616c12b on 3353-batch-job-import into 75a4146 on develop.

…atasetCommand.class)

coveralls · 2016-12-08T19:11:54Z

Coverage decreased (-0.1%) to 8.598% when pulling c9402b9 on 3353-batch-job-import into 7591e36 on develop.

coveralls · 2016-12-08T19:14:41Z

Coverage decreased (-0.1%) to 8.598% when pulling c9402b9 on 3353-batch-job-import into 7591e36 on develop.

… a detailed log file (in addition to json)

coveralls · 2017-01-03T17:11:09Z

Coverage decreased (-0.1%) to 8.59% when pulling e7e65d5 on 3353-batch-job-import into 7591e36 on develop.

…package files".

…dle both the "package file" imports, and imports of files as individual datafiles, as was originally developed.

…ticket) agreed upon during the last code review).

landreev · 2017-02-02T22:49:34Z

OK, we agreed to make a number of changes during the last code review, 5 total, addressed in my latest commit:

Reversed to the logic of trusting the external file copier to create the top-level folder for the files in the destination dataset directory. No extra folder is being created now.
However, the final folder gets renamed, to a standard Dataverse-generated name, as used for "normal" files.
Added an optional size parameter ("totalSize") to the batch job API call. If supplied, the long integer will be used as the size for the package Datafile. If not supplied, the import job will calculate the sum of all the files in the folder.
Instead of calculating the checksum on the new line-separated list of individual cheksums, the checksum of the manifest file is used for the final datafile checksum.
The functionality for the "individual file mode" (importing individual files as separate Datafiles) is preserved in the FileRecordWriter, but disabled on the API level; i.e., not available to the API user, for now.
The mime type for a "file package" datafile is changed to a classier "application/vnd.edu.harvard.iq.dataverse.file-package".
A boolean method isFilePackage() added to DataFile.java.

Please review and see if anything's missing.

Also, Phil just found something else that I missed - Bill created some RestAssured tests that are failing, now that the import process has been changed. Let's discuss how to proceed; should we move it to QA and treat the RA tests as a separate issue? Or fix it now?

Also, the branch needs to be synced up to the develop, before it can be QAed.

Resolved Conflicts: src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java

coveralls · 2017-02-07T06:26:27Z

Coverage decreased (-0.1%) to 9.387% when pulling 7d8b467 on 3353-batch-job-import into 1ba73ed on develop.

coveralls · 2017-02-08T20:40:04Z

Coverage increased (+0.02%) to 9.532% when pulling b1be2dc on 3353-batch-job-import into 1ba73ed on develop.

fixes the "multiple passes problem" - size, etc. per the final conversation with Pete: a new parameter - "uploadFolder"; the manifest goes back to specifying the files by the relative path inside the upload folder; the manifest should be placed inside the upload folder.

coveralls · 2017-02-08T23:44:57Z

Coverage increased (+0.09%) to 9.602% when pulling a1c46ef on 3353-batch-job-import into 1ba73ed on develop.

bmckinney added 16 commits November 10, 2016 12:02

adds support for batch file system import via JSR 352

74f58c4

adds more messages when retries occur

f2fde2e

removes dataverse data dir and log dir properties from batch job xml …

42f3664

…in favor of system properties

adds notifications for checksum and file system import; adds notifica…

3237a62

…tions to admin in all cases

fixes bug in prior merge conflict resolution

02a3c30

adds import type enumeration

aa8f83c

minor refactor

5fcc9c0

comments out debugging println statement

a239f57

modifications to api based on code review feedback

403aa2c

adds single version constraint; integration test teardown destroys pu…

774ce3c

…blished dataset tests

action log stores path to json log instead of the full, or potentiall…

17cfdcf

…y truncated, json

failed attempt to use command engine to save dataset (help!)

e2046e9

backs out the failed command engine attempt

afa7cb9

uses command engine to update dataset version; consolidates update in…

f676399

… the writer class

bootstraps constraint validation for the api; adds unauthorized user …

d51185f

…test; misc. cleanup

Merge branch 'develop' into 3353-batch-job-import

a99f929

Merge branch 'develop' into 3353-batch-job-import

4838dbd

creates DataverseRequest with null HttpServletRequest instead of null…

45a90ac

… IpAddress; removes catching EJBException

addresses QA feedback: fixes action log bug; allows system property t…

9de9b24

…o override checksumType and checksumManifest; API returns NOT_IMPLEMENTED error if mode isn't MERGE; adds a dataset lock during import; adds integration test for DataverseRole.EDITOR

fixes bug where directoryLabel is set with a filename (e.g., there is…

16caeb2

… no subdirectory, should be empty)

bmckinney added 3 commits December 8, 2016 10:08

fixes target of permission to dataset, not parent dataverse

8cbbbb6

fixes database merge errors thrown by invoking the command engine too…

de8a649

… early on a changing datasetversion

Merge branch 'develop' into 3353-batch-job-import

993ec9e

adds required boilerplate code to support email notifications

616c12b

bmckinney added 2 commits December 8, 2016 14:07

replaces explicit EditDataset permission check with: canIssue(UpdateD…

702bbd9

…atasetCommand.class)

Merge branch 'develop' into 3353-batch-job-import

c9402b9

bmckinney added 2 commits December 10, 2016 13:19

removes command engine since it fails when a dataset has 5,000+ files

0254cf8

removes checksum step by adding the property in the import step; adds…

e7e65d5

… a detailed log file (in addition to json)

This was referenced Jan 4, 2017

Add new APIs in support of rsync #3352

Closed

File Upload: Allow users to skip ingest as tabular data. #2199

Closed

landreev added 2 commits January 30, 2017 17:12

FileRecordWriter modified to handle batch ingest of file batches as "…

b28f520

…package files".

reorganized the parameters of the batch import job, to be able to han…

c7335d7

…dle both the "package file" imports, and imports of files as individual datafiles, as was originally developed.

djbrooke mentioned this pull request Jan 31, 2017

File Import Batch job in support of rsync #3353

Closed

landreev added 2 commits February 2, 2017 13:31

reorganized the structure of the final package file folder

b182f98

The changes (5 total - will post the list in the file request and/or …

e69211d

…ticket) agreed upon during the last code review).

landreev added 3 commits February 6, 2017 23:23

the last few fixes for the batch upload.

fc3fade

fix for the notification page.

2a1f915

Merge branch 'develop' into 3353-batch-job-import

7d8b467

Resolved Conflicts: src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java

update mime type for file package #3353

b1be2dc

kcondon merged commit 7162f5a into develop Feb 9, 2017

kcondon deleted the 3353-batch-job-import branch February 9, 2017 16:15

pdurbin mentioned this pull request Mar 7, 2017

File upload hangs for large zip file (>= ~1 GB) #3634

Closed

pdurbin added this to the 4.6.1 - ORCID and File Replace milestone Mar 9, 2017

pdurbin mentioned this pull request May 15, 2017

Dataverse -> DCM APIs #3725

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3353 batch job import #3497

3353 batch job import #3497

bmckinney commented Nov 29, 2016 •

edited by kcondon

Loading

coveralls commented Nov 29, 2016

coveralls commented Nov 30, 2016

coveralls commented Nov 30, 2016

coveralls commented Dec 7, 2016

coveralls commented Dec 7, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Jan 3, 2017

landreev commented Feb 2, 2017

coveralls commented Feb 7, 2017

coveralls commented Feb 8, 2017

coveralls commented Feb 8, 2017

3353 batch job import #3497

3353 batch job import #3497

Conversation

bmckinney commented Nov 29, 2016 • edited by kcondon Loading

RFI Checklist

1. Related Issues

2. Pull Request Checklist

3. Review Checklist

coveralls commented Nov 29, 2016

coveralls commented Nov 30, 2016

coveralls commented Nov 30, 2016

coveralls commented Dec 7, 2016

coveralls commented Dec 7, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Dec 8, 2016

coveralls commented Jan 3, 2017

landreev commented Feb 2, 2017

coveralls commented Feb 7, 2017

coveralls commented Feb 8, 2017

coveralls commented Feb 8, 2017

bmckinney commented Nov 29, 2016 •

edited by kcondon

Loading