Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3353 batch job import #3497

Merged
merged 39 commits into from
Feb 9, 2017
Merged

3353 batch job import #3497

merged 39 commits into from
Feb 9, 2017

Commits on Nov 10, 2016

  1. adds support for batch file system import via JSR 352

    @bmckinney
    bmckinney committed Nov 10, 2016
    10 Configuration menu
    Copy the full SHA
    74f58c4 View commit details
    Browse the repository at this point in the history

  2. adds more messages when retries occur

    @bmckinney
    bmckinney committed Nov 10, 2016
    Configuration menu
    Copy the full SHA
    f2fde2e View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2016

  1. removes dataverse data dir and log dir properties from batch job xml … 

    …in favor of system properties
    @bmckinney
    bmckinney committed Nov 14, 2016
    Configuration menu
    Copy the full SHA
    42f3664 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2016

  1. adds notifications for checksum and file system import; adds notifica… 

    …tions to admin in all cases
    @bmckinney
    bmckinney committed Nov 15, 2016
    1 Configuration menu
    Copy the full SHA
    3237a62 View commit details
    Browse the repository at this point in the history

  2. fixes bug in prior merge conflict resolution

    @bmckinney
    bmckinney committed Nov 15, 2016
    Configuration menu
    Copy the full SHA
    02a3c30 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2016

  1. adds import type enumeration

    @bmckinney
    bmckinney committed Nov 16, 2016
    Configuration menu
    Copy the full SHA
    aa8f83c View commit details
    Browse the repository at this point in the history

  2. minor refactor

    @bmckinney
    bmckinney committed Nov 16, 2016
    Configuration menu
    Copy the full SHA
    5fcc9c0 View commit details
    Browse the repository at this point in the history

  3. comments out debugging println statement

    @bmckinney
    bmckinney committed Nov 16, 2016
    Configuration menu
    Copy the full SHA
    a239f57 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2016

  1. modifications to api based on code review feedback

    @bmckinney
    bmckinney committed Nov 18, 2016
    Configuration menu
    Copy the full SHA
    403aa2c View commit details
    Browse the repository at this point in the history

  2. adds single version constraint; integration test teardown destroys pu… 

    …blished dataset tests
    @bmckinney
    bmckinney committed Nov 18, 2016
    Configuration menu
    Copy the full SHA
    774ce3c View commit details
    Browse the repository at this point in the history

  3. action log stores path to json log instead of the full, or potentiall… 

    …y truncated, json
    @bmckinney
    bmckinney committed Nov 18, 2016
    Configuration menu
    Copy the full SHA
    17cfdcf View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2016

  1. failed attempt to use command engine to save dataset (help!)

    @bmckinney
    bmckinney committed Nov 21, 2016
    2 Configuration menu
    Copy the full SHA
    e2046e9 View commit details
    Browse the repository at this point in the history

  2. backs out the failed command engine attempt

    @bmckinney
    bmckinney committed Nov 21, 2016
    Configuration menu
    Copy the full SHA
    afa7cb9 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2016

  1. uses command engine to update dataset version; consolidates update in… 

    … the writer class
    @bmckinney
    bmckinney committed Nov 23, 2016
    Configuration menu
    Copy the full SHA
    f676399 View commit details
    Browse the repository at this point in the history

  2. bootstraps constraint validation for the api; adds unauthorized user … 

    …test; misc. cleanup
    @bmckinney
    bmckinney committed Nov 23, 2016
    Configuration menu
    Copy the full SHA
    d51185f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into 3353-batch-job-import

    @bmckinney
    bmckinney committed Nov 23, 2016
    Configuration menu
    Copy the full SHA
    a99f929 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2016

  1. Merge branch 'develop' into 3353-batch-job-import

    @bmckinney
    bmckinney committed Nov 30, 2016
    Configuration menu
    Copy the full SHA
    4838dbd View commit details
    Browse the repository at this point in the history

  2. creates DataverseRequest with null HttpServletRequest instead of null… 

    … IpAddress; removes catching EJBException
    @bmckinney
    bmckinney committed Nov 30, 2016
    Configuration menu
    Copy the full SHA
    45a90ac View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2016

  1. addresses QA feedback: fixes action log bug; allows system property t… 

    …o override checksumType and checksumManifest; API returns NOT_IMPLEMENTED error if mode isn't MERGE; adds a dataset lock during import; adds integration test for DataverseRole.EDITOR
    @bmckinney
    bmckinney committed Dec 7, 2016
    Configuration menu
    Copy the full SHA
    9de9b24 View commit details
    Browse the repository at this point in the history

  2. fixes bug where directoryLabel is set with a filename (e.g., there is… 

    … no subdirectory, should be empty)
    @bmckinney
    bmckinney committed Dec 7, 2016
    Configuration menu
    Copy the full SHA
    16caeb2 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2016

  1. fixes target of permission to dataset, not parent dataverse

    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    8cbbbb6 View commit details
    Browse the repository at this point in the history

  2. fixes database merge errors thrown by invoking the command engine too… 

    … early on a changing datasetversion
    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    de8a649 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into 3353-batch-job-import

    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    993ec9e View commit details
    Browse the repository at this point in the history

  4. allows any whitespace to separate filename and checksum value in mani… 

    …fest
    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    84fff7b View commit details
    Browse the repository at this point in the history

  5. looks like datafile has to be set to mergeable=true now

    @bmckinney
    bmckinney committed Dec 8, 2016
    1 Configuration menu
    Copy the full SHA
    347807c View commit details
    Browse the repository at this point in the history

  6. adds required boilerplate code to support email notifications

    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    616c12b View commit details
    Browse the repository at this point in the history

  7. replaces explicit EditDataset permission check with: canIssue(UpdateD… 

    …atasetCommand.class)
    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    702bbd9 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'develop' into 3353-batch-job-import

    @bmckinney
    bmckinney committed Dec 8, 2016
    Configuration menu
    Copy the full SHA
    c9402b9 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2016

  1. removes command engine since it fails when a dataset has 5,000+ files

    @bmckinney
    bmckinney committed Dec 10, 2016
    Configuration menu
    Copy the full SHA
    0254cf8 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2017

  1. removes checksum step by adding the property in the import step; adds… 

    … a detailed log file (in addition to json)
    @bmckinney
    bmckinney committed Jan 3, 2017
    Configuration menu
    Copy the full SHA
    e7e65d5 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2017

  1. FileRecordWriter modified to handle batch ingest of file batches as "… 

    …package files".
    @landreev
    landreev committed Jan 30, 2017
    Configuration menu
    Copy the full SHA
    b28f520 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2017

  1. reorganized the parameters of the batch import job, to be able to han… 

    …dle both the "package file" imports, and

imports of files as individual datafiles, as was originally developed.
    @landreev
    landreev committed Jan 31, 2017
    Configuration menu
    Copy the full SHA
    c7335d7 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2017

  1. reorganized the structure of the final package file folder

    @landreev
    landreev committed Feb 2, 2017
    Configuration menu
    Copy the full SHA
    b182f98 View commit details
    Browse the repository at this point in the history

  2. The changes (5 total - will post the list in the file request and/or … 

    …ticket) agreed upon during the last code review).
    @landreev
    landreev committed Feb 2, 2017
    Configuration menu
    Copy the full SHA
    e69211d View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2017

  1. the last few fixes for the batch upload.

    @landreev
    landreev committed Feb 7, 2017
    Configuration menu
    Copy the full SHA
    fc3fade View commit details
    Browse the repository at this point in the history

  2. fix for the notification page.

    @landreev
    landreev committed Feb 7, 2017
    Configuration menu
    Copy the full SHA
    2a1f915 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into 3353-batch-job-import 

    Resolved Conflicts:
	src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java
    @landreev
    landreev committed Feb 7, 2017
    Configuration menu
    Copy the full SHA
    7d8b467 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2017

  1. update mime type for file package #3353

    @pdurbin
    pdurbin committed Feb 8, 2017
    Configuration menu
    Copy the full SHA
    b1be2dc View commit details
    Browse the repository at this point in the history

  2. some last minute fixes/changes: 

    fixes the "multiple passes problem" - size, etc.
per the final conversation with Pete:
a new parameter - "uploadFolder";
the manifest goes back to specifying the files by the relative path inside the upload folder;
the manifest should be placed inside the upload folder.
    @landreev
    landreev committed Feb 8, 2017
    Configuration menu
    Copy the full SHA
    a1c46ef View commit details
    Browse the repository at this point in the history