Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

Merged
merged 22 commits into from
Dec 20, 2018
Merged

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

merged 22 commits into from
Dec 20, 2018

Commits on Nov 21, 2018

  1. Changes to xdmod-ingestor and xdmod-shredder to support cloud based e… 

    …vent data ingestion
    @eiffel777
    eiffel777 committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    ed0a524 View commit details
    Browse the repository at this point in the history

  2. adding checking for exception for when cloud table do not exist. adde… 

    …d etl pipeline function. removed previously added command line flags
    @eiffel777
    eiffel777 committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    56ae3f9 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2018

  1. building filter list after running cloud aggregation in ingestor script

    @eiffel777
    eiffel777 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    c10909b View commit details
    Browse the repository at this point in the history

  2. add cloud shredder abstract class to reduce code duplication. renamed… 

    … Generalcloud.php to Genericcloud.php. replaces etl_overseer script calls in bootstrap.sh to use xdmod-shredder and xdmod-ingestor instead
    @eiffel777
    eiffel777 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    b55927a View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2018

  1. added new function to check if a realm is enabled before trying to in… 

    …gest data for that realm. moved ingestion of organizations and resource_types to their own etl json files so they can be ingested independent of the jobs pipelines.
    @eiffel777
    eiffel777 committed Nov 28, 2018
    Configuration menu
    Copy the full SHA
    a1d9388 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2018

  1. adding tests for new shredders

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    7074fc5 View commit details
    Browse the repository at this point in the history

  2. adding phpdocs for realmEnabled function

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    eef0291 View commit details
    Browse the repository at this point in the history

  3. documentation changes

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    901679b View commit details
    Browse the repository at this point in the history

  4. style and test changes

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    b6d5609 View commit details
    Browse the repository at this point in the history

  5. style changes

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    ddf1eba View commit details
    Browse the repository at this point in the history

  6. adding newline that shouldn't have been deleted

    @eiffel777
    eiffel777 committed Nov 30, 2018
    Configuration menu
    Copy the full SHA
    29c3801 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2018

  1. use false instead of null to determine if realm should be aggregated

    @eiffel777
    eiffel777 committed Dec 3, 2018
    Configuration menu
    Copy the full SHA
    e228e6d View commit details
    Browse the repository at this point in the history

  2. change $realmToAggregate to default to false instead of null

    @eiffel777
    eiffel777 committed Dec 3, 2018
    Configuration menu
    Copy the full SHA
    ebe18ad View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2018

  1. addressing nitpicks from @plessbd

    @eiffel777
    eiffel777 committed Dec 14, 2018
    Configuration menu
    Copy the full SHA
    64abdda View commit details
    Browse the repository at this point in the history

  2. fixing merge conflict

    @eiffel777
    eiffel777 committed Dec 14, 2018
    Configuration menu
    Copy the full SHA
    37564c8 View commit details
    Browse the repository at this point in the history

  3. removing minmaxdate statement from bootstrap.sh

    @eiffel777
    eiffel777 committed Dec 14, 2018
    Configuration menu
    Copy the full SHA
    30a5391 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2018

  1. Merge branch 'xdmod8.1' of https://github.com/ubccr/xdmod into add-cl… 

    …oud-ingestion-to-ingestor-script
    @eiffel777
    eiffel777 committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    3ba69fc View commit details
    Browse the repository at this point in the history

  2. setting etl pipline to use for cloud shredders in constructor.

    @eiffel777
    eiffel777 committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    c1af695 View commit details
    Browse the repository at this point in the history

  3. changes to get tests to pass

    @eiffel777
    eiffel777 committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    45db850 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2018

  1. update suggested by @plessbd

    @eiffel777
    eiffel777 committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    b60aa6b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'xdmod8.1' into add-cloud-ingestion-to-ingestor-script

    Greg Dean authored Dec 20, 2018
    Configuration menu
    Copy the full SHA
    f3dc0ec View commit details
    Browse the repository at this point in the history

  3. Merge branch 'xdmod8.1' into add-cloud-ingestion-to-ingestor-script

    Greg Dean authored Dec 20, 2018
    Configuration menu
    Copy the full SHA
    0eb33e0 View commit details
    Browse the repository at this point in the history