Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

eiffel777 · 2018-11-30T15:26:56Z

This PR adds support for shredding and ingesting cloud event logs in either the OpenStack or Eucalyptus(soon to be renamed Generic Cloud) log file formats.

For shredding the cloud log files two shredder classes are added that implement the shredDirectory function that is used when xdmod-shredder is called using the -d option. Shredding cloud logs by file using the -i flag is not supported and throws an exception.

For ingesting the shredded data a new command line option called datatype is added that allows you to specify what sort of data you want to ingest. Currently only openstack and genericcloud are supported.

The aggregate option now takes an optional parameter to specify what data you want to aggregate. Currently this is either jobs or cloud. If no parameter is specified it will attempt to aggregate both jobs and cloud data.

If no command line options are specified when running xdmod-ingestor then it will try to ingest both cloud and jobs data. There may be cases where only the jobs realm is being used but xdmod-ingestor with no command line options is run. In this example xdmod-ingestor will attempt to ingest jobs data and cloud data. In order to prevent errors from trying to ingest cloud data that doesn't exist the new function realmEnabled is used in order to only attempt ingestion of realms that are enabled.

There are some pipeline actions that needed to exist outside of the jobs pipelines becuase of foreign key constraints on tables used to ingest resources. This was done for actions pertaining to ingesting organizations and resource_types from their respective config files.

The runEtlPipeline function was added to /ETL/Utilities class. It is similar to the runEtlPipeline function in DataWarehouseInitializer but is not private and does not require you to instantiate the DataWarehouseInitializer class and the two other classes that are passed to the DataWarehouseInitializer constructor but not used in runEtlPipeline. In a subsequent pull request I will remove the runEtlPipeline function from the DataWarehouseInitializer class and point anything that uses it to the ETL\Utilities::runEtlPipeline().

Motivation and Context

We want to have a unified way of shredding and ingesting all types of data that xdmod needs.

Tests performed

Manual testing in docker and added new tests for shredder classes that test to make sure if a directory that does not exist is passed to the shredder then false is returned.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project as found in the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

…vent data ingestion

…d etl pipeline function. removed previously added command line flags

… Generalcloud.php to Genericcloud.php. replaces etl_overseer script calls in bootstrap.sh to use xdmod-shredder and xdmod-ingestor instead

…gest data for that realm. moved ingestion of organizations and resource_types to their own etl json files so they can be ingested independent of the jobs pipelines.

jpwhite4 · 2018-12-03T16:45:35Z

bin/xdmod-ingestor

@@ -108,6 +111,7 @@ function main()
                break;
            case 'aggregate':
                $aggregate = true;
+                $realmToAggregate = $value;


According to the docs the $value for an optional argument is false if it is not specified. Assuming the documentation is correct you'll need to check for === false rather than === null on line 287. Please confirm.

@jpwhite4 Yeah, that does seem to be the case. It should be changed now.

bin/xdmod-ingestor

jpwhite4 · 2018-12-14T19:30:35Z

bin/xdmod-shredder

@@ -233,7 +233,7 @@ function main()
        $logger->notice("Job errors written to '$jobErrorLogFile'");
    }

-    if (!$dryRun) {
+    if (!$dryRun && in_array($format, array('pbs', 'slurm', 'lsf', 'sge', 'uge'))) {


Please can you explain the reason for this change.

plessbd · 2018-12-14T19:37:10Z

classes/ETL/Utilities.php

@@ -323,4 +323,54 @@ public static function quoteVariables(array $variables, VariableStore $variableS

        return $localVariableMap;
    }  // quoteVariables()
+
+    public static function runEtlPipeline(array $pipelines, $logger, array $params = array())


It feels like this should be a separate pull request that also removes similar code from the code base:

xdmod/classes/OpenXdmod/Migration/Etlv2Migration.php

Line 54 in 6149358

$overseer = new EtlOverseer($overseerOptions, $this->logger);

xdmod/classes/OpenXdmod/DataWarehouseInitializer.php

Line 322 in 6149358

$overseer = new EtlOverseer($overseerOptions, $this->logger);

xdmod/classes/OpenXdmod/Setup/DatabaseSetup.php

Line 163 in 133a3ae

$overseer = new \ETL\EtlOverseer($overseerOptions, $logger);

xdmod/bin/acl-config

Line 2251 in 133a3ae

$overseer = new EtlOverseer($overseerOptions, $log);

Specifically so we dont have the many places this is done in our code base and we dont forget about it.

You do state in the description that there is a subsequent PR for this can you please link to it.

Here is the branch, https://github.com/eiffel777/xdmod/tree/move-runEtlPipeline-references-to-etl-utilities-class, that has the changes talked about in the last paragraph of the description. It's a bit behind xdmod8.1 right now so there is a some extra code in the diff that can be ignored

…oud-ingestion-to-ingestor-script

classes/OpenXdmod/Shredder/Genericcloud.php

plessbd · 2018-12-17T15:30:33Z

classes/OpenXdmod/Shredder/Openstack.php

@@ -7,14 +7,14 @@

 namespace OpenXdmod\Shredder;

+use CCR\DB\iDatabase;
+
 class OpenStack extends aCloud


Should the filename match the class name OpenStack vs Openstack

Thanks for the catch. It's changed now.

plessbd · 2018-12-20T13:07:24Z

classes/OpenXdmod/DataWarehouseInitializer.php

+    private function realmEnabled($realm)
+    {
+        $realms = $this->warehouseDb->query("SELECT * FROM moddb.realms WHERE display = :realm", [':realm' => $realm]);
+        return (count($realms) > 0) ? true : false;


Suggested change

return (count($realms) > 0) ? true : false;

return (count($realms) > 0);

configuration/etl/etl.d/jobs_cloud_eucalyptus.json

plessbd · 2018-12-20T16:11:47Z

bin/xdmod-shredder

@@ -234,11 +234,16 @@ function main()
    }

    if (!$dryRun) {
-        $logger->notice('Normalizing data');
+        $logger->notice('Normalizing data!');


I am so excited we are normalizing this data, before I was just fine.

eiffel777 added 8 commits November 21, 2018 11:42

Changes to xdmod-ingestor and xdmod-shredder to support cloud based e…

ed0a524

…vent data ingestion

adding checking for exception for when cloud table do not exist. adde…

56ae3f9

…d etl pipeline function. removed previously added command line flags

building filter list after running cloud aggregation in ingestor script

c10909b

add cloud shredder abstract class to reduce code duplication. renamed…

b55927a

… Generalcloud.php to Genericcloud.php. replaces etl_overseer script calls in bootstrap.sh to use xdmod-shredder and xdmod-ingestor instead

added new function to check if a realm is enabled before trying to in…

a1d9388

…gest data for that realm. moved ingestion of organizations and resource_types to their own etl json files so they can be ingested independent of the jobs pipelines.

adding tests for new shredders

7074fc5

adding phpdocs for realmEnabled function

eef0291

documentation changes

901679b

eiffel777 self-assigned this Nov 30, 2018

eiffel777 added Category:ETL Extract Transform Load Category:Cloud Cloud Realm labels Nov 30, 2018

eiffel777 added this to the 8.1.0 milestone Nov 30, 2018

eiffel777 added 3 commits November 30, 2018 13:22

style and test changes

b6d5609

style changes

ddf1eba

adding newline that shouldn't have been deleted

29c3801

jpwhite4 reviewed Dec 3, 2018

View reviewed changes

use false instead of null to determine if realm should be aggregated

e228e6d

eiffel777 force-pushed the add-cloud-ingestion-to-ingestor-script branch from 902b7ba to e228e6d Compare December 3, 2018 19:06

change $realmToAggregate to default to false instead of null

ebe18ad

plessbd reviewed Dec 14, 2018

View reviewed changes

bin/xdmod-ingestor Outdated Show resolved Hide resolved

plessbd reviewed Dec 14, 2018

View reviewed changes

bin/xdmod-ingestor Show resolved Hide resolved

plessbd reviewed Dec 14, 2018

View reviewed changes

bin/xdmod-ingestor Show resolved Hide resolved

addressing nitpicks from @plessbd

64abdda

eiffel777 force-pushed the add-cloud-ingestion-to-ingestor-script branch from edd9609 to 64abdda Compare December 14, 2018 18:43

eiffel777 added 2 commits December 14, 2018 14:06

fixing merge conflict

37564c8

removing minmaxdate statement from bootstrap.sh

30a5391

jpwhite4 reviewed Dec 14, 2018

View reviewed changes

plessbd reviewed Dec 14, 2018

View reviewed changes

eiffel777 added 2 commits December 17, 2018 09:26

Merge branch 'xdmod8.1' of https://github.com/ubccr/xdmod into add-cl…

3ba69fc

…oud-ingestion-to-ingestor-script

setting etl pipline to use for cloud shredders in constructor.

c1af695

plessbd reviewed Dec 17, 2018

View reviewed changes

classes/OpenXdmod/Shredder/Genericcloud.php Show resolved Hide resolved

plessbd reviewed Dec 17, 2018

View reviewed changes

changes to get tests to pass

45db850

plessbd reviewed Dec 20, 2018

View reviewed changes

configuration/etl/etl.d/jobs_cloud_eucalyptus.json Show resolved Hide resolved

update suggested by @plessbd

b60aa6b

plessbd reviewed Dec 20, 2018

View reviewed changes

plessbd approved these changes Dec 20, 2018

View reviewed changes

Greg Dean added 2 commits December 20, 2018 11:26

Merge branch 'xdmod8.1' into add-cloud-ingestion-to-ingestor-script

f3dc0ec

Merge branch 'xdmod8.1' into add-cloud-ingestion-to-ingestor-script

0eb33e0

eiffel777 merged commit 0febec4 into ubccr:xdmod8.1 Dec 20, 2018

jtpalmer mentioned this pull request Jan 28, 2019

Add storage shredder/ingestor support #786

Merged

6 tasks

eiffel777 added the new feature New functionality label Mar 28, 2019

smgallo changed the title ~~Add cloud data to xdmod-shredder and xdmod-ingestor~~ Add support for cloud data to xdmod-shredder and xdmod-ingestor Apr 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

eiffel777 commented Nov 30, 2018

jpwhite4 Dec 3, 2018

eiffel777 Dec 3, 2018

jpwhite4 Dec 14, 2018

plessbd Dec 14, 2018 •

edited

Loading

eiffel777 Dec 14, 2018

plessbd Dec 17, 2018

eiffel777 Dec 17, 2018

plessbd Dec 20, 2018 •

edited

Loading

plessbd Dec 20, 2018

	return (count($realms) > 0) ? true : false;
	return (count($realms) > 0);

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

Add support for cloud data to xdmod-shredder and xdmod-ingestor #739

Conversation

eiffel777 commented Nov 30, 2018

Motivation and Context

Tests performed

Types of changes

Checklist:

jpwhite4 Dec 3, 2018

Choose a reason for hiding this comment

eiffel777 Dec 3, 2018

Choose a reason for hiding this comment

jpwhite4 Dec 14, 2018

Choose a reason for hiding this comment

plessbd Dec 14, 2018 • edited Loading

Choose a reason for hiding this comment

eiffel777 Dec 14, 2018

Choose a reason for hiding this comment

plessbd Dec 17, 2018

Choose a reason for hiding this comment

eiffel777 Dec 17, 2018

Choose a reason for hiding this comment

plessbd Dec 20, 2018 • edited Loading

Choose a reason for hiding this comment

plessbd Dec 20, 2018

Choose a reason for hiding this comment

plessbd Dec 14, 2018 •

edited

Loading

plessbd Dec 20, 2018 •

edited

Loading