Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skeleton code structure for Glue L2 construct #26688

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/conditionaltrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Conditional Trigger Class
*
* Conditional triggers have a predicate and actions associated with them.
* When the predicateCondition is true, the trigger actions will be executed.
*
*/
4 changes: 4 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/constants.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
/**
* Enums used by Job Types/Triggers
*
*/
13 changes: 13 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/flexjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/**
* Flex Jobs class
*
* Flex jobs supports Python and Scala language.
* The flexible execution class is appropriate for non-urgent jobs such as
* pre-production jobs, testing, and one-time data loads.
* Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or
* G.2X worker types but will default to the latest version of Glue (currently Glue 3.0.)
*
* Similar to ETL, we’ll enable these features: —enable-metrics, —enable-spark-ui,
* —enable-continuous-cloudwatch-log
*
*/
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/notifyeventtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/**
* Notify Event Trigger Class
*
* Workflows are mandatory for this trigger type. There are two types of notify event triggers,
* batching and non-batching trigger.
* For batching triggers, developers must specify BatchSize but for non-batching BatchSize will
* be set to 1.
* For both triggers, BatchWindow will be default to 900 seconds.
*
*/
8 changes: 8 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/ondemandtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/**
* On Demand Trigger Class
*
* On demand triggers can start glue jobs or crawlers.
* The trigger method will take an optional description but abstract the requirement of an
* actions list using the job or crawler objects using conditional types.
*
*/
8 changes: 8 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/pythonshelljob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/**
* Python Shell Jobs class
*
* A Python shell job runs Python scripts as a shell and supports a Python version that
* depends on the AWS Glue version you are using.
* This can be used to schedule and run tasks that don't require an Apache Spark environment.
*
*/
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/rayjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Ray Jobs class
*
* Glue ray only supports worker type Z.2X and Glue version 4.0.
* Runtime will default to Ray2.3 and min workers will default to 3.
*
*/
12 changes: 12 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/scheduledtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/**
* Scheduled Trigger Base Class
*
* Schedule triggers are a way for developers to create jobs using cron expressions.
* We’ll provide daily, weekly, and monthly convenience functions, as well as a custom function
* that will allow developers to create their own custom timing using the existing
* event Schedule object without having to build their own cron expressions.
*
* The trigger method will take an optional description and list of Actions
* which can refer to Jobs or crawlers via conditional types.
*
*/
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/sparketljob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/**
* Spark ETL Jobs class
*
* ETL jobs supports Python and Scala language.
* ETL job type supports G1, G2, G4 and G8 worker type default as G2, which customer can override.
* It wil default to the best practice version of ETL 4.0, but allow developers to override to 3.0.
* We will also default to best practice enablement the following ETL features:
* —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.
*
*/
13 changes: 13 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/streamingjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/**
* Streaming Jobs class
*
* A Streaming job is similar to an ETL job, except that it performs ETL on data streams
* using the Apache Spark Structured Streaming framework.
* These jobs will default to use Python 3.9.
*
* Similar to ETL jobs, streaming job supports Scala and Python languages. Similar to ETL,
* it supports G1 and G2 worker type and 2.0, 3.0 and 4.0 version. We’ll default to G2 worker
* and 4.0 version for streaming jobs which developers can override.
* We will enable —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.
*
*/
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/trigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Workflow Trigger Base Class
*
* In AWS Glue, developers can use workflows to create and visualize complex extract,
* transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers.
*
*/
Loading