Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(glue-alpha): glue L2 alpha construct #30833

Open
wants to merge 75 commits into
base: main
Choose a base branch
from
Open

refactor(glue-alpha): glue L2 alpha construct #30833

wants to merge 75 commits into from
+14,764 −3,646

Commits on Feb 23, 2024

  1. Propagate initia glue l2 jobl refactor from natalie-white-aws/aws-cdk… 

    …-glue-l2 to mjanardhan/aws-cdk-glue-l2 to resolve non-trivial merge situation
    @natalie-white-aws
    natalie-white-aws committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    38bbbce View commit details
    Browse the repository at this point in the history

  2. Merge branch 'aws:main' into main

    @natalie-white-aws
    natalie-white-aws authored Feb 23, 2024
    Configuration menu
    Copy the full SHA
    5b9a16c View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. added Ray jobs

    @askarserikov
    askarserikov committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2632f78 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Create raw CDK assets for Python Shell Jobs

    @chrisw-devops
    chrisw-devops committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    c67cd81 View commit details
    Browse the repository at this point in the history

  2. Fixed broken path.join

    @chrisw-devops
    chrisw-devops committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    273d603 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Updated Python Shell Jobs integration test output to match updated in… 

    …tegration test.
    @chrisw-devops
    chrisw-devops committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    f66e0e0 View commit details
    Browse the repository at this point in the history

  2. Updated Python Shell Job unit tests to validate default values

    @chrisw-devops
    chrisw-devops committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    e844449 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. PySpark Streaming job

    @mjanardhan
    mjanardhan committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    8b367fa View commit details
    Browse the repository at this point in the history

  2. PySpark Streaming job

    @mjanardhan
    mjanardhan committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    d19f5cc View commit details
    Browse the repository at this point in the history

  3. Scala Spark Streaming Job class

    @mjanardhan
    mjanardhan committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    0809778 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. PySpark and Scala Flex ETL jobs L2 constructs

    @pras-b
    pras-b committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    6118d80 View commit details
    Browse the repository at this point in the history

  2. Modifications to comments

    @pras-b
    pras-b committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    6ed7021 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Scala Spark Streaming Job class

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    087f411 View commit details
    Browse the repository at this point in the history

  2. Streaming Jobs - integration tests

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    23a76a9 View commit details
    Browse the repository at this point in the history

  3. Streaming Jobs

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ac4b457 View commit details
    Browse the repository at this point in the history

  4. Streaming jobs CDK L2

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    9d936dc View commit details
    Browse the repository at this point in the history

  5. Python Streaming Jobs - Integration test updates

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    b28dc55 View commit details
    Browse the repository at this point in the history

  6. Scala Streaming Jobs - Integration test updates

    @mjanardhan
    mjanardhan committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    8afbb59 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Python Streaming Job updates

    @mjanardhan
    mjanardhan committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    ffca357 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. Modifications to scala & pyspark flex etl jobs based on PR review

    @pras-b
    pras-b committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    1da5519 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Modified the job definition to add logging and metrics. Added unit an… 

    …d integ tests.
    @askarserikov
    askarserikov committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1762b46 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Initial Commit for Workflow Triggers

    @dkovvuri
    dkovvuri committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    a234b6b View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. ETL Jobs and tests

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    3ff57ce View commit details
    Browse the repository at this point in the history

  2. ETL Jobs and tests

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    f9675bc View commit details
    Browse the repository at this point in the history

  3. ETL Jobs and tests

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    06c7d0a View commit details
    Browse the repository at this point in the history

  4. ETL Jobs and tests

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    9457e35 View commit details
    Browse the repository at this point in the history

  5. Update default python version

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    0c412cb View commit details
    Browse the repository at this point in the history

  6. ETL jobs & tests

    @mjanardhan
    mjanardhan committed May 28, 2024
    Configuration menu
    Copy the full SHA
    5413e85 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge pull request #6 from mjanardhan/etl-jobs 

    Etl jobs
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    121b21e View commit details
    Browse the repository at this point in the history

  2. Merge pull request #5 from dkovvuri/main 

    Workflow Triggers
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    72818b0 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #4 from mjanardhan/ray-jobs 

    Ray jobs
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    4d8a4f9 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #3 from mjanardhan/streaming-jobs 

    Streaming jobs
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    053ab32 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into python-shell-jobs

    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    c9b0ae9 View commit details
    Browse the repository at this point in the history

  6. Merge pull request #1 from mjanardhan/python-shell-jobs 

    Python shell jobs
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    2a26c5a View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into prasbal-spark-flex

    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    cfb13a5 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #2 from mjanardhan/prasbal-spark-flex 

    Spark ETL FLEX jobs - L2 Constructs
    @natalie-white-aws
    natalie-white-aws authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    f976c88 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Updates based on glue team feedback

    @mjanardhan
    mjanardhan committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5fd8569 View commit details
    Browse the repository at this point in the history

  2. Updated tests and results

    @mjanardhan
    mjanardhan committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c33b0b9 View commit details
    Browse the repository at this point in the history

  3. Rename flex jobs per naming convention

    @mjanardhan
    mjanardhan committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    029c52a View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Fix s3 path specified in --spark-event-logs-path and update glue version

    @mjanardhan
    mjanardhan committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    75a4e6e View commit details
    Browse the repository at this point in the history

  2. Updated default glue version to v2

    @mjanardhan
    mjanardhan committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1cf6d29 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #7 from mjanardhan/glue_job_updates 

    Glue job updates
    @natalie-white-aws
    natalie-white-aws authored Jun 26, 2024
    Configuration menu
    Copy the full SHA
    28f3b27 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Delete Job Legacy classes, change default WorkerType back to G_1X

    @natalie-white-aws
    natalie-white-aws committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    d24c24a View commit details
    Browse the repository at this point in the history

  2. Run tests and snapshot output

    @mjanardhan
    mjanardhan committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    d1f3dfc View commit details
    Browse the repository at this point in the history

  3. Merge pull request #8 from mjanardhan/run_tests 

    Run tests and snapshot output
    @mjanardhan
    mjanardhan authored Jun 27, 2024
    Configuration menu
    Copy the full SHA
    65237a1 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. Refactor coninuous logging default enabled plus unit tests in pyspark… 

    …-etl-job and fixed other unit tests
    @natalie-white-aws
    natalie-white-aws committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    593e877 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #9 from mjanardhan/refactorLogging 

    Refactor coninuous logging default enabled plus unit tests in pyspark…
    @natalie-white-aws
    natalie-white-aws authored Jun 29, 2024
    Configuration menu
    Copy the full SHA
    081c94b View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Merging upstream CDK Core commits

    @natalie-white-aws
    natalie-white-aws committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7c629c2 View commit details
    Browse the repository at this point in the history

  2. Delete legacy Glue Job classes and tests post-merge

    @natalie-white-aws
    natalie-white-aws committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    a75ac69 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #10 from mjanardhan/pull-from-cdk-main 

    Pull from cdk main
    @mjanardhan
    mjanardhan authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    cea1cef View commit details
    Browse the repository at this point in the history

  4. Updated snapshots for the jobs integ tests

    @mjanardhan
    mjanardhan committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    a17a4df View commit details
    Browse the repository at this point in the history

  5. Merge pull request #11 from mjanardhan/update_snapshot 

    Updated snapshots for the jobs integration tests
    @natalie-white-aws
    natalie-white-aws authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    b30055e View commit details
    Browse the repository at this point in the history

  6. Final README update

    @natalie-white-aws
    natalie-white-aws committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    28c97fd View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Resolve README linter issues

    @natalie-white-aws
    natalie-white-aws committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    316c670 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Increase unit test coverage, especially for pyspark etl jobs and ray … 

    …jobs
    @natalie-white-aws
    natalie-white-aws committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6343ad2 View commit details
    Browse the repository at this point in the history

  2. Increase unit test coverage for python shell and pyspark streaming jobs

    @natalie-white-aws
    natalie-white-aws committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f4b2315 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Added unit test cases to the scala jobs

    @pras-b
    pras-b committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    7a74777 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #13 from mjanardhan/scala-unit-test 

    Added Scala jobs unit tests
    @natalie-white-aws
    natalie-white-aws authored Jul 18, 2024
    Configuration menu
    Copy the full SHA
    e051e05 View commit details
    Browse the repository at this point in the history

  3. fixed the unit test for scala etl and flex etl

    @pras-b
    pras-b committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    b601916 View commit details
    Browse the repository at this point in the history

  4. Fixing scala etl and pyspark streaming unit tests

    @natalie-white-aws
    natalie-white-aws committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    cdba008 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Merge pull request #14 from mjanardhan/scala-unit-test 

    fixed the unit test for scala etl and flex etl
    @mjanardhan
    mjanardhan authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    df4b1d0 View commit details
    Browse the repository at this point in the history

  2. WorkerType and numberofWorkers defaults are enforced when not set

    @mjanardhan
    mjanardhan committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    3943206 View commit details
    Browse the repository at this point in the history

  3. Fix tests

    @mjanardhan
    mjanardhan committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c077b11 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #15 from mjanardhan/fix-ray-integ-tests 

    WorkerType and numberofWorkers defaults are enforced when not set
    @natalie-white-aws
    natalie-white-aws authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    79339c4 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. Updated snapshots

    @mjanardhan
    mjanardhan committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    dbbaabc View commit details
    Browse the repository at this point in the history

  2. Merge pull request #16 from mjanardhan/integtests_snapshots 

    Integ tests snapshots
    @mjanardhan
    mjanardhan authored Jul 24, 2024
    Configuration menu
    Copy the full SHA
    cffa1b7 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Resolve glue alpha README compilation errors

    @natalie-white-aws
    natalie-white-aws committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7077f98 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'glue-alpha-refacor' of https://github.com/mjanardhan/aw… 

    …s-cdk-glue-l2 into glue-alpha-refacor
    @natalie-white-aws
    natalie-white-aws committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    f2b4a97 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Resolve additional README compilation issues

    @natalie-white-aws
    natalie-white-aws committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9f1edd6 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Remove code examples from README and reference unit tests for examples

    @natalie-white-aws
    natalie-white-aws committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    f4886a7 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Merge branch 'main' into glue-alpha-refacor

    @natalie-white-aws
    natalie-white-aws authored Aug 7, 2024
    Configuration menu
    Copy the full SHA
    1c31609 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge branch 'main' into glue-alpha-refacor

    @natalie-white-aws
    natalie-white-aws authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    84c6c55 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Updates based on feedback from @GavinZZ and Paul Sun

    @natalie-white-aws
    natalie-white-aws committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    0d8c5eb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into glue-alpha-refacor

    @natalie-white-aws
    natalie-white-aws authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    961a812 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'main' into glue-alpha-refacor

    @natalie-white-aws
    natalie-white-aws authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    a54574c View commit details
    Browse the repository at this point in the history