Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate external transform wrappers using a script #29834

Merged
merged 72 commits into from
Feb 22, 2024
Merged

Generate external transform wrappers using a script #29834

merged 72 commits into from
Feb 22, 2024

Commits on Dec 13, 2023

  1. checkpoint

    @ahmedabu98
    ahmedabu98 committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    f80f1ac View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    2517445 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. gen_xlang_wrappers workflow

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    e5bf704 View commit details
    Browse the repository at this point in the history

  2. add tests; integrate with setup.py

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    9295861 View commit details
    Browse the repository at this point in the history

  3. remove duplicate changes; adjust transform positions in standard serv… 

    …ice config
    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    1ec6c11 View commit details
    Browse the repository at this point in the history

  4. undo deleted line

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    f9bbf42 View commit details
    Browse the repository at this point in the history

  5. add more config modifications

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    6a2d3d2 View commit details
    Browse the repository at this point in the history

  6. warn when generation script not found

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    23450fd View commit details
    Browse the repository at this point in the history

  7. add jinja2 dependency; include script in MANIFEST.in; create a xlang … 

    …IO direct GHA workflow
    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    01dbc63 View commit details
    Browse the repository at this point in the history

  8. add MarkupSafe==2.1.3 because jinja2 needs it

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    0bc4ff4 View commit details
    Browse the repository at this point in the history

  9. lints and fixes

    @ahmedabu98
    ahmedabu98 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    390d8f8 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2023

  1. jinja template abides more by lint/format rules in case yapf doesn't … 

    …exist
    @ahmedabu98
    ahmedabu98 committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    45a71ea View commit details
    Browse the repository at this point in the history

  2. lint and fixes

    @ahmedabu98
    ahmedabu98 committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    8847562 View commit details
    Browse the repository at this point in the history

  3. no yapf; use random dir name for tests

    @ahmedabu98
    ahmedabu98 committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    18a157d View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2023

  1. lint

    @ahmedabu98
    ahmedabu98 committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    c64d75b View commit details
    Browse the repository at this point in the history

  2. lint

    @ahmedabu98
    ahmedabu98 committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    7049f80 View commit details
    Browse the repository at this point in the history

  3. format fix

    @ahmedabu98
    ahmedabu98 committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    ddd5b2f View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. address comments

    @ahmedabu98
    ahmedabu98 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    91fe758 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. commit the transform config yaml file; in setup, only generate from t… 

    …his config file; create postcommit test that checks if this config file is in sync
    @ahmedabu98
    ahmedabu98 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    29d50ac View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    fc739fd View commit details
    Browse the repository at this point in the history

  3. format generated files

    @ahmedabu98
    ahmedabu98 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    3f843dd View commit details
    Browse the repository at this point in the history

  4. add yapf dep

    @ahmedabu98
    ahmedabu98 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    4347cd4 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. template fixes

    @ahmedabu98
    ahmedabu98 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    8093519 View commit details
    Browse the repository at this point in the history

  2. set default inside init

    @ahmedabu98
    ahmedabu98 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    1c8a17f View commit details
    Browse the repository at this point in the history

  3. yapf best effort

    @ahmedabu98
    ahmedabu98 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    f33e3f8 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2024

  1. add support for test suites to spin up multiple services; add PreComm… 

    …it test that checks if standard_external_transform config is in-sync; add a gradle command that builds relevant expansion jars and runs the script
    @ahmedabu98
    ahmedabu98 committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    da03ff3 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    c356ba8 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. address comments

    @ahmedabu98
    ahmedabu98 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    2690d0a View commit details
    Browse the repository at this point in the history

  2. try catch imports

    @ahmedabu98
    ahmedabu98 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    0b6827a View commit details
    Browse the repository at this point in the history

  3. camel_case_to_snake_case in provider module

    @ahmedabu98
    ahmedabu98 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    cf04b3d View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    a20e2a3 View commit details
    Browse the repository at this point in the history

  5. lint

    @ahmedabu98
    ahmedabu98 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    2bd52e8 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2024

  1. skip unit tests if importing out of apache_beam

    @ahmedabu98
    ahmedabu98 committed Jan 27, 2024
    Configuration menu
    Copy the full SHA
    cc60dfd View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2024

  1. generate wrappers in transforms/_external_transforms.py subdir and im… 

    …port into packages from there; update tests and documentation
    @ahmedabu98
    ahmedabu98 committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    6672d61 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. add _external_transforms package

    @ahmedabu98
    ahmedabu98 committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    7d81370 View commit details
    Browse the repository at this point in the history

  2. lint

    @ahmedabu98
    ahmedabu98 committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    1c8cafc View commit details
    Browse the repository at this point in the history

  3. fix precommit workflow name

    @ahmedabu98
    ahmedabu98 committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    fe02266 View commit details
    Browse the repository at this point in the history

  4. add workflow_dispatch option

    @ahmedabu98
    ahmedabu98 committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    37d844e View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. address comments; add description to CHANGES.md

    @ahmedabu98
    ahmedabu98 committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    dcc37fa View commit details
    Browse the repository at this point in the history

  2. resolve branch conflict

    @ahmedabu98
    ahmedabu98 committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    346346f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    3da5f1f View commit details
    Browse the repository at this point in the history

  4. python deps

    @ahmedabu98
    ahmedabu98 committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    d1f103e View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. manually add greenlet dep

    @ahmedabu98
    ahmedabu98 committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    92664fb View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2024

  1. address comments

    @ahmedabu98
    ahmedabu98 committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    11fbd99 View commit details
    Browse the repository at this point in the history

  2. add xlang __init__

    @ahmedabu98
    ahmedabu98 committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    51c2086 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. lint

    @ahmedabu98
    ahmedabu98 committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    19358f0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    128fdbc View commit details
    Browse the repository at this point in the history

  3. remove jinja from test dependencies; allow option to pass in extra de… 

    …pendencies to xlang tests
    @ahmedabu98
    ahmedabu98 committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    1c32d2a View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. address comments; always generate wrappers

    @ahmedabu98
    ahmedabu98 committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    9cb5a98 View commit details
    Browse the repository at this point in the history

  2. add greenlet back

    @ahmedabu98
    ahmedabu98 committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    a20154d View commit details
    Browse the repository at this point in the history

  3. better link to docs

    @ahmedabu98
    ahmedabu98 committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    f28b053 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. don't add unneeded io direct test suite

    @ahmedabu98
    ahmedabu98 committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    da8d51d View commit details
    Browse the repository at this point in the history

  2. add bounds to jinja2 install

    @ahmedabu98
    ahmedabu98 committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    538b40a View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    27b7f6a View commit details
    Browse the repository at this point in the history

  2. try pass import; command only generates config; install pyyaml before… 

    … running script
    @ahmedabu98
    ahmedabu98 committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    889b17c View commit details
    Browse the repository at this point in the history

  3. lint

    @ahmedabu98
    ahmedabu98 committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    5dd9477 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. let python automatically start up expansion services instead of manua… 

    …lly starting each one beforehand; skip existing transforms from gcp service config; cleanup precommit test config
    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    9568de5 View commit details
    Browse the repository at this point in the history

  2. remove merge conflict

    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    1f49d75 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into gen_wrappers_script

    @ahmedabu98
    ahmedabu98 authored Feb 15, 2024
    Configuration menu
    Copy the full SHA
    bdf290f View commit details
    Browse the repository at this point in the history

  4. touch postcommit files to trigger GHA

    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c96ade5 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'gen_wrappers_script' of https://github.com/ahmedabu98/beam 

     into gen_wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    082295d View commit details
    Browse the repository at this point in the history

  6. rename to external_provider_it_test.py to avoid running on unit test … 

    …suites, where expansion service wouldn't be built
    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    28146bd View commit details
    Browse the repository at this point in the history

  7. rename tests ..Test -> ..IT

    @ahmedabu98
    ahmedabu98 committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    5f9a227 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. small adjustments to pass python unit tests: import script from file … 

    …path, use simple function to return beam jar exp service
    @ahmedabu98
    ahmedabu98 committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    06d5fd5 View commit details
    Browse the repository at this point in the history

  2. run tests only when expansion jars are built

    @ahmedabu98
    ahmedabu98 committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    e42b3fe View commit details
    Browse the repository at this point in the history

  3. load the script after importing

    @ahmedabu98
    ahmedabu98 committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    0d11059 View commit details
    Browse the repository at this point in the history

  4. skip test if jars not built

    @ahmedabu98
    ahmedabu98 committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    c1b29f3 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2024

  1. Merge branch 'master' of https://github.com/ahmedabu98/beam into gen_… 

    …wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    620344c View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. Merge branch 'master' into gen_wrappers_script

    @ahmedabu98
    ahmedabu98 authored Feb 21, 2024
    Configuration menu
    Copy the full SHA
    8fdb02a View commit details
    Browse the repository at this point in the history

  2. touch postcommit files to trigger GHA

    @ahmedabu98
    ahmedabu98 committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    35da8b9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'gen_wrappers_script' of https://github.com/ahmedabu98/beam 

     into gen_wrappers_script
    @ahmedabu98
    ahmedabu98 committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    bc43578 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. correct command name (generateExternalTransformsConfig)

    @ahmedabu98
    ahmedabu98 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    89e139e View commit details
    Browse the repository at this point in the history