Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-65: Add DAG versioning support #42913

Merged
merged 49 commits into from
Nov 5, 2024
Merged

AIP-65: Add DAG versioning support #42913

merged 49 commits into from
Nov 5, 2024

Commits on Nov 5, 2024

  1. AIP-65: Add DAG versioning support 

    This commit introduces versioning for DAGs

Changes:
- Introduced DagVersion model to handle versioning of DAGs.
- Added version_name field to DAG for use in tracking the dagversion by users
- Added support for version retrieval in the get_dag_source API endpoint
- Modified DAG execution logic to reference dag_version_id instead of the
dag_hash to ensure DAG runs are linked to specific versions.

Fix tests

revert RESTAPI changes
    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a3275a7 View commit details
    Browse the repository at this point in the history

  2. fixup! AIP-65: Add DAG versioning support

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ae5b6da View commit details
    Browse the repository at this point in the history

  3. fixup! fixup! AIP-65: Add DAG versioning support

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ed89ca6 View commit details
    Browse the repository at this point in the history

  4. fix migration

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ef506f9 View commit details
    Browse the repository at this point in the history

  5. fix test

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3821dbb View commit details
    Browse the repository at this point in the history

  6. more test fixes

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    683bbd7 View commit details
    Browse the repository at this point in the history

  7. update query count

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    fe655de View commit details
    Browse the repository at this point in the history

  8. fix static checks

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ff9be22 View commit details
    Browse the repository at this point in the history

  9. Fix query and add created_at to dag_version table

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    7f50a3c View commit details
    Browse the repository at this point in the history

  10. improve code

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    57bd887 View commit details
    Browse the repository at this point in the history

  11. Change to using UUID for primary keys

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    e2bdad3 View commit details
    Browse the repository at this point in the history

  12. DagCode.bulk_write_code is no longer used

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a4e3c78 View commit details
    Browse the repository at this point in the history

  13. fixup! Change to using UUID for primary keys

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    e994074 View commit details
    Browse the repository at this point in the history

  14. fix tests

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    e06b752 View commit details
    Browse the repository at this point in the history

  15. fixup! fix tests

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    0bef933 View commit details
    Browse the repository at this point in the history

  16. use uuid for version_name

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    d2bd271 View commit details
    Browse the repository at this point in the history

  17. fixup! use uuid for version_name

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    99abb36 View commit details
    Browse the repository at this point in the history

  18. use row lock when writing dag version

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    23204b8 View commit details
    Browse the repository at this point in the history

  19. use row lock when writing dag version

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    2525902 View commit details
    Browse the repository at this point in the history

  20. fixup! use row lock when writing dag version

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    610d3b1 View commit details
    Browse the repository at this point in the history

  21. deactivating dag should not remove serialized dags

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    2cf03f5 View commit details
    Browse the repository at this point in the history

  22. save version_name as string not uuid

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ad91949 View commit details
    Browse the repository at this point in the history

  23. Make dag_version_id unique

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    11de3a2 View commit details
    Browse the repository at this point in the history

  24. fixup! Make dag_version_id unique

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    108c80c View commit details
    Browse the repository at this point in the history

  25. Fix tests

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    53a4837 View commit details
    Browse the repository at this point in the history

  26. Use uuid7

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    554dd15 View commit details
    Browse the repository at this point in the history

  27. fix test

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    72f2cc0 View commit details
    Browse the repository at this point in the history

  28. fixup! fix test

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    903ed3c View commit details
    Browse the repository at this point in the history

  29. use binary=False for uuid field to fix sqlite issue

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    db44170 View commit details
    Browse the repository at this point in the history

  30. apply suggestions from code review

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    c4ed8af View commit details
    Browse the repository at this point in the history

  31. Remove unnecessary version_name on dagmodel

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4f011d2 View commit details
    Browse the repository at this point in the history

  32. Fix sqlalchemy 2 warning

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4bf8b1f View commit details
    Browse the repository at this point in the history

  33. Fix conflicts

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    0e0c127 View commit details
    Browse the repository at this point in the history

  34. Apply suggestions from code review 

    Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
    @ephraimbuddy @jedcunningham
    ephraimbuddy and jedcunningham committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    e410f2c View commit details
    Browse the repository at this point in the history

  35. fixup! Apply suggestions from code review

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    76982f3 View commit details
    Browse the repository at this point in the history

  36. fixup! fixup! Apply suggestions from code review

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    c4171cb View commit details
    Browse the repository at this point in the history

  37. add test for dagversion model and make version_name, number and dag_i… 

    …d unique
    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    dad0f36 View commit details
    Browse the repository at this point in the history

  38. Remove commented test as serdag can no longer disappear

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    13777e7 View commit details
    Browse the repository at this point in the history

  39. Add SQLAlchemy-utils to requirements

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    9f96cb0 View commit details
    Browse the repository at this point in the history

  40. mark test_dag_version.py as db_test

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    b2c4844 View commit details
    Browse the repository at this point in the history

  41. make version_name nullable

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    611a5cb View commit details
    Browse the repository at this point in the history

  42. Apply suggestions from code review

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    cc41fb6 View commit details
    Browse the repository at this point in the history

  43. fixup! Apply suggestions from code review

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    621a407 View commit details
    Browse the repository at this point in the history

  44. remove file_updater

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    108965b View commit details
    Browse the repository at this point in the history

  45. Use dag_version for creating dagruns instead of dag_version_id

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a51585e View commit details
    Browse the repository at this point in the history

  46. fix conflicts

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    01f3c58 View commit details
    Browse the repository at this point in the history

  47. use if TYPE_CHECKING

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    11de1e9 View commit details
    Browse the repository at this point in the history

  48. Add docstrings to methods

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    569eb53 View commit details
    Browse the repository at this point in the history

  49. Move getting latest serdags to SerializedDagModel

    @ephraimbuddy
    ephraimbuddy committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a7bba02 View commit details
    Browse the repository at this point in the history