Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-65: Track the serialized DAG across DagRun & TaskInstance #42690

Closed
wants to merge 12 commits into from
Closed

AIP-65: Track the serialized DAG across DagRun & TaskInstance #42690

wants to merge 12 commits into from

Commits on Oct 4, 2024

  1. AIP-65: Add versioning to the SerializedDagModel 

    This commit adds versioning to the serializedDagModel.

Changes:
Added new columns, id, and version_number to the SDM and made id the
primary key.

Updated the write_dag method of the SDM to add the SDs correctly.

Updated the queries so the scheduler/webserver runs with the latest SDM

The version_number was added to help us track the evolution of a DAG.
Suppose a DAG with dag_hash AB is changed, and the dag_hash becomes CD.
If the change is reverted, we will have a dag_hash of AB again. In this
case, the version_number would still increment, letting us know that the
DAG was changed three times. I feel it's a meaningful way to track the changes,
independent of the id column, which is database internals.
    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    69e600d View commit details
    Browse the repository at this point in the history

  2. fix rebase

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    26747d2 View commit details
    Browse the repository at this point in the history

  3. Fix migration

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    a9ec801 View commit details
    Browse the repository at this point in the history

  4. fixup! Fix migration

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    2bebaa9 View commit details
    Browse the repository at this point in the history

  5. add tests

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    fac8475 View commit details
    Browse the repository at this point in the history

  6. fixup! add tests

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    4b6bc79 View commit details
    Browse the repository at this point in the history

  7. Revert change in example dag

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    eb02bd0 View commit details
    Browse the repository at this point in the history

  8. a little refactor

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    b7968e8 View commit details
    Browse the repository at this point in the history

  9. fixup! a little refactor

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    bf37518 View commit details
    Browse the repository at this point in the history

  10. AIP-65: Track the serialized DAG across DagRun & TaskInstance 

    This helps to track the serialized DAG version the task instance ran
with by establishing a relationship between the entities instead of using
the dag_hash.
    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    c587d3a View commit details
    Browse the repository at this point in the history

  11. Use lazy='select' since we want just the dag_hash

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    177aa86 View commit details
    Browse the repository at this point in the history

  12. Make a direct query for dag_hash

    @ephraimbuddy
    ephraimbuddy committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9c5ea0d View commit details
    Browse the repository at this point in the history