Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status #36564

Closed
wants to merge 53 commits into from
Closed

[SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status #36564

wants to merge 53 commits into from
+60 −20

Commits on May 16, 2022

  1. [SPARK-39195][SQL] Spark should use two step update of outputCommitCo… 

    …ordinator
    @AngersZhuuuu
    AngersZhuuuu committed May 16, 2022
    Configuration menu
    Copy the full SHA
    19135e0 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2022

  1. Update

    @AngersZhuuuu
    AngersZhuuuu committed May 17, 2022
    Configuration menu
    Copy the full SHA
    7062d32 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 18, 2022
    Configuration menu
    Copy the full SHA
    f4caa22 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2022

  1. Spark OutputCommitCoordinator should keep consistent

    @AngersZhuuuu
    AngersZhuuuu committed May 24, 2022
    Configuration menu
    Copy the full SHA
    5e6c0be View commit details
    Browse the repository at this point in the history

  2. Merge branch 'SPARK-39195' of https://github.com/AngersZhuuuu/spark i… 

    …nto SPARK-39195
    @AngersZhuuuu
    AngersZhuuuu committed May 24, 2022
    Configuration menu
    Copy the full SHA
    171bd6a View commit details
    Browse the repository at this point in the history

  3. revert

    @AngersZhuuuu
    AngersZhuuuu committed May 24, 2022
    Configuration menu
    Copy the full SHA
    b070c9d View commit details
    Browse the repository at this point in the history

  4. Update OutputCommitCoordinator.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 24, 2022
    Configuration menu
    Copy the full SHA
    796c08c View commit details
    Browse the repository at this point in the history

Commits on May 25, 2022

  1. Merge remote-tracking branch 'upstream/master' into SPARK-39195

    @AngersZhuuuu
    AngersZhuuuu committed May 25, 2022
    Configuration menu
    Copy the full SHA
    1d79aae View commit details
    Browse the repository at this point in the history

  2. Update SparkHadoopMapRedUtil.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 25, 2022
    Configuration menu
    Copy the full SHA
    5f5729b View commit details
    Browse the repository at this point in the history

  3. Update

    @AngersZhuuuu
    AngersZhuuuu committed May 25, 2022
    Configuration menu
    Copy the full SHA
    759814b View commit details
    Browse the repository at this point in the history

Commits on May 26, 2022

  1. trigger

    @AngersZhuuuu
    AngersZhuuuu committed May 26, 2022
    Configuration menu
    Copy the full SHA
    b4e11cc View commit details
    Browse the repository at this point in the history

Commits on May 27, 2022

  1. Update SparkHadoopMapRedUtil.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    7642cdb View commit details
    Browse the repository at this point in the history

  2. Revert "Update SparkHadoopMapRedUtil.scala" 

    This reverts commit 7642cdb.
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    e6dce26 View commit details
    Browse the repository at this point in the history

  3. Revert "Update" 

    This reverts commit 759814b.
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    6a403d7 View commit details
    Browse the repository at this point in the history

  4. Revert "Update SparkHadoopMapRedUtil.scala" 

    This reverts commit 5f5729b.
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    bc1214c View commit details
    Browse the repository at this point in the history

  5. Revert "Update OutputCommitCoordinator.scala" 

    This reverts commit 796c08c.
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    b5d2885 View commit details
    Browse the repository at this point in the history

  6. Revert "Spark OutputCommitCoordinator should keep consistent" 

    This reverts commit 5e6c0be.
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    b13dfbe View commit details
    Browse the repository at this point in the history

  7. [SPARK-39195][SQL] Spark should use two step update of outputCommitCo… 

    …ordinator
    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    ad67d0d View commit details
    Browse the repository at this point in the history

  8. Update

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    0e366a8 View commit details
    Browse the repository at this point in the history

  9. Update

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    11ba4b7 View commit details
    Browse the repository at this point in the history

  10. Update OutputCommitCoordinator.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    cc71ddc View commit details
    Browse the repository at this point in the history

  11. Update OutputCommitCoordinator.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    e7204df View commit details
    Browse the repository at this point in the history

  12. Update

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    9426f30 View commit details
    Browse the repository at this point in the history

  13. Update OutputCommitCoordinator.scala

    @AngersZhuuuu
    AngersZhuuuu committed May 27, 2022
    Configuration menu
    Copy the full SHA
    60e03f3 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2022

  1. follow comment

    @AngersZhuuuu
    AngersZhuuuu committed May 30, 2022
    Configuration menu
    Copy the full SHA
    f77c9c3 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2022

  1. Update SQLQuerySuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    c1faddd View commit details
    Browse the repository at this point in the history

  2. Update SQLQuerySuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    f7b92e1 View commit details
    Browse the repository at this point in the history

  3. re-trigger

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    ec5ef4b View commit details
    Browse the repository at this point in the history

  4. Update

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    58ea1a9 View commit details
    Browse the repository at this point in the history

  5. Update OutputCommitCoordinator.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    58b6c0f View commit details
    Browse the repository at this point in the history

  6. Update

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    beed831 View commit details
    Browse the repository at this point in the history

  7. Update SQLQuerySuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    87fadf0 View commit details
    Browse the repository at this point in the history

  8. Update DAGScheduler.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    46aa5e0 View commit details
    Browse the repository at this point in the history

  9. Update core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoo… 

    …rdinator.scala

Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
    @AngersZhuuuu @cloud-fan
    AngersZhuuuu and cloud-fan authored Jun 8, 2022
    Configuration menu
    Copy the full SHA
    3924127 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2022

  1. Follow comment

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    9c24d8e View commit details
    Browse the repository at this point in the history

  2. Update SQLQuerySuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    60beea1 View commit details
    Browse the repository at this point in the history

  3. Follow comment

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    c0bbcb8 View commit details
    Browse the repository at this point in the history

  4. Update SparkEnv.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    35b613a View commit details
    Browse the repository at this point in the history

  5. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    7bc078f View commit details
    Browse the repository at this point in the history

  6. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    b52a617 View commit details
    Browse the repository at this point in the history

  7. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    ce5ac35 View commit details
    Browse the repository at this point in the history

  8. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    7ca7ca3 View commit details
    Browse the repository at this point in the history

  9. Update OutputCommitCoordinatorIntegrationSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    79815ab View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2022

  1. trigegr

    @AngersZhuuuu
    AngersZhuuuu committed Jun 10, 2022
    Configuration menu
    Copy the full SHA
    c6c73e1 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2022

  1. update

    @AngersZhuuuu
    AngersZhuuuu committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    77f124c View commit details
    Browse the repository at this point in the history

  2. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    56543db View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2022

  1. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    b524a87 View commit details
    Browse the repository at this point in the history

  2. Update OutputCommitCoordinatorIntegrationSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    4a1a092 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into SPARK-39195

    @AngersZhuuuu
    AngersZhuuuu authored Jun 14, 2022
    Configuration menu
    Copy the full SHA
    cd404d3 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2022

  1. Update OutputCommitCoordinatorIntegrationSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    a6be796 View commit details
    Browse the repository at this point in the history

  2. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    d23bcbc View commit details
    Browse the repository at this point in the history

  3. Revert "Update OutputCommitCoordinatorSuite.scala" 

    This reverts commit d23bcbc.
    @AngersZhuuuu
    AngersZhuuuu committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    c29c9fb View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2022

  1. Update OutputCommitCoordinatorSuite.scala

    @AngersZhuuuu
    AngersZhuuuu committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    11d3ef2 View commit details
    Browse the repository at this point in the history