Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote Shuffle Implementation for Spark 3.3.1 #77

Open
wants to merge 28 commits into
base: branch-1.1-spark-3.1.1
Choose a base branch
from
Open

Remote Shuffle Implementation for Spark 3.3.1 #77

wants to merge 28 commits into from

Commits on Jun 11, 2021

  1. Issue 18 (oap-project#23)upgrade to Spark-3.1.1 

    [REMOTE-SHUFFLE-18] upgrade to Spark-3.1.1
    @jiafuzha
    jiafuzha authored Jun 11, 2021
    Configuration menu
    Copy the full SHA
    492eefc View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2021

  1. [REMOTE-SHUFFLE-24] Enhance executor memory release (oap-project#25) 

    * [REMOTE-SHUFFLE-24] Enhance executor memory release

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Jun 15, 2021
    Configuration menu
    Copy the full SHA
    85d0698 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2021

  1. [REMOTE-SHUFFLE-26] Follow OAP name convention (oap-project#27) 

    [REMOTE-SHUFFLE-26] Follow OAP name convention
    @jiafuzha
    jiafuzha authored Jun 17, 2021
    Configuration menu
    Copy the full SHA
    06fea5e View commit details
    Browse the repository at this point in the history

  2. [REMOTE-SHUFFLE-28] Follow OAP name convention ${project.artifactId}-… 

    …${project.version}-with-spark-${spark.version} (oap-project#29)

[REMOTE-SHUFFLE-28] Follow OAP name convention ${project.artifactId}-${project.version}-with-spark-${spark.version}
    @jiafuzha
    jiafuzha authored Jun 17, 2021
    Configuration menu
    Copy the full SHA
    1fde89d View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2021

  1. [REMOTE-SHUFFLE-31] Avoid re-computation of shuffle write when execut… 

    …or lost during shuffle read
    @jiafuzha
    jiafuzha committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    9621d70 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2021

  1. [REMOTE-SHUFFLE-35] Update version to 1.2.0 (oap-project#36)

    @jiafuzha
    jiafuzha authored Jul 9, 2021
    Configuration menu
    Copy the full SHA
    9edf922 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2021

  1. [REMOTE-SHUFFLE-40] Update Changelog and OAP guide

    @zhixingheyi-tian
    HongW2019 authored and zhixingheyi-tian committed Sep 10, 2021
    Configuration menu
    Copy the full SHA
    3dc0f2d View commit details
    Browse the repository at this point in the history

  2. [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering (oap-project#38 

    )

* reconstruct project and add new shuffle-daos plugin

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* corrected location of scalastyle-config

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* use daos-java version of 1.1.4

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-34] Enhance map-combine and map-ordering

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Sep 10, 2021
    Configuration menu
    Copy the full SHA
    f0b6558 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. [REMOTE-SHUFFLE-44] Set mapIndex to -1 when FetchFailedException to a… 

    …void clear map output (oap-project#45)

* reconstruct project and add new shuffle-daos plugin

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* corrected location of scalastyle-config

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* use daos-java version of 1.1.4

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-44] Set mapIndex to -1 when FetchFailedException to avoid clear map output

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    169929c View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2021

  1. [REMOTE-SHUFFLE-46] Fixed some issues when run with I/O synchronously (… 

    …oap-project#47)

* reconstruct project and add new shuffle-daos plugin

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* corrected location of scalastyle-config

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* use daos-java version of 1.1.4

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-46] Fixed some issues when run with I/O synchronously

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-46] Fixed some issues when run with I/O synchronously

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-46] Fixed some issues when run with I/O synchronously

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-46] Fixed some issues when run with I/O synchronously

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Nov 2, 2021
    Configuration menu
    Copy the full SHA
    50427b8 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2021

  1. [REMOTE-SHUFFLE-50] Support Spark-3.1 and Spark-3.2 in shuffle-daos (o… 

    …ap-project#51)

[REMOTE-SHUFFLE-50] Support Spark-3.1 and Spark-3.2 in shuffle-daos

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Dec 7, 2021
    Configuration menu
    Copy the full SHA
    3c1ae1d View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2021

  1. [REMOTE-SHUFFLE-43] Read Partitions from Highly Compressed Map Status (… 

    …oap-project#49)
    @jiafuzha
    jiafuzha authored Dec 14, 2021
    Configuration menu
    Copy the full SHA
    55f143b View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2021

  1. Create SECURITY.md

    @carsonwang
    carsonwang authored Dec 15, 2021
    Configuration menu
    Copy the full SHA
    3f9ef69 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2021

  1. [REMOTE-SHUFFLE-53] Make spark dependency provided (oap-project#54)

    @jiafuzha
    jiafuzha authored Dec 20, 2021
    Configuration menu
    Copy the full SHA
    350067d View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2022

  1. [REMOTE-SHUFFLE-52] Update shuffle write to not encode/decode paramet… 

    …ers (oap-project#55)

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Jan 7, 2022
    Configuration menu
    Copy the full SHA
    a1fddc1 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2022

  1. [REMOTE-SHUFFLE-56] Make shuffle object class and object hint configu… 

    …rable (oap-project#59)

* [REMOTE-SHUFFLE-56] Make shuffle object class and object hint configurable

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-56] Make shuffle object class and object hint configurable

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-56] Make shuffle object class and object hint configurable

Signed-off-by: Jifu Zhang <jiafu.zhang@intel.com>

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
Signed-off-by: Jifu Zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Nov 4, 2022
    Configuration menu
    Copy the full SHA
    acb81db View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2022

  1. [REMOTE-SHUFFLE-60] Support akey offset size bigger than 2GB (oap-pro… 

    …ject#61)

Signed-off-by: Jifu Zhang <jiafu.zhang@intel.com>

Signed-off-by: Jifu Zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Nov 7, 2022
    Configuration menu
    Copy the full SHA
    abc0063 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2022

  1. [REMOTE-SHUFFLE-62] Adjust some default parameters for aurora env (oa… 

    …p-project#63)

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Dec 16, 2022
    Configuration menu
    Copy the full SHA
    57b2a3f View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2023

  1. [REMOTE-SHUFFLE-64] remove password from url in shuffle hadoop script (… 

    …oap-project#65)

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Jan 5, 2023
    Configuration menu
    Copy the full SHA
    006a05b View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2023

  1. [REMOTE-SHUFFLE-66] Enhance shuffle removing logic (oap-project#67) 

    * [REMOTE-SHUFFLE-66] Enhance shuffle removing logic

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

* [REMOTE-SHUFFLE-66] Enhance shuffle removing logic

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>

---------

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Feb 21, 2023
    Configuration menu
    Copy the full SHA
    50b79bc View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. [REMOTE-SHUFFLE-66] Enhance shuffle removing logic (oap-project#69) 

    Signed-off-by: Jiafu Zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Apr 26, 2023
    Configuration menu
    Copy the full SHA
    4b33318 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. [REMOTE-SHUFFLE-70] use DAOS object class hint instead of DAOS object… 

    … class (oap-project#71)

Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha authored Jun 5, 2023
    Configuration menu
    Copy the full SHA
    38e3822 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. [REMOTE-SHUFFLE-101] fixed UT errors 

    Signed-off-by: jiafu zhang <jiafu.zhang@intel.com>
    @jiafuzha
    jiafuzha committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    eb785b3 View commit details
    Browse the repository at this point in the history

  2. Merge pull request oap-project#72 from jiafuzha/ISSUE_101 

    [REMOTE-SHUFFLE-101] fixed UT errors
    @carsonwang
    carsonwang authored Jun 7, 2023
    Configuration menu
    Copy the full SHA
    d957ef0 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Bump org.apache.spark:spark-core_2.12 from 3.1.1 to 3.3.3 (oap-projec… 

    …t#74)

Bumps org.apache.spark:spark-core_2.12 from 3.1.1 to 3.3.3.

---
updated-dependencies:
- dependency-name: org.apache.spark:spark-core_2.12
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    14cc7ee View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. fix OpenSSF

    @minmingzhu
    minmingzhu committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    4127802 View commit details
    Browse the repository at this point in the history

  2. Update dev_cron.yml

    @minmingzhu
    minmingzhu authored Feb 28, 2024
    Configuration menu
    Copy the full SHA
    93f7463 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. Merge pull request oap-project#76 from minmingzhu/fix_openssf 

    [REMOTE-SUFFULE 75] fix OpenSSF issues
    @carsonwang
    carsonwang authored Feb 29, 2024
    Configuration menu
    Copy the full SHA
    28e1b35 View commit details
    Browse the repository at this point in the history