v0.10.0a1

[qinxuye]

This is the release notes of v0.10.0a1. See here for the complete list of solved issues and merged PRs.

New Features

• Oscar
  • Stop importing main module when starting Mars local cluster (Stop importing main module when starting Mars local cluster #3110)
• Tensor
  • Integrate special error functions (Integrate special error functions #3060)
  • Integrate part of scipy elliptic functions and integrals (Integrate part of scipy elliptic functions and integrals #3111)
• DataFrame
  • Support sort=True for Groupby (Support sort=True for Groupby #2959, thanks @sak2002!)

Enhancements

• Disable bloom filter in merge for now (Disable bloom filter in merge for now #2967)
• [Ray] Implement ray task executor progress ([Ray] Implement ray task executor progress #3008)
• Dump remote tracebacks to make local ones more friendly (Dump remote tracebacks to make local ones more friendly #3028)
• Use tell when remove mapper data after execution (Use tell when remove mapper data after execution #3027)
• Optimize import speed for Mars package (Optimize import speed for Mars package #3022)
• Do not aggressively choose tree method in tile of groupby for distributed setting (Do not aggressively choose tree method in tile of groupby for distributed setting #3032)
• [Ray] Implements get_chunks_result for Ray execution context ([Ray] Implements get_chunks_result for Ray execution context #3023)
• Refine ThreadedServiceContext.get_chunks_meta usage (Refine ThreadedServiceContext.get_chunks_meta usage #3037)
• Shuffle both sides at the same time for md.merge (Shuffle both sides at the same time for md.merge #3041)
• Assign reducer ops in task assigner to make them more balanced across cluster (Assign reducer ops in task assigner to make them more balanced across cluster #3048)
• [Ray] Destroy Ray executor when the task finish ([Ray] Destroy Ray executor when the task finish #3049)
• [Ray] Implements get_chunks_meta for Ray execution context ([Ray] Implements get_chunks_meta for Ray execution context #3052)
• [Ray] Support basic subtask retry and lineage reconstruction ([Ray] Support basic subtask retry and lineage reconstruction #2969)
• Combine tree and shuffle methods in DataFrameGroupBy.agg tile (Combine tree and shuffle methods in DataFrameGroupBy.agg tile #3051)
• [Ray] Implements get_total_n_cpu for Ray execution context ([Ray] Implements get_total_n_cpu for Ray execution context #3059)
• [Ray] Implement cancel method on Ray task executor ([Ray] Implement cancel method on Ray task executor #3044)
• Use OS-designated ports instead of random ports to create sub pools (Use OS-designated ports instead of random ports to create sub pools #3053)
• Unify DataFrameGroupByAgg's tile logic for auto method (Unify DataFrameGroupByAgg's tile logic for auto method #3084)
• Simplify router clean up when pools or clusters ends (Simplify router clean up when pools or clusters ends #3086)
• Call immutable web API only once when previous call blocks (Call immutable web API only once when previous call blocks #3085)
• [Ray] Create RayTaskState actor as needed by default ([Ray] Create RayTaskState actor as needed by default #3081)
• [Ray] Implement gc for ray task executor context ([Ray] Implement gc for ray task executor context #3061)
• Simplify argument passing in actor batch calls (Simplify argument passing in actor batch calls #3098)
• Optimize performance of transfer (Optimize performance of transfer #3091)
• Add n_reducers and reducer_ordinal to shuffle operands (Add n_reducers and reducer_ordinal to shuffle operands #3055)
• Optimize serializable memory (Optimize serializable memory #3120)

Bug fixes

• Fix errors when deleting mapper data (Fix errors when deleting mapper data #3018)
• Fix recursive_tile that it may cause duplicated tile for one tileable (Fix recursive_tile that it may cause duplicated tile for one tileable #3021)
• Fix error message when sparse data format not supported (Fix error message when sparse data format not supported #3046)
• Patch pandas to make pickle compatible between 1.2 and 1.3 (Patch pandas to make pickle compatible between 1.2 and 1.3 #3047)
• Fix chunk index error in auto_merge_chunks (Fix chunk index error in auto_merge_chunks #3057)
• [Ray] Fix ray worker failover ([Ray] Fix ray worker failover #3080)
• [Metric] Fix prometheus metric backend ([Metric] Fix prometheus metric backend #3124)
• Fix mt.{cumsum, cumprod} when the first chunk is empty (Fix mt.{cumsum, cumprod} when the first chunk is empty #3134)

Tests

• Check initialization of serializables on CI (Check initialization of serializables on CI #3007)
• Use @pytest_asyncio.fixture instead of @pytest.fixture for async fixtures (Use @pytest_asyncio.fixture instead of @pytest.fixture for async fixtures #3025)
• Change code owners to Mars PMC maintainers (Change code owners to Mars PMC maintainers #3031)
• [Ray] Fix ray executor progress test ([Ray] Fix ray executor progress test #3033)
• [Ray] Optimize Ray CI execution time and stability ([Ray] Optimize Ray CI execution time and stability #3102)
• Make test_session_set_progress more stable under Ray tests (Make test_session_set_progress more stable under Ray tests #3103)
• Update pytest imports for test_special.py (Update pytest imports for test_special.py #3129)
• [Ray] Fix flaky test test_optional_supervisor_node ([Ray] Fix flaky test test_optional_supervisor_node #3133)

Others

• Build web code before CIBW when deploying to PyPI (Build web code before CIBW when deploying to PyPI #3014)
• Make PyPI user name configurable (Make PyPI user name configurable #3130)

---

This discussion was created from the release v0.10.0a1.