Skip to content

Commit

Permalink
[SPARK-46307][PS][TESTS] Enable fill_value tests for `GroupByTests.…
Browse files Browse the repository at this point in the history
…test_shift`

### What changes were proposed in this pull request?

This PR proposes to enable `fill_value` tests for `GroupByTests.test_shift` since the Pandas bug has been fixed.

### Why are the changes needed?

To increase the test coverage.

### Does this PR introduce _any_ user-facing change?

No, it's test-only.

### How was this patch tested?

Enable the commented tests and manually verify.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44237 from itholic/enable_shift_test.

Authored-by: Haejoon Lee <haejoon.lee@databricks.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
itholic authored and dongjoon-hyun committed Dec 7, 2023
1 parent 06c1317 commit 0692856
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions python/pyspark/pandas/tests/groupby/test_groupby.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,9 +200,10 @@ def test_shift(self):
self.assert_eq(
psdf.groupby("a").shift().sort_index(), pdf.groupby("a").shift().sort_index()
)
# TODO: seems like a pandas' bug when fill_value is not None?
# self.assert_eq(psdf.groupby(['a', 'b']).shift(periods=-1, fill_value=0).sort_index(),
# pdf.groupby(['a', 'b']).shift(periods=-1, fill_value=0).sort_index())
self.assert_eq(
psdf.groupby(["a", "b"]).shift(periods=-1, fill_value=0).sort_index(),
pdf.groupby(["a", "b"]).shift(periods=-1, fill_value=0).sort_index(),
)
self.assert_eq(
psdf.groupby(["b"])["a"].shift().sort_index(),
pdf.groupby(["b"])["a"].shift().sort_index(),
Expand Down Expand Up @@ -247,11 +248,10 @@ def test_shift(self):
psdf.groupby(("x", "a")).shift().sort_index(),
pdf.groupby(("x", "a")).shift().sort_index(),
)
# TODO: seems like a pandas' bug when fill_value is not None?
# self.assert_eq(psdf.groupby([('x', 'a'), ('x', 'b')]).shift(periods=-1,
# fill_value=0).sort_index(),
# pdf.groupby([('x', 'a'), ('x', 'b')]).shift(periods=-1,
# fill_value=0).sort_index())
self.assert_eq(
psdf.groupby([("x", "a"), ("x", "b")]).shift(periods=-1, fill_value=0).sort_index(),
pdf.groupby([("x", "a"), ("x", "b")]).shift(periods=-1, fill_value=0).sort_index(),
)

@staticmethod
def test_is_multi_agg_with_relabel():
Expand Down

0 comments on commit 0692856

Please sign in to comment.