Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-45123][PS] Raise TypeError for DataFrame.interpolate when all columns are object-dtype. #42878

Closed
wants to merge 1 commit into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Sep 12, 2023

What changes were proposed in this pull request?

This PR proposes to aise TypeError for DataFrame.interpolate when all columns are object-dtype.

Why are the changes needed?

To match the behavior of Pandas:

>>> pd.DataFrame({"A": ['a', 'b', 'c'], "B": ['a', 'b', 'c']}).interpolate()
...
TypeError: Cannot interpolate with all object-dtype columns in the DataFrame. Try setting at least one column to a numeric dtype.

We currently return empty DataFrame instead of raise TypeError:

>>> pd.DataFrame({"A": ['a', 'b', 'c'], "B": ['a', 'b', 'c']}).interpolate()
Empty DataFrame
Columns: []
Index: [0, 1, 2]

Does this PR introduce any user-facing change?

Compute DataFrame.interpolate on DataFrame that has all object-dtype columns will raise TypeError instead of returning an empty DataFrame.

How was this patch tested?

Added UT.

Was this patch authored or co-authored using generative AI tooling?

No.

@itholic
Copy link
Contributor Author

itholic commented Sep 12, 2023

cc @zhengruifeng

@zhengruifeng
Copy link
Contributor

merged to master

@ueshin
Copy link
Member

ueshin commented Sep 12, 2023

@itholic The JIRA iD in the title seems to be wrong?

@itholic itholic changed the title [SPARK-43123][PS] Raise TypeError for DataFrame.interpolate when all columns are object-dtype. [SPARK-45123][PS] Raise TypeError for DataFrame.interpolate when all columns are object-dtype. Sep 13, 2023
@itholic
Copy link
Contributor Author

itholic commented Sep 13, 2023

Oops.. Updated. Thanks, @ueshin !

@itholic itholic deleted the SPARK-45123 branch November 20, 2023 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants