Adding check process to alignment at apply pars time #422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fix for line_terminator set indexer fix removed moouu tests starting to fix applys with np functions ugh more apply ufunc fixes more apply ufunc fixes more apply ufunc fixes more pandas bs more fixes ugh more more fixes more more less reals for py3.9 mem issue
Looks like issues are less related to the use of ufunc (e.g. np.log10) on Object type series (which has be deprecated since before Pandas 1.5) but rather related to assignments with `df.loc[:,col] = RHS` casting RHS into the dtype of `df[col]` and, therefore, when RHS is expected to update the series dtype (e.g. to float or int [using apply() or astype()]) the action does nothing. Conversion of pst data cols was not occuring as expected and subsequent ufunc calls (e.g. log10) were failing. A big WTF but seems to work if the assignment is of the form `df[col]=RHS`. Have stripped back most of JTW mods after making this change in pst_handler.py
jtwhite79
approved these changes
Apr 20, 2023
| @@ -968,7 +968,7 @@ def _par_prep( | |||
| sep = " " | |||
| if rel_filepath.suffix.lower() == ".csv": | |||
| sep = "," | |||
| if df.columns.is_integer(): | |||
| if pd.api.types.is_integer_dtype(df.columns): # df.columns.is_integer(): # really!??? | |||
There was a problem hiding this comment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding additional data store during add pars and checking against as apply pars time.
Hopefully should catch misalignment of
index_colswhen applying pars.Caveat: might cause unexpected failures... may need refining.