TypeError: ufunc 'isnan' not supported for the input types... #9
Comments
|
That's weird .isnan() of all things is giving the error. Did you already
look at where the "strict" flag is set in your environment?
…On Tue, Oct 6, 2020, 5:53 PM Mitch ***@***.***> wrote:
Hoping for some clarification .
using socketcan .log files , I receive the following errors after
installing on my new laptop, (mint 20.04, same as desktop)
File "Main.py", line 85, in
id_dictionary, j1979_dictionary =
pre_processor.generate_arb_id_dictionary(a_timer,
File
"/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line
87, in generate_arb_id_dictionary
self.import_csv(a_timer, self.data_filename)
File
"/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line
43, in import_csv
self.data = read_csv(filename,
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
686, in read_csv
return _read(filepath_or_buffer, kwds)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
458, in _read
data = parser.read(nrows)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1196, in read
ret = self._engine.read(nrows)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
2231, in read
index, names = self._make_index(data, alldata, names)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1677, in _make_index
index = self._agg_index(index)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1770, in _agg_index
arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1871, in _infer_types
mask = algorithms.isin(values, list(na_values))
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py",
line 443, in isin
if np.isnan(values).any():
TypeError: ufunc 'isnan' not supported for the input types, and the inputs
could not be safely coerced to any supported types according to the casting
rule ''safe''
odd thing is, it doesn't happen on my desktop.. (Same OS)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#9>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEG4NYDSMPOEBKVSIOSMNXDSJOGUDANCNFSM4SGS6CZQ>
.
|
After some digging, it appears to be an issue with pandas 1.1.3 specifically, and the issue is already being worked on in the pandas-dev repository. Here's a link to the issue tracker on the pandas-dev repo: https://github.com/pandas-dev/pandas/issues/37094 Currently the bug is slated to be fixed in the 1.1.4 release which is due to come out tomorrow on October 30th. It appears that it is related to the index_col argument for read_csv() when > 1000000 lines, so a quick fix would be to set index separately from read_csv() instead of including it as an argument or to process fewer than 1000000 lines per file. Here's an example from the issue tracker:
In short, upgrade to pandas 1.1.4 or downgrade to pandas 1.1.2 and this should be fixed. Since the issue is not with this repo I don't think any changes are needed besides changing requirements.txt line 2 from |
|
Thanks for following up on this.
There's three lines in this project for bandaid fixing another pandas
indexing bug. I spent about two weeks trying to figure out what I was doing
wrong. Only problem is it would took sharing a private data file to
demonstrate how to reproduce the error so I didn't create an issue with
Pandas :(
…On Thu, Oct 29, 2020, 10:50 AM Joshua Arking ***@***.***> wrote:
Hoping for some clarification .
using socketcan .log files , I receive the following errors after
installing on my new laptop, (mint 20.04, same as desktop)
File "Main.py", line 85, in
id_dictionary, j1979_dictionary =
pre_processor.generate_arb_id_dictionary(a_timer,
File
"/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line
87, in generate_arb_id_dictionary
self.import_csv(a_timer, self.data_filename)
File
"/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line
43, in import_csv
self.data = read_csv(filename,
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
686, in read_csv
return _read(filepath_or_buffer, kwds)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
458, in _read
data = parser.read(nrows)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1196, in read
ret = self._engine.read(nrows)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
2231, in read
index, names = self._make_index(data, alldata, names)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1677, in _make_index
index = self._agg_index(index)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1770, in _agg_index
arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line
1871, in _infer_types
mask = algorithms.isin(values, list(na_values))
File
"/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py",
line 443, in isin
if np.isnan(values).any():
TypeError: ufunc 'isnan' not supported for the input types, and the inputs
could not be safely coerced to any supported types according to the casting
rule ''safe''
odd thing is, it doesn't happen on my desktop.. (Same OS)
After some digging, it appears to be an issue with pandas 1.1.3
specifically, and the issue is already being worked on in the pandas-dev
repository. Here's a link to the issue tracker on the pandas-dev repo:
pandas-dev/pandas#37094 <http://url> Currently
the bug is slated to be fixed in the 1.1.4 release which is due to come out
tomorrow on October 30th. It appears that it is related to the index_col
argument for read_csv() when > 1000000 lines, so a quick fix would be to
set index separately from read_csv() instead of including it as an argument
or to process fewer than 1000000 lines per file. Here's an example from the
issue tracker:
I can confirm I have the same problem, which arises as soon as I pass the
threshold of 1M rows.
I only need to specify index_col to get the bug, though. Specifying dtypes
is not needed.
Pandas 1.1.3
And as a temporary workaround, I am reading without index_col, and then
setting my index. E.g.
df = pd.read_csv(filepath, nrows=1000001)df.set_index(0) # for example if I wanted the first column
In short, upgrade to pandas 1.1.4 or downgrade to pandas 1.1.2 and this
should be fixed. Since the issue is not with this repo I don't think any
changes are needed besides changing requirements.txt line 2 from
pandas
to
pandas>=1.1.4
or
pandas<=1.1.2
I am not familiar with requirements.txt syntax, so if there's a way to
specifically exclude only version 1.1.3, that would be ideal.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#9 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEG4NYC5CIUAC4VHHYTEOY3SNF6MZANCNFSM4SGS6CZQ>
.
|
Hoping for some clarification .
using socketcan .log files , I receive the following errors after installing on my new laptop, (mint 20.04, same as desktop)
File "Main.py", line 85, in
id_dictionary, j1979_dictionary = pre_processor.generate_arb_id_dictionary(a_timer,
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 87, in generate_arb_id_dictionary
self.import_csv(a_timer, self.data_filename)
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 43, in import_csv
self.data = read_csv(filename,
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read
data = parser.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read
ret = self._engine.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 2231, in read
index, names = self._make_index(data, alldata, names)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1677, in _make_index
index = self._agg_index(index)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1770, in _agg_index
arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1871, in _infer_types
mask = algorithms.isin(values, list(na_values))
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py", line 443, in isin
if np.isnan(values).any():
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
odd thing is, it doesn't happen on my desktop.. (Same OS)
The text was updated successfully, but these errors were encountered: