Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fillna after resample doesn't work on version 0.20.1 #16479

Closed
jmilhomens opened this issue May 24, 2017 · 2 comments
Closed

fillna after resample doesn't work on version 0.20.1 #16479

jmilhomens opened this issue May 24, 2017 · 2 comments
Labels
Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations Regression Functionality that used to work in a prior pandas version
Milestone

Comments

@jmilhomens
Copy link

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.DataFrame({ 'values': range(5)}, index = pd.date_range(start = '2017-01-01 00:00:00', periods = 5, freq='S'))
resampled_df = df.resample('500ms').mean()
resampled_df.fillna(method='ffill')

Problem description

The fillna method is not working after a resample. The result of the above code is:

values
2017-01-01 00:00:00.000 0.0
2017-01-01 00:00:00.500 NaN
2017-01-01 00:00:01.000 1.0
2017-01-01 00:00:01.500 NaN
2017-01-01 00:00:02.000 2.0
2017-01-01 00:00:02.500 NaN
2017-01-01 00:00:03.000 3.0
2017-01-01 00:00:03.500 NaN
2017-01-01 00:00:04.000 4.0

This code worked with version 0.19.2

Expected Output

The expected output in this case can be also obtained with:

df.resample('500ms').ffill()

The result is:

values
2017-01-01 00:00:00.000 0
2017-01-01 00:00:00.500 0
2017-01-01 00:00:01.000 1
2017-01-01 00:00:01.500 1
2017-01-01 00:00:02.000 2
2017-01-01 00:00:02.500 2
2017-01-01 00:00:03.000 3
2017-01-01 00:00:03.500 3
2017-01-01 00:00:04.000 4

However, in some cases one may need to do the re-sample and fill missing values in separate steps.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 16.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

@TomAugspurger TomAugspurger added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations Regression Functionality that used to work in a prior pandas version labels May 24, 2017
@TomAugspurger TomAugspurger added this to the 0.20.2 milestone May 24, 2017
@TomAugspurger
Copy link
Contributor

Seems at least partially related to

def check_int_bool(self, inplace):
returning True, when it should be False (we have ints + nans, so it should be float). I guess the root cause is resampled_df having an IntBlock

In [6]: resampled_df._data
Out[6]:
BlockManager
Items: Index(['values'], dtype='object')
Axis 1: DatetimeIndex([       '2017-01-01 00:00:00', '2017-01-01 00:00:00.500000',
                      '2017-01-01 00:00:01', '2017-01-01 00:00:01.500000',
                      '2017-01-01 00:00:02', '2017-01-01 00:00:02.500000',
                      '2017-01-01 00:00:03', '2017-01-01 00:00:03.500000',
                      '2017-01-01 00:00:04'],
              dtype='datetime64[ns]', freq='500L')
IntBlock: slice(0, 1, 1), 1 x 9, dtype: float64

potentially related to #16361

@jreback
Copy link
Contributor

jreback commented May 25, 2017

this is a dupe of #16361

@jreback jreback closed this as completed May 25, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

No branches or pull requests

3 participants