Description
Code Sample, a copy-pastable example if possible
import pandas as pd
import numpy as np
df = pd.DataFrame({'B': [0, 1, np.nan, 3, 4], 'C': [4, 3, np.nan, 1, 0],
'Time': [pd.Timestamp('20130101 09:00:00'),
pd.Timestamp('20130101 09:00:01'),
pd.Timestamp('20130101 09:00:02'),
pd.Timestamp('20130101 09:00:03'),
pd.Timestamp('20130101 09:00:04')]})
df.rolling('4s', on='Time').max()
df.rolling('4s', on='Time').min()Problem description
When running a rolling max or min window on a datetime column, a NaN value seems to prevent the max or min function from considering values that follow it, even if those values are within the window.
dataframe
B Time C
0 0.0 2013-01-01 09:00:00 4.0
1 1.0 2013-01-01 09:00:01 3.0
2 NaN 2013-01-01 09:00:02 NaN
3 3.0 2013-01-01 09:00:03 1.0
4 4.0 2013-01-01 09:00:04 0.0
max() Output (column B)
In[1]: df.rolling('4s', on='Time').max()
Out[1]:
B Time C
0 0.0 2013-01-01 09:00:00 4.0
1 1.0 2013-01-01 09:00:01 4.0
2 1.0 2013-01-01 09:00:02 4.0
3 1.0 2013-01-01 09:00:03 4.0
4 1.0 2013-01-01 09:00:04 3.0
Expected max() Output (column B)
In[1]: df.rolling('4s', on='Time').max()
Out[1]:
B Time C
0 0.0 2013-01-01 09:00:00 4.0
1 1.0 2013-01-01 09:00:01 4.0
2 1.0 2013-01-01 09:00:02 4.0
3 3.0 2013-01-01 09:00:03 4.0
4 4.0 2013-01-01 09:00:04 3.0
min() Output (column C)
In[2]: df.rolling('4s', on='Time').min()
Out[2]:
B Time C
0 0.0 2013-01-01 09:00:00 4.0
1 0.0 2013-01-01 09:00:01 3.0
2 0.0 2013-01-01 09:00:02 3.0
3 0.0 2013-01-01 09:00:03 3.0
4 1.0 2013-01-01 09:00:04 3.0
Expected min() Output (column C)
In[2]: df.rolling('4s', on='Time').min()
Out[2]:
B Time C
0 0.0 2013-01-01 09:00:00 4.0
1 0.0 2013-01-01 09:00:01 3.0
2 0.0 2013-01-01 09:00:02 3.0
3 0.0 2013-01-01 09:00:03 1.0
4 1.0 2013-01-01 09:00:04 0.0
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None