fix: datetime.data in series

[zhaoyongjie]

Background

the pd.api.types.is_datetime64_any_dtype(series) is not able to check a series that contains datetime.date, the dtype of the dataframe will be object e.g.:

>>> import datetime
>>> import pandas as pd

>>> series = pd.Series([datetime.date(2018, 1, 1), datetime.date(2018, 1, 2)])
>>> pd.api.types.is_datetime64_any_dtype(series)
False
>>> series
0    2018-01-01
1    2018-01-02
dtype: object

if a series is a DateTime series, the pd.api.types.is_datetime64_any_dtype(series) will correctly check dtype.

>>> import datetime
>>> import pandas as pd

>>> series = pd.Series([datetime.datetime(2018, 1, 1), datetime.datetime(2018, 1, 2)])
>>> pd.api.types.is_datetime64_any_dtype(series)
True
>>> series
0    2018-01-01
1    2018-01-02
dtype: datetime64[ns]

Solution

add a new function is_datetime_series to only check series type instead of dtype.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

date.type.in.series.mov

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #20618 (4ae05e5) into master (818962c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #20618   +/-   ##
=======================================
  Coverage   66.82%   66.82%           
=======================================
  Files        1752     1752           
  Lines       65609    65616    +7     
  Branches     6938     6938           
=======================================
+ Hits        43842    43849    +7     
  Misses      20007    20007           
  Partials     1760     1760           

Flag	Coverage Δ
hive	53.80% <41.66%> (-0.01%)	⬇️
mysql	82.38% <100.00%> (+<0.01%)	⬆️
postgres	82.45% <100.00%> (+<0.01%)	⬆️
presto	53.67% <41.66%> (-0.01%)	⬇️
python	82.89% <100.00%> (+<0.01%)	⬆️
sqlite	82.24% <100.00%> (+<0.01%)	⬆️
unit	50.69% <75.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/common/query_context_processor.py	90.27% <100.00%> (ø)
superset/common/utils/dataframe_utils.py	95.23% <100.00%> (+2.38%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 818962c...4ae05e5. Read the comment docs.

[john-bodley]

It's a shame that Pandas doesn't have native support for datetime.date objects. Maybe this is yet another reason we should eventually replace it with PyArrow.

[zhaoyongjie]

Thanks, @john-bodley! I am impressed with the Arrow zero-copy feature.