Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Always use temporal type for dttm columns [ID-2] #17458

Merged
merged 4 commits into from
Nov 22, 2021

Conversation

kgabryje
Copy link
Member

@kgabryje kgabryje commented Nov 17, 2021

SUMMARY

If a column is marked as is_dttm, always return it's generic type as TEMPORAL. This change fixes a bug in databases like Druid that sometimes return dates as string instead of datetime format.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

  1. Set a column as temporal in Edit Dataset modal
  2. Add that column to chart
  3. Verify that coltype returned for that column in response is equal to 2 - Temporal

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

CC @jinghua-qa @junlincc

@kgabryje kgabryje requested a review from villebro November 17, 2021 07:22
@kgabryje kgabryje marked this pull request as draft November 17, 2021 07:22
@pull-request-size pull-request-size bot added size/M and removed size/S labels Nov 17, 2021
@villebro villebro changed the title fix: Always use temporal type for dttm columns [WIP] fix: Always use temporal type for dttm columns Nov 17, 2021
@villebro villebro marked this pull request as ready for review November 17, 2021 09:31
@codecov
Copy link

codecov bot commented Nov 17, 2021

Codecov Report

Merging #17458 (9ee7201) into master (e2a429b) will decrease coverage by 0.21%.
The diff coverage is 81.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #17458      +/-   ##
==========================================
- Coverage   76.94%   76.73%   -0.22%     
==========================================
  Files        1042     1042              
  Lines       56248    56257       +9     
  Branches     7784     7784              
==========================================
- Hits        43282    43168     -114     
- Misses      12707    12830     +123     
  Partials      259      259              
Flag Coverage Δ
hive ?
mysql 81.95% <81.81%> (-0.01%) ⬇️
postgres 81.96% <81.81%> (-0.01%) ⬇️
presto ?
python 82.05% <81.81%> (-0.42%) ⬇️
sqlite 81.64% <81.81%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/charts/post_processing.py 67.74% <50.00%> (+0.26%) ⬆️
superset/charts/data/api.py 88.65% <75.00%> (+0.08%) ⬆️
superset/common/query_actions.py 92.85% <100.00%> (ø)
superset/utils/core.py 89.64% <100.00%> (-0.10%) ⬇️
superset/db_engines/hive.py 0.00% <0.00%> (-85.19%) ⬇️
superset/db_engine_specs/hive.py 69.49% <0.00%> (-16.99%) ⬇️
superset/db_engine_specs/presto.py 83.47% <0.00%> (-6.91%) ⬇️
superset/views/database/mixins.py 81.03% <0.00%> (-1.73%) ⬇️
superset/connectors/sqla/models.py 86.79% <0.00%> (-1.59%) ⬇️
superset/utils/webdriver.py 79.78% <0.00%> (-1.12%) ⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2a429b...9ee7201. Read the comment docs.

@kgabryje kgabryje changed the title [WIP] fix: Always use temporal type for dttm columns fix: Always use temporal type for dttm columns [ID-2] Nov 19, 2021
@kgabryje kgabryje requested a review from zhaoyongjie November 19, 2021 16:14
Copy link
Member

@zhaoyongjie zhaoyongjie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. frontend follow-up suggestion. If a string can't be parsed, we will use NaT or original text.

After

image

Before

image

@kgabryje kgabryje merged commit 1f8eff7 into apache:master Nov 22, 2021
AAfghahi pushed a commit that referenced this pull request Jan 10, 2022
* fix: Always use temporal type for dttm columns

* move inference and implement in chart postproc

* fix test

* fix test case

Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/M 🚢 1.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants