Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sqla): copy temporal range logic to helper #22405

Merged
merged 1 commit into from
Dec 14, 2022

Conversation

villebro
Copy link
Member

@villebro villebro commented Dec 13, 2022

SUMMARY

When exploring a query containing a temporal column I noticed that the query would fail unless persisted as a virtual dataset when the GENERIC_CHART_AXES feature flag is enabled. Digging around I noticed that the ExploreMixin class in the superset.models.helpers module (introduced in #20281) duplicates essentially all logic from the
SqlaTable class in the superset.connectors.sqla.models module. From a maintenance perspective this is highly problematic, as the SQLA connector model is by far one of the biggest pieces of tech debt in the backend with poor test coverage, while simultaneously being very fragile and frequently receiving minor tweaks/fixes. Duplicating this code without having tests in place to ensure that they're in sync makes it difficult for contributors to know that changes need to be applied in both modules.

This is just a hotfix to make the ExploreMixin class from the helpers module work with GENERIC_CHART_AXES. However, IMO we need to deduplicate this code ASAP, as they've already diverged quite a bit, with many fixes to the SQLA model not having made their way into the helpers module (#22280 is an example of such). I haven't reviewed the original PR, but a simple solution that comes to mind is removing the original logic and adding ExploreMixin to SqlaTable, which ensures that the legacy model (which is still the main one in use) uses the exact same logic as the new models.

AFTER

Now a query that hasn't been saved as a virtual dataset works with temporal filters:
image

BEFORE

Previously the same chart would fail.
image

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Dec 13, 2022

Codecov Report

Merging #22405 (afa119b) into master (a235078) will decrease coverage by 0.00%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##           master   #22405      +/-   ##
==========================================
- Coverage   66.87%   66.86%   -0.01%     
==========================================
  Files        1847     1847              
  Lines       70574    70578       +4     
  Branches     7748     7748              
==========================================
- Hits        47195    47192       -3     
- Misses      21378    21385       +7     
  Partials     2001     2001              
Flag Coverage Δ
hive 52.52% <25.00%> (-0.01%) ⬇️
mysql 77.97% <25.00%> (-0.01%) ⬇️
postgres 78.02% <25.00%> (-0.02%) ⬇️
presto 52.42% <25.00%> (-0.01%) ⬇️
python 81.22% <25.00%> (-0.02%) ⬇️
sqlite 76.50% <25.00%> (-0.01%) ⬇️
unit 50.91% <25.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/models/helpers.py 38.13% <25.00%> (-0.06%) ⬇️
superset/reports/commands/log_prune.py 85.71% <0.00%> (-3.58%) ⬇️
superset/commands/importers/v1/utils.py 92.20% <0.00%> (-1.30%) ⬇️
superset/common/query_object.py 93.84% <0.00%> (-0.52%) ⬇️
superset/reports/commands/execute.py 91.55% <0.00%> (-0.34%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@zhaoyongjie zhaoyongjie self-requested a review December 13, 2022 14:17
Copy link
Member

@zhaoyongjie zhaoyongjie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExploreMixin has made the already confusing superset.connectors even more difficult to maintain. I agree with this PR change and hope the community can reduce the superset.connectors complexity.

@villebro
Copy link
Member Author

Tagging @hughhhh and @AAfghahi for a review as you authored and approved #20281 to see what your thoughts are on DRYing up ExploreMixin and SqlaTable.

@eschutho
Copy link
Member

HI @villebro. Thanks for flagging this. Yes, @hughhhh will be working on removing the code duplication in the next few weeks. The original plan was to incorporate SIP68 work here and remove the sqlaTable and migrate the tables and columns over to the new SIP68 models and at the same time use the new mixin, but we decided not to allow charts to be created by tables and columns and the new dataset models and now we need to dig right into sqlaTable instead. But the plan moving forward is to use the exploreMixin and clean up the sqlaTable code during the process of moving the code over.

@villebro
Copy link
Member Author

villebro commented Dec 14, 2022

HI @villebro. Thanks for flagging this. Yes, @hughhhh will be working on removing the code duplication in the next few weeks. The original plan was to incorporate SIP68 work here and remove the sqlaTable and migrate the tables and columns over to the new SIP68 models and at the same time use the new mixin, but we decided not to allow charts to be created by tables and columns and the new dataset models and now we need to dig right into sqlaTable instead. But the plan moving forward is to use the exploreMixin and clean up the sqlaTable code during the process of moving the code over.

Thanks @eschutho for the context. Please feel free to tag me in the cleanup PRs to help push this cleanup effort forward.

@villebro villebro merged commit 1014a32 into apache:master Dec 14, 2022
@villebro villebro deleted the villebro/hotfix-temporal-helper branch December 14, 2022 07:43
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 and removed 🚢 2.1.3 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/S 🚢 2.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants