Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(postgresql): dynamic schema #23401

Merged
merged 4 commits into from
Mar 18, 2023
Merged

feat(postgresql): dynamic schema #23401

merged 4 commits into from
Mar 18, 2023

Conversation

betodealmeida
Copy link
Member

@betodealmeida betodealmeida commented Mar 16, 2023

SUMMARY

Builds on #23385. Renames the adjust_database_uri method to adjust_engine_params and makes it accept connect_args as well, so we can implement dynamic schema in Postgres. In addition, the method also takes an optional catalog, so in the future we can implement proper catalog support.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before, when using Postgres the SQL Lab schema dropdown is not used to run the query. Note that in the example below information_schema is selected, but the query actually runs in the public schema:

Screenshot 2023-03-16 at 15-23-18 Superset

With this PR:

Screenshot 2023-03-16 at 15-23-34 Superset

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@betodealmeida betodealmeida force-pushed the postgres_set_schema branch 2 times, most recently from 0ba2db6 to 05b1e7e Compare March 16, 2023 22:15
@codecov
Copy link

codecov bot commented Mar 16, 2023

Codecov Report

Merging #23401 (f465853) into master (42e8d1b) will increase coverage by 1.64%.
The diff coverage is 80.00%.

❗ Current head f465853 differs from pull request most recent head 2db69f2. Consider uploading reports for the commit 2db69f2 to get more accurate results

@@            Coverage Diff             @@
##           master   #23401      +/-   ##
==========================================
+ Coverage   65.96%   67.61%   +1.64%     
==========================================
  Files        1907     1907              
  Lines       73590    73611      +21     
  Branches     7982     7982              
==========================================
+ Hits        48546    49772    +1226     
+ Misses      22996    21791    -1205     
  Partials     2048     2048              
Flag Coverage Δ
hive 52.73% <36.92%> (-0.02%) ⬇️
mysql 78.40% <67.69%> (+<0.01%) ⬆️
postgres 78.46% <67.69%> (-0.04%) ⬇️
presto 52.65% <38.46%> (-0.02%) ⬇️
python 82.32% <80.00%> (+3.39%) ⬆️
sqlite 76.97% <67.69%> (?)
unit 52.51% <58.46%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/snowflake.py 66.86% <14.28%> (+19.27%) ⬆️
superset/db_engine_specs/drill.py 88.88% <40.00%> (+33.33%) ⬆️
superset/dashboards/commands/importers/v1/utils.py 85.00% <50.00%> (+6.79%) ⬆️
superset/models/core.py 89.74% <77.77%> (-0.36%) ⬇️
superset/dashboards/schemas.py 98.98% <83.33%> (-0.50%) ⬇️
superset/db_engine_specs/base.py 90.80% <100.00%> (+8.75%) ⬆️
superset/db_engine_specs/hive.py 87.79% <100.00%> (+1.01%) ⬆️
superset/db_engine_specs/mysql.py 98.79% <100.00%> (+9.63%) ⬆️
superset/db_engine_specs/postgres.py 97.01% <100.00%> (+5.95%) ⬆️
superset/db_engine_specs/presto.py 88.05% <100.00%> (+1.25%) ⬆️

... and 81 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@betodealmeida betodealmeida changed the base branch from default_schema to master March 17, 2023 15:39
@@ -1057,30 +1057,33 @@ def extract_errors(
]

@classmethod
def adjust_database_uri( # pylint: disable=unused-argument
def adjust_engine_params( # pylint: disable=unused-argument
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@betodealmeida what do you think will be the impact of this change on any custom db engine specs that have been added to Superset instances?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, nm, I see how you covered this below.

) -> URL:
connect_args: Dict[str, Any],
catalog: Optional[str],
schema: Optional[str],
Copy link
Member

@eschutho eschutho Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that in some of the class overrides, catalog and schema have kwargs with defaults instead of positional arguments. Do we want to be consistent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point, let me add defaults here.

Copy link
Member

@eschutho eschutho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I just had the one question about args vs kwargs, but not a blocker.

@betodealmeida betodealmeida merged commit 2c6f581 into master Mar 18, 2023
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024
@mistercrunch mistercrunch deleted the postgres_set_schema branch March 26, 2024 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 3.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants