Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: remove shadow write of new sip 68 dataset models #21986

Merged
merged 1 commit into from
Nov 14, 2022

Conversation

eschutho
Copy link
Member

@eschutho eschutho commented Oct 31, 2022

SUMMARY

We're seeing some sqlalchemy errors related to the dual writes of the sip68 models, so I'm removing the dual writing so that we can decide whether to move forward again and do another backfill migration or start over smaller with one model at a time. We also haven't been writing permissions to the models, so we would need to run a migration for that in the future regardless of whether we stopped dual writing or not, so it seemed to make sense to take a step back and stop the errors for now.

I also fixed a test on dataset deletion due to sqlalchemy not having access to the database relationship in the session. The fix was to eagerly load the database with the dataset on lookup. I found other logs of this happening for this command, and fixed it here for the tests to pass. There was a query in the after update hook that I removed that was re-loading the database into the session as an unexpected side-effect.

TESTING INSTRUCTIONS

Normal dataset crud operations should work as normal.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 648f708 to a0008fe Compare October 31, 2022 23:44
@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch 2 times, most recently from 839ea7e to 92a8d28 Compare October 31, 2022 23:52
@eschutho eschutho changed the title chore: remove shadow write of new sip 68 dataset models DRAFT: chore: remove shadow write of new sip 68 dataset models Nov 1, 2022
@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch 2 times, most recently from de7fb3c to ee16eca Compare November 2, 2022 18:59
@pull-request-size pull-request-size bot added size/XL and removed size/L labels Nov 2, 2022
@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from ee16eca to 880f9d3 Compare November 4, 2022 23:59
@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 880f9d3 to 3384e12 Compare November 9, 2022 17:51
@codecov
Copy link

codecov bot commented Nov 9, 2022

Codecov Report

Merging #21986 (0373446) into master (33fc578) will decrease coverage by 0.04%.
The diff coverage is 77.91%.

❗ Current head 0373446 differs from pull request most recent head d2523c8. Consider uploading reports for the commit d2523c8 to get more accurate results

@@            Coverage Diff             @@
##           master   #21986      +/-   ##
==========================================
- Coverage   66.95%   66.91%   -0.05%     
==========================================
  Files        1807     1818      +11     
  Lines       69222    69471     +249     
  Branches     7403     7496      +93     
==========================================
+ Hits        46351    46484     +133     
- Misses      20966    21049      +83     
- Partials     1905     1938      +33     
Flag Coverage Δ
hive 52.50% <40.94%> (-0.42%) ⬇️
mysql 78.14% <79.13%> (-0.23%) ⬇️
postgres ?
presto 52.40% <40.55%> (-0.43%) ⬇️
python 81.19% <87.40%> (-0.30%) ⬇️
sqlite ?
unit 50.73% <59.84%> (-0.36%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ackages/superset-ui-chart-controls/src/fixtures.ts 100.00% <ø> (ø)
...d/packages/superset-ui-chart-controls/src/index.ts 100.00% <ø> (ø)
...perset-ui-chart-controls/src/sections/sections.tsx 71.42% <0.00%> (-16.08%) ⬇️
...chart-controls/src/shared-controls/dndControls.tsx 58.33% <ø> (ø)
...d/packages/superset-ui-chart-controls/src/types.ts 100.00% <ø> (ø)
...kages/superset-ui-core/src/query/types/Operator.ts 100.00% <ø> (ø)
...packages/superset-ui-core/src/query/types/Query.ts 100.00% <ø> (ø)
...set-ui-core/src/ui-overrides/ExtensionsRegistry.ts 100.00% <ø> (ø)
.../legacy-plugin-chart-histogram/src/controlPanel.ts 40.00% <0.00%> (ø)
...lugin-chart-handlebars/src/plugin/controlPanel.tsx 50.00% <ø> (ø)
... and 137 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@eschutho eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 3384e12 to d2523c8 Compare November 10, 2022 18:28
@eschutho eschutho changed the title DRAFT: chore: remove shadow write of new sip 68 dataset models chore: remove shadow write of new sip 68 dataset models Nov 10, 2022
Copy link
Member

@ktmud ktmud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@betodealmeida definitely is more qualified of reviewing this than me, but I agree the dual-write is probably not ideal if we aren't going to resume working on the new models any time soon.

If it's something you are allow to share, can you elaborate what were the errors you saw?

@@ -35,6 +37,26 @@ class DatasetDAO(BaseDAO): # pylint: disable=too-many-public-methods
model_cls = SqlaTable
base_filter = DatasourceFilter

@classmethod
def find_by_ids(cls, model_ids: Union[List[str], List[int]]) -> List[SqlaTable]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this related?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks unrelated... maybe this was supposed to be in another branch?

Copy link
Member Author

@eschutho eschutho Nov 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I thought I had mentioned it in the PR description, but yes tests were failing on dataset deletion due to sqlalchemy not having access to the database relationship in the session. I found other logs of this happening for this command, and fixed it here for the tests to pass. There was a query in the after update hook that I removed that was re-loading the database into the session as an unexpected side-effect.

Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -35,6 +37,26 @@ class DatasetDAO(BaseDAO): # pylint: disable=too-many-public-methods
model_cls = SqlaTable
base_filter = DatasourceFilter

@classmethod
def find_by_ids(cls, model_ids: Union[List[str], List[int]]) -> List[SqlaTable]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks unrelated... maybe this was supposed to be in another branch?

@eschutho eschutho merged commit 86d52fc into apache:master Nov 14, 2022
@john-bodley
Copy link
Member

@betodealmeida and @eschutho what’s the status of SIP-68? Are there blockers for preventing us from reading from the new datamodel tables?

@eschutho
Copy link
Member Author

@john-bodley after this change, the new datasets/models and the sqlaTable/columns, etc will be out of sync. Prior to this change, some of the permissions may have been out of sync as well. We're going to try this again, but maybe with a model by model approach so that we can maybe knock out some of the new models quicker and then do the dataset change last, which will be the most complex. We just haven't had much time to contribute to wrapping this up unfortunately.

AnushaErrabelli pushed a commit to preset-io/superset that referenced this pull request Nov 15, 2022
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 and removed 🚢 2.1.3 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XXL 🚢 2.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants