Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add option to load big/wide tables #13448

Merged
merged 4 commits into from
Mar 12, 2021

Conversation

betodealmeida
Copy link
Member

SUMMARY

This PR adds a new CLI option to load big/wide tables into Superset. The goal is to create tables with synthetic data that help developers test new features and DB migrations in realistic scenarios.

This first PR just adds a function called add_data that generates the data given a schema. When running superset load-examples -b it will also create a wide table with 100 columns, to help test SQL Lab (which behaves differently if the table has more than 50 columns).

The next steps include adding more tables, as well as implementing functionality to add more rows to the Superset models, to help benchmark DB migrations.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Ran superset load-examples -b -t and verified that the table wide_table was created, with 1000 rows and 100 columns.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Mar 4, 2021

Codecov Report

Merging #13448 (37f76e1) into master (974f447) will decrease coverage by 5.80%.
The diff coverage is 41.85%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #13448      +/-   ##
==========================================
- Coverage   77.14%   71.34%   -5.81%     
==========================================
  Files         865      825      -40     
  Lines       44961    41180    -3781     
  Branches     5415     4260    -1155     
==========================================
- Hits        34686    29378    -5308     
- Misses      10152    11802    +1650     
+ Partials      123        0     -123     
Flag Coverage Δ
cypress 57.17% <41.85%> (-1.38%) ⬇️
hive 79.96% <ø> (?)
javascript ?
mysql 80.29% <ø> (?)
postgres 80.33% <ø> (?)
presto 79.99% <ø> (?)
python 80.82% <ø> (+0.01%) ⬆️
sqlite 79.95% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...erset-frontend/src/SqlLab/components/SouthPane.jsx 64.10% <ø> (-17.95%) ⬇️
...et-frontend/src/SqlLab/components/TableElement.jsx 5.88% <0.00%> (-80.64%) ⬇️
...end/src/SqlLab/components/TemplateParamsEditor.jsx 23.80% <ø> (ø)
...et-frontend/src/SqlLab/reducers/getInitialState.js 50.00% <ø> (ø)
superset-frontend/src/SqlLab/reducers/sqlLab.js 17.47% <0.00%> (-23.10%) ⬇️
...src/SqlLab/utils/reduxStateToLocalStorageHelper.js 61.53% <ø> (-38.47%) ⬇️
superset-frontend/src/chart/ChartContainer.jsx 100.00% <ø> (ø)
...-frontend/src/common/components/Collapse/index.tsx 94.73% <ø> (ø)
...ontend/src/common/components/InfoTooltip/index.tsx 33.33% <ø> (ø)
...set-frontend/src/common/components/Modal/Modal.tsx 100.00% <ø> (ø)
... and 650 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 974f447...e3aba40. Read the comment docs.

load_test_data: bool,
load_big_data: bool,
only_metadata: bool = False,
force: bool = False,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a bit of a nit, but for consistency, do you think it's better to have default False values on all of these or allow nulls?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case click is adding the default values when parsing the CLI options, but I'll add some default values to the function signature as well.

Copy link
Member

@eschutho eschutho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome!

@betodealmeida betodealmeida merged commit 1470e70 into apache:master Mar 12, 2021
allanco91 pushed a commit to allanco91/superset that referenced this pull request May 21, 2021
* WIP

* feat: add option to load big/wide tables

* Fix lint

* Address comments
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.2.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 1.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants