-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add option to load big/wide tables #13448
Conversation
Codecov Report
@@ Coverage Diff @@
## master #13448 +/- ##
==========================================
- Coverage 77.14% 71.34% -5.81%
==========================================
Files 865 825 -40
Lines 44961 41180 -3781
Branches 5415 4260 -1155
==========================================
- Hits 34686 29378 -5308
- Misses 10152 11802 +1650
+ Partials 123 0 -123
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
load_test_data: bool, | ||
load_big_data: bool, | ||
only_metadata: bool = False, | ||
force: bool = False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a bit of a nit, but for consistency, do you think it's better to have default False
values on all of these or allow nulls?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case click
is adding the default values when parsing the CLI options, but I'll add some default values to the function signature as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome!
* WIP * feat: add option to load big/wide tables * Fix lint * Address comments
SUMMARY
This PR adds a new CLI option to load big/wide tables into Superset. The goal is to create tables with synthetic data that help developers test new features and DB migrations in realistic scenarios.
This first PR just adds a function called
add_data
that generates the data given a schema. When runningsuperset load-examples -b
it will also create a wide table with 100 columns, to help test SQL Lab (which behaves differently if the table has more than 50 columns).The next steps include adding more tables, as well as implementing functionality to add more rows to the Superset models, to help benchmark DB migrations.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A
TEST PLAN
Ran
superset load-examples -b -t
and verified that the tablewide_table
was created, with 1000 rows and 100 columns.ADDITIONAL INFORMATION