Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(explore): export csv data pivoted for Pivot Table [ID-9] #17512

Merged
merged 7 commits into from
Dec 3, 2021

Conversation

kgabryje
Copy link
Member

@kgabryje kgabryje commented Nov 22, 2021

SUMMARY

Apply pivot table post processing to data downloaded as CSV in Pivot Table V2 chart. The CSV data will have pivoted format (see the difference on screenshots).

When on Pivot Table chart, the .CSV button is replaced by a dropdown containing 2 options - Original and Pivoted. Clicking the latter triggers download of pivoted data.
On all other charts, the CSV button works just like before.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Example pivot table:

image

Before:
image

After:
image

image

TESTING INSTRUCTIONS

Create a pivot table chart, click on download as csv icon. A dropdown with 2 options should open - Original and Pivoted.
Clicking Original should download a CSV file formatted like the one on the "Before" screenshot.
Clicking Pivoted should download a CSV file formatted like the one on the "After" screenshot

The CSV button on any other chart should work like before - begin download immediately instead of opening a dropdown menu.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

CC @junlincc

@codecov
Copy link

codecov bot commented Nov 22, 2021

Codecov Report

Merging #17512 (334210b) into master (e6212b1) will increase coverage by 0.00%.
The diff coverage is 94.28%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #17512    +/-   ##
========================================
  Coverage   68.57%   68.57%            
========================================
  Files        1588     1596     +8     
  Lines       64947    65214   +267     
  Branches     6963     6949    -14     
========================================
+ Hits        44537    44721   +184     
- Misses      18520    18610    +90     
+ Partials     1890     1883     -7     
Flag Coverage Δ
hive ?
javascript 57.42% <93.10%> (+0.35%) ⬆️
mysql 81.80% <100.00%> (?)
postgres 81.81% <100.00%> (-0.23%) ⬇️
presto ?
python 81.90% <100.00%> (-0.60%) ⬇️
sqlite 81.50% <100.00%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...es/superset-ui-core/src/query/buildQueryContext.ts 100.00% <ø> (ø)
...packages/superset-ui-core/src/query/types/Query.ts 100.00% <ø> (ø)
superset/common/query_context_factory.py 96.42% <ø> (ø)
...c/explore/components/ExportToCSVDropdown/index.tsx 90.00% <90.00%> (ø)
...nd/src/explore/components/ExploreActionButtons.tsx 74.00% <100.00%> (+3.54%) ⬆️
superset/charts/data/api.py 88.15% <100.00%> (+0.07%) ⬆️
superset/charts/schemas.py 99.33% <100.00%> (+<0.01%) ⬆️
superset/common/query_context.py 92.72% <100.00%> (+0.27%) ⬆️
superset/db_engines/hive.py 0.00% <0.00%> (-85.19%) ⬇️
superset/db_engine_specs/hive.py 70.27% <0.00%> (-16.99%) ⬇️
... and 61 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6212b1...334210b. Read the comment docs.

@pull-request-size pull-request-size bot added size/L and removed size/S labels Dec 1, 2021
@kgabryje kgabryje changed the title [WIP] feat(explore): export csv data pivoted for Pivot Table [ID-9] feat(explore): export csv data pivoted for Pivot Table [ID-9] Dec 1, 2021
@kgabryje
Copy link
Member Author

kgabryje commented Dec 1, 2021

/testenv up

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2021

@kgabryje Ephemeral environment spinning up at http://52.39.232.35:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small comments, but this works and looks great! It would be nice if we could add a few tests:

  1. Ensure that a query context with form_data validates successfully
  2. Add tests for ExportToCSVDropdown that ensures it renders correctly with standard/pivoted options + the children.

@@ -221,6 +221,7 @@ def data(self) -> Response:
if json_body is None:
return self.response_400(message=_("Request is not JSON"))

form_data = json_body.pop("form_data", None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to pop here? Can't we just do

Suggested change
form_data = json_body.pop("form_data", None)
form_data = json_body.get("form_data")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need form_data in json_body. We pass it as a separate argument

@kgabryje
Copy link
Member Author

kgabryje commented Dec 2, 2021

A few small comments, but this works and looks great! It would be nice if we could add a few tests:

  1. Ensure that a query context with form_data validates successfully
  2. Add tests for ExportToCSVDropdown that ensures it renders correctly with standard/pivoted options + the children.

Added tests for ExportToCSVDropdown. I didn't add tests for query context validation, as we pop form_data from json_body before it's used to create context, so there's no change there.

@kgabryje
Copy link
Member Author

kgabryje commented Dec 2, 2021

A few small comments, but this works and looks great! It would be nice if we could add a few tests:

  1. Ensure that a query context with form_data validates successfully
  2. Add tests for ExportToCSVDropdown that ensures it renders correctly with standard/pivoted options + the children.

Added tests for ExportToCSVDropdown. I didn't add tests for query context validation, as we pop form_data from json_body before it's used to create context, so there's no change there.

After some discussion, I implemented Ville's suggestions. We already had schema validations tests in place, so I just added form_data to query context generator.

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few last comments

@@ -158,6 +163,7 @@ export interface QueryContext {
/** Response format */
result_format: string;
queries: QueryObject[];
form_data: QueryFormData;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be optional, as it is in the python code?

  form_data?: QueryFormData;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -1157,6 +1157,8 @@ class ChartDataQueryContextSchema(Schema):
result_type = EnumField(ChartDataResultType, by_value=True)
result_format = EnumField(ChartDataResultFormat, by_value=True)

form_data = fields.Raw()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add allow_none=True and required=False here, as we probably want to support explicitly passing a null value

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

tests/common/query_context_generator.py Outdated Show resolved Hide resolved
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for this great improvement!

@kgabryje kgabryje merged commit 07e8837 into apache:master Dec 3, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2021

Ephemeral environment shutdown and build artifacts deleted.

@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 1.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants