[viz] Revert dropna logic for pivot tables

[john-bodley]

CATEGORY

Choose one

• Bug Fix
• Enhancement (new features, refinement)
• Refactor
• Add tests
• Build / Development Environment
• Documentation

SUMMARY

When I authored #4905 I updated the pandas.pivot_table calls to not drop the NaN (NULL) values as per the documentation,

dropna : boolean, default True
Do not include columns whose entries are all NaN

as I wanted to ensure that NaN's were preserved.

The issue with this is pivots (cross-product) may result in combinations where the values are all NaN which pollutes the chart. I misread the important all which means that only empty combinations are removed, i.e., NaNs from the original data-frame are preserved (where necessary), i.e., for this example,

Current

>>> df.pivot_table(index=['x', 'y'], columns='parameter_id', values='result_value', dropna=False)
parameter_id    a    b    c   d
x y                            
0 0           3.0  1.0  3.0 NaN
  1           1.0  3.0  NaN NaN
  2           1.0  3.0  NaN NaN
1 0           1.0  3.0  NaN NaN
  1           NaN  NaN  NaN NaN
  2           NaN  NaN  NaN NaN

Proposed (Revert)

>>> df.pivot_table(index=['x', 'y'], columns='parameter_id', values='result_value')
parameter_id    a    b    c
x y                        
0 0           3.0  1.0  3.0
  1           1.0  3.0  NaN
  2           1.0  3.0  NaN
1 0           1.0  3.0  NaN

Note in this example the d column is NaN for all values and has been dropped which could be problematic under certain visualization types. @mistercrunch any thoughts on this?

TEST PLAN

CI.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

REVIEWERS

to: @etr2460 @graceguo-supercat @michellethomas @mistercrunch

[codecov-io]

Codecov Report

Merging #8040 into master will decrease coverage by 0.14%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8040      +/-   ##
==========================================
- Coverage   65.56%   65.42%   -0.15%     
==========================================
  Files         469      469              
  Lines       22478    22478              
  Branches     2442     2442              
==========================================
- Hits        14738    14706      -32     
- Misses       7620     7652      +32     
  Partials      120      120

Impacted Files	Coverage Δ
superset/viz.py	71.61% <100%> (ø)	⬆️
superset/db_engine_specs/mysql.py	34.88% <0%> (-58.14%)	⬇️
superset/models/core.py	81.12% <0%> (-0.65%)	⬇️
superset/views/core.py	71% <0%> (-0.22%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 613dcf5...1d1c89e. Read the comment docs.