Issues upgrading to 0.17.0 - Explore migrations not functioning as expected

[alanmcruickshank]

Superset version

Upgrade from 0.15.4 to 0.17.0

Expected results

Migration from explore v1 to explore v2, introduces nice new features but core functionality remains the same.

Actual results

• Time filters, field choices, where clauses, having clauses and format arguments all migrate fine.
• Filters don't survive when first opened in explore v2, resulting in no filters being applied, these all have to be recreated.
• Within the NVD3 charts, most checkboxes migrate in the opposite setting than they should. e.g. Contribution being unchecked before becomes checked, likewise with reducing X ticks and the other checkbox-like fields.

Steps to reproduce

Take productions database running 0.15.4 with slices which have filiters and checkboxes applied and then upgrade to 0.17.0

[mistercrunch]

Wait how did that work for us?!?!?!

[mistercrunch]

Running tests...

[mistercrunch]

I started with a fresh DB, pointed to 0.15.4, upgraded the db and loaded examples, moved to 0.17.0 and upgraded again, and things look good.

Is it possible that you've been pointing to [bad versions of] master on this particular database?

[alanmcruickshank]

This database has never been pointing at master, but it has been running since some fairly early versions of caravel. On a quick inspection we've definitely been running since caravel 0.8.7.

Some of our slices are quite old so it's possible that it's specific to our data structures. Here's an example of the params on one of our slices generated on or before 0.15.4, but which renders wrong on 0.17.0. In this case specifically the contribution flag.

{
    "add_to_dash": "false", 
    "bar_stacked": "y", 
    "bottom_margin": "50", 
    "collapsed_fieldsets": "", 
    "contribution": "false", 
    "datasource_id": "41", 
    "datasource_name": "shipment_package", 
    "datasource_type": "table", 
    "flt_col_0": "courier", 
    "flt_eq_0": "", 
    "flt_op_0": "in", 
    "goto_dash": "false", 
    "granularity_sqla": "dispatch_date", 
    "groupby": [
        "courier", 
        "patched_courier_delivery_service"
    ], 
    "having": "", 
    "limit": "0", 
    "line_interpolation": "linear", 
    "metrics": [
        "count"
    ], 
    "new_dashboard_name": "", 
    "new_slice_name": "", 
    "num_period_compare": "", 
    "period_ratio_type": "growth", 
    "rdo_save": "overwrite", 
    "reduce_x_ticks": "false", 
    "resample_fillmethod": "", 
    "resample_how": "", 
    "resample_rule": "", 
    "rich_tooltip": "y", 
    "rolling_periods": "", 
    "rolling_type": "None", 
    "save_to_dashboard_id": "", 
    "show_bar_value": "false", 
    "show_brush": "false", 
    "show_controls": "false", 
    "show_legend": "y", 
    "since": "45 days ago", 
    "slice_id": "243", 
    "slice_name": "Packages by Courier Service", 
    "time_compare": "", 
    "time_grain_sqla": "day", 
    "timeseries_limit_metric": "", 
    "until": "now", 
    "userid": "1", 
    "viz_type": "bar", 
    "where": "", 
    "x_axis_format": "smart_date", 
    "x_axis_label": "", 
    "x_axis_showminmax": "y", 
    "y_axis_format": ",", 
    "y_axis_label": "", 
    "y_axis_zero": "false", 
    "y_log_scale": "false"
}

post migration to 0.17.0 the params field remains the same:

{
    "add_to_dash": "false", 
    "bar_stacked": "y", 
    "bottom_margin": "50", 
    "collapsed_fieldsets": "", 
    "contribution": "false", 
    "datasource_id": "41", 
    "datasource_name": "shipment_package", 
    "datasource_type": "table", 
    "flt_col_0": "courier", 
    "flt_eq_0": "", 
    "flt_op_0": "in", 
    "goto_dash": "false", 
    "granularity_sqla": "dispatch_date", 
    "groupby": [
        "courier", 
        "patched_courier_delivery_service"
    ], 
    "having": "", 
    "limit": "0", 
    "line_interpolation": "linear", 
    "metrics": [
        "count"
    ], 
    "new_dashboard_name": "", 
    "new_slice_name": "", 
    "num_period_compare": "", 
    "period_ratio_type": "growth", 
    "rdo_save": "overwrite", 
    "reduce_x_ticks": "false", 
    "resample_fillmethod": "", 
    "resample_how": "", 
    "resample_rule": "", 
    "rich_tooltip": "y", 
    "rolling_periods": "", 
    "rolling_type": "None", 
    "save_to_dashboard_id": "", 
    "show_bar_value": "false", 
    "show_brush": "false", 
    "show_controls": "false", 
    "show_legend": "y", 
    "since": "45 days ago", 
    "slice_id": "243", 
    "slice_name": "Packages by Courier Service", 
    "time_compare": "", 
    "time_grain_sqla": "day", 
    "timeseries_limit_metric": "", 
    "until": "now", 
    "userid": "1", 
    "viz_type": "bar", 
    "where": "", 
    "x_axis_format": "smart_date", 
    "x_axis_label": "", 
    "x_axis_showminmax": "y", 
    "y_axis_format": ",", 
    "y_axis_label": "", 
    "y_axis_zero": "false", 
    "y_log_scale": "false"
}

but when we open that slice in the explore view, these are the options which are applied, and you can see that contribution and a load of the other options are checked rather than using the false property within the parameters:

As an example for filters, here's another params field for a different slice where I've trimmed it down to just he filter fields:

as at 0.15.4:

...
    "flt_col_0": "template", 
    "flt_col_1": "template", 
    "flt_eq_0": "", 
    "flt_eq_1": "delivery-impending-ongoing,delivery-shipping-soon", 
    "flt_op_0": "in", 
    "flt_op_1": "in", 
    "goto_dash": "false", 
    "granularity_sqla": "date", 
    "groupby": [
        "template"
    ], 
   ...

params post migration to 0.17.0 (unchanged)

...
    "flt_col_0": "template", 
    "flt_col_1": "template", 
    "flt_eq_0": "", 
    "flt_eq_1": "delivery-impending-ongoing,delivery-shipping-soon", 
    "flt_op_0": "in", 
    "flt_op_1": "in", 
    "goto_dash": "false", 
    "granularity_sqla": "date", 
    "groupby": [
        "template"
    ], 
...

as rendered in 0.17.0:

It doesn't seem to be as simple as malformed JSON because the groupby argument migrates just fine. Any ideas?

[alanmcruickshank]

Found the issue.

Migrations hadn't run properly. You can tell from the fact that the params field for the slice before and after migrations hadn't changed. As part of the migrations all of the JSON blobs are upgraded rather than it being performed on read.

Running migrations manually worked a charm - the reason they didn't run is related to our own deployment methods and not related to superset. Good work on this update guys - I love it.

[xiaoidea]

@alanmcruickshank
Could you share how to upgrade caravel to superset.
db upgrade does not take effect. the table structure not changed from caravel's to superset's