EXPORT_CSV decimal separator format doesn't work for Decimal MySQL column

[mbcsa]

Description

"Export to CSV" does not respect EXPORT_CSV decimal separator option for a MySQL Decimal(x,y) column. I see that column is mapped as NEWDECIMAL.

To make it work I have to CAST to FLOAT on SQL query. Then it's mapped as FLOAT and respects decimal separator as configured.

How to reproduce the bug

1. Configure superset_config.py with

CSV_EXPORT = {
    'encoding': 'utf-8',
    'sep': ';',
    'decimal': ','
}

2. Create a query with a Decimal(x,y) field, Save and click Explore
3. In Dataset options, click "Edit Dataset"
   
4. Verify that decimal column is shown as "NEWDECIMAL"
   
5. Configure "$,.2f" as format for column
   
6. Save Chart and add to a new Dashboard
7. Go to Dashboard and Export CSV
8. Verify that decimal separator is not being formatted as configured in step 1
   

Expected results

Export CSV Option USES configured decimal separator when Datasource Column has as Decimal data type in a MySQL Datasource

Actual results

Export CSV Option DOES NOT use configured decimal separator when a Datasource Column has as Decimal data type in a MySQL Datasource

Environment

• browser type and version: User-Agent | Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0
• superset version: Superset 0.0.0dev (SHA: 80f5154)
• python version: Python 3.9.5
• node.js version: v14.18.3
• any feature flags active:

FEATURE_FLAGS = {
    "ALERT_REPORTS": True,
    "ENABLE_TEMPLATE_PROCESSING": True
}

• Datasource MySQL version: mariadb-10.5.9

Checklist

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[frheinze]

I ran into the identical bug, I would really appreciate a fix :)

[rusackas]

Has anyone managed to validate that this is an issue on Superset 2.0?

[fyod0r1]

@rusackas have the same issue

[nfalco79]

@rusackas I can confirm that with superset 2.1.0 we have the same issue. The CSV separator is taken (from , to ;) but the decimal separator for money column in CSV does not work.
We have customize the currency column with

    CSV_EXPORT = {
        "encoding": "utf-8",
        "sep": ";",
        "decimal": ","
    }
    D3_FORMAT = {
        "decimal": ",",
        "thousands": ".",
        "currency": ["€", ""]
    }

[fyod0r1]

Actually there is a bug. I made some changes to the code on my local repository, but I'm not sure if this solves the root of the problem, so I didn't make a PR. The problem was in the function that handled python escape characters on the backend, in the superset/utils/csv.py file.

[rusackas]

Actually there is a bug. I made some changes to the code on my local repository, but I'm not sure if this solves the root of the problem, so I didn't make a PR. The problem was in the function that handled python escape characters on the backend, in the superset/utils/csv.py file.

Feel free to open a PR, or even a Draft PR, to better illustrate the point, and maybe it can lead to a resolution 🤞

[villebro]

I also ran into this - the issue appears to be that Superset changes decimals to string rather than number during JSON serialization. I will work on a fix for this.

[villebro]

@mbcsa @frheinze I'm not sure if you noticed the PR #24241 which attempts to fix this issue. During code review a question came up which dialect this is happening on. Can you confirm which mysql driver/dialect you're getting this bug with?