[feature] Global customise-able d3 locale

[alanmcruickshank]

Feature description

The default locale (notably for currency values) is US English. For other countries where currency might be in Euros or Pounds (or who might want to use commas or stops differently), this makes number formatting hard.

To do this we would need to set a locale for d3 before we call d3.format() on anything as per the docs:
https://github.com/d3/d3-3.x-api-reference/blob/master/Localization.md

I think this should be an option in superset_config.py so that it can be optionally set for any given deployment.

As a bonus point, I think it could be neat to override locales for given metrics. This could be powerful for organisations which have data in multiple currencies. For example setting one column in yen, one in euros and one in pounds. If this is easy to implement we could just do this instead of enabling a global override locale.

Potential implementation

For the basic implementation we would need to pass through a json object which was set in a config variable to each visualisation.

in superset_config.py (for UK):

SUPERSET_D3_LOCALE = """
{
  "decimal": ".",
  "thousands": ",",
  "grouping": [3],
  "currency": ["£", ""],
  "dateTime": "%a %e %b %X %Y",
  "date": "%d/%m/%Y",
  "time": "%H:%M:%S",
  "periods": ["AM", "PM"],
  "days": ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"],
  "shortDays": ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"],
  "months": ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"],
  "shortMonths": ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
} """

and then somewhere in the visualisation javascript we would call:

// for d3 v4
 d3.formatDefaultLocale(definition)
// for d4 v3
 d3.locale(definition)

where definition is that JSON object which we defined in superset.py

column level formatting

If we wanted to also allow column level formatting instructions then one option is allowing a JSON payload on the SqlMetric model (and somewhere on the druid metric too?) which was then passed to d3.format() as an override locale.

@mistercrunch @graceguo-supercat @xrmx - does this sound like:
a) a good idea for a feature?
b) a good idea for implementation?
c) what's the best way to pass through the global locale config to the right javascript? I assume on page load but I don't know exactly where would make the most sense. Would appreciate some guidance.

Any other feedback welcome - I'm happy to do the legwork to actually implement it because we've got enough users that would value it.

[botasky11]

I agree this is a nice feature，recently I encountered this problem。

[tristix]

@alanmcruickshank
Is there a work around for showing £ instead of dollars at the moment? I try and format it as such on a big number slice and it just ignores it.

[alanmcruickshank]

@tristix - no work around as far as I know at the moment. This feature is relatively self contained so could be a good starter-feature if you fancy having a go at it?

[mistercrunch]

FYI there's an easy way to get backend configs to the frontend. Just add the SUPERSET_D3_LOCALE key name here:
https://github.com/apache/incubator-superset/blob/8dd052de4bbd8176f78318e3beb81526b1ecddd4/superset/views/base.py#L28

Then it should show up in the redux state on the frontend.

[tomsej]

Any progress on this issue @alanmcruickshank?

[craigappl]

I think the bigger issue here would be to align locales in Superset and D3. D3 already supports localized number and time formats.

There seems to be an alignment issue between Superset and D3 locales. For example, Brazilian Portuguese in Superset is PT_BR and the D3 pt-BR. I think the most robust solution would be to have each chart call the D3 locale based on the user's Superset locale when the chart is displayed. You can see in the sources below that D3 already has everything we need, we just aren't applying it to the user's selected locale.

FYI @mistercrunch and @csc5k

Sources:
Superset Translations: https://github.com/apache/incubator-superset/tree/master/superset/translations
D3-Format: https://github.com/d3/d3-format/blob/master/locale/pt-BR.json
D3-Time-Format: https://github.com/d3/d3-time-format/blob/master/locale/pt-BR.json

[benleibert]

Being able to use D3's time-format as @craigappl described above would definitely be helpful.

[avinasht1]

+1 to this feature

[Benji81]

Any update on this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[PeterCahn]

@tristix I have found a way to change the default locale for the number formatting in the Docker image.

You need to modify the file superset/assets/node_modules/d3-format/src/defaultLocale.js which is there after the command npm ci which installs the node_modules.

At this point I used sed to edit that file, like in these commands:

sed -i 's/\([[:space:]]*\)decimal: "\.",/\1decimal: ",",/g' /home/superset/superset/assets/node_modules/d3-format/src/defaultLocale.js
sed -i 's/\([[:space:]]*\)thousands: ",",/\1thousands: "\.",/g' /home/superset/superset/assets/node_modules/d3-format/src/defaultLocale.js
sed -i 's/\([[:space:]]*\)currency: \["$"\(.*\)/\1currency: \["€"\2/g'  /home/superset/superset/assets/node_modules/d3-format/src/defaultLocale.js

In this way the npm run build apply the modifications in the javascripts used by Superset.

I am able also to change the file defaultLocal.js at runtime when the image is running in development mode, with the folder superset mounted and accessible from inside the container.

I am afraid that even if you are not using Docker, the only way is to rebuild everything.

Hope this help, waiting for the new feature.

[SWoto]

Thanks @PeterCahn, it worked here.
I changed some other stuffs too and it worked.

decimal: ",",
  thousands: ".",
  grouping: [3],
  currency: ["R$", ""],
  dateTime: "%d/%m/%Y %H:%M:%S",
  date: "%d/%m/%Y",
  time: "%H:%M:%S",
  periods: ["AM", "PM"],
  days: ["Domingo", "Segunda", "Terça", "Quarta", "Quinta", "Sexta", "Sábado"],
  shortDays: ["Dom", "Seg", "Ter", "Qua", "Qui", "Sex", "Sáb"],
  months: ["Janeiro", "Fevereiro", "Março", "Abril", "Maio", "Junho", "Julho", "Agosto", "Setembro", "Outubro", "Novembro", "Dezembro"],
  shortMonths: ["Jan", "Fev", "Mar", "Abr", "Mai", "Jun", "Jul", "Ago", "Set", "Out", "Nov", "Dez"]});

[andy-clapson]

+1 to this feature

[SWoto]

@mistercrunch, please reopen this issue.

[jfunez]

Any update?

[nfbrentano]

Any update?

[simski]

Any updates?

[brandfocus]

For anyone coming across this the easiest way is to build a new image for Superset:

Dockerfile

FROM apache/superset

COPY ./custom/D3Formatting.ts /app/superset-frontend/packages/superset-ui-chart-controls/src/utils/D3Formatting.ts

USER root

RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
    apt-get install -y nodejs && \
    npm install -g npm@7

ENV NODE_OPTIONS=--max_old_space_size=24576

ENV CYPRESS_INSTALL_BINARY=0

RUN cd /tmp && bash frontend-mem-nag.sh && \
    cd /app/superset-frontend && \
    npm ci --unsafe-perm=true --allow-root && \
    sed -i 's/\([[:space:]]*\)currency: \["$"\(.*\)/\1currency: \["£"\2/g'  /app/superset-frontend/node_modules/d3-format/src/defaultLocale.js && \
    npm run build --unsafe-perm=true --allow-root && \
    rm -rf node_modules

[matheusbsilva]

Any updates on how to override locales for months?