Default locale is C #19

wch · 2014-10-03T02:09:14Z

This is in the eddelbuettel/ubuntu-rstudio image:

> Sys.getlocale(category = "LC_ALL")
[1] "C"

For best interoperability, it should be UTF-8.

Some information about it here:
http://jaredmarkell.com/docker-and-locales/
https://crosbymichael.com/dockerfile-best-practices-take-2.html

eddelbuettel · 2014-10-03T02:11:47Z

Another thing to add to r-base so that it bubbles up.

[ That said, I am a 7-bit snob now and rarely ever set these... But we probably should. ]

cboettig · 2014-10-07T21:57:29Z

Note, I set locale to C.UTF-8 as in the second example, rather than en_US.UTF-8 as in the first example; and just set the Debian base. (A summary of C.UTF-8 vs en_US.UTF-8 here, but happy for input on which locale @wch had in mind).

wch · 2014-10-08T02:36:59Z

I'm not an expert in this stuff, but I think that en_US.UTF-8 would be better, since it defines proper sorting for non-ASCII characters, while C_UTF-8 does not -- it probably just uses the unicode value for sorting.

For example, in en_US.UTF-8, all the a's with accents come before b:

> sort(c('A', 'a', 'Ä', 'ä', 'À', 'à', 'b'))
[1] "a" "A" "à" "À" "ä" "Ä" "b"

But it's not true in C.UTF-8:

> sort(c('A', 'a', 'Ä', 'ä', 'À', 'à', 'b'))
[1] "A" "a" "b" "À" "Ä" "à" "ä"

So I think that, despite the provincial-sounding label, en_US actually supports non-English languages better than C.

cboettig · 2014-10-08T04:17:44Z

@wch sounds reasonable to me.

For reasons that are not obvious to me, just switching C.UTF-8 to en_US.UTF-8 in this Dockerfile results in an error:

*** update-locale: Error: invalid locale settings:  LANG=en_US.UTF-8
2014/10/07 20:12:37 The command [/bin/sh -c dpkg-reconfigure locales     && locale-gen en_US.UTF-8     && /usr/sbin/update-locale LANG=en_US.UTF-8] returned a non-zero code: 255

No idea why, en_US.UTF-8 is on the list of locales returned by the command...

wch · 2014-10-08T05:26:41Z

This seems to work:

RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
   && locale-gen en_US.utf8 \
   && /usr/sbin/update-locale LANG=en_US.UTF-8

The starting state for /etc/locale.gen has en_US.UTF-8 commented out, along with all the other entries. Running dpkg-configure interactively and selecting en_US.UTF-8 has the same effect as this set of commands, I think.

Edit: FWIW, I found another Dockerfile that uses a similar strategy: https://registry.hub.docker.com/u/etna/drone-debian/dockerfile/

eddelbuettel · 2014-10-08T13:32:38Z

+1 -- I don't think I have ever seen C.UTF-8 in the wild anywhere. Not that I pay much attention though...

eddelbuettel · 2014-10-08T14:34:00Z

Blech:

root@e5b38b5f638c:/# du -csh /usr/share/locale/
87M     /usr/share/locale/
87M     total
root@e5b38b5f638c:/#

wch · 2014-10-09T00:22:53Z

Doesn't seem so bad when I do it:

$ docker run --rm -ti eddelbuettel/debian-r-base /bin/bash

root@1dbe56be3aa1:/# du -csh /usr/share/locale
43M /usr/share/locale
43M total

root@1dbe56be3aa1:/# apt-get install -qq -y locales

root@1dbe56be3aa1:/# du -csh /usr/share/locale
47M /usr/share/locale
47M total

root@1dbe56be3aa1:/# echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
>    && locale-gen en_US.utf8 \
>    && /usr/sbin/update-locale LANG=en_US.UTF-8
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.

root@1dbe56be3aa1:/# du -csh /usr/share/locale
47M /usr/share/locale
47M total

eddelbuettel · 2014-10-09T01:00:44Z

I was using the 'drd' (ie daily r-devel) which has more packages hence more po files. Anyway, on my home system it is 177 mb so ... that's just a cost of doing business.

I learned something new which may help shrink the image some more.

cboettig · 2014-10-09T19:42:25Z

Testing: docker run -it rocker/r-base R

> Sys.getlocale(category = "LC_ALL")

[1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C"

@wch Look good?

cboettig · 2014-10-09T19:49:09Z

For some reason, the rstudio image (and thus hadleyverse) object to the locale settings.

The container throws a warning on startup:

$ docker run --rm -it rocker/rstudio R
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

And likewise R complains as well:

R version 3.1.1 (2014-07-10) -- "Sock it to Me"
...
Type 'q()' to quit R.

During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C" 
2: Setting LC_COLLATE failed, using "C" 
3: Setting LC_TIME failed, using "C" 
4: Setting LC_MESSAGES failed, using "C" 
5: Setting LC_MONETARY failed, using "C" 
6: Setting LC_PAPER failed, using "C" 
7: Setting LC_MEASUREMENT failed, using "C"

and then defaults to the "C" locale:

> Sys.getlocale(category = "LC_ALL")
[1] "C"

eddelbuettel · 2014-10-09T21:14:08Z

That rings a bell but I don;t quite recall what to do. Should be a generic issue for Debian-based VMs etc though. Maybe as simple as setting it in /etc/bash/bashrc, or profile or ...

jangorecki · 2015-10-27T13:59:06Z

I get C locale on recent image r-base image.

docker run -it r-base

Sys.getlocale(category = "LC_ALL")
# [1] "C"

According to discussion here I should get US UTF8 so it looks like this issue needs to be reopened.

jangorecki · 2015-10-27T14:19:41Z

I used this SO answer to solve that issue on Ubuntu 14.04.

RUN locale-gen en_US.UTF-8  
ENV LANG en_US.UTF-8  
ENV LANGUAGE en_US:en  
ENV LC_ALL en_US.UTF-8

Sys.getlocale(category = "LC_ALL")
# [1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C"

I tried the same on official debian's r-base but it throws a lot of warnings about locale while build and in R console after run. So it cannot be directly applied to debian too.

cboettig · 2015-10-27T20:36:25Z

Really? On r-base I see:


> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux stretch/sid

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     
> Sys.getenv("LC_ALL")
[1] "en_US.UTF-8"
> Sys.getlocale(category="LC_ALL")
[1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C"
>

Are you sure you have the latest r-base image? (Not sure what you mean by 'official debian's r-base' or what warnings you're seeing either)

Yes, ubuntu and debian set locales differently; both are described in the link above. (And of course the debian way is also illustrated at the top of the r-base Dockerfile.

Does anyone else still see the C locale in r-base?

eddelbuettel · 2015-10-27T20:52:49Z

I get the same as Carl:

$ docker run --rm -ti r-base R -e 'sessionInfo()'

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux stretch/sid

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     
> 
> 
$

wch · 2015-10-27T20:55:46Z

@cboettig, I get the same result as you.

jangorecki · 2015-10-27T22:38:58Z

heh, I cannot reproduce it anymore... so it was likely some issue on my side, maybe overlapping name of an image I've build a while ago.

By default, the locale is set to just C, which then gets inherited by R and passed along to pandoc, and that causes all sorts of problems. - rstudio/rmarkdown#383 - rocker-org/rocker#19 - http://crosbymichael.com/dockerfile-best-practices-take-2.html

hakanai · 2018-09-13T07:07:16Z

So I think that, despite the provincial-sounding label, en_US actually supports non-English languages better than C.

Not sure how you jump to that conclusion.

What about languages where, for instance, "ä" or "å" are supposed to sort after "z"?

cboettig added a commit that referenced this issue Oct 7, 2014

set default UTF-8 locale in r-base (#19)

1e0321a

cboettig closed this as completed in af105cd Oct 12, 2014

This was referenced Nov 18, 2014

switch to jessie and drop dist-upgrade #87

Merged

Add an official image for r-base (R language) docker-library/official-images#310

Merged

cboettig mentioned this issue Feb 28, 2015

render with bibtex bibliography always throws error: pandoc-citeproc: could not find locale data for C-UTF-8 rstudio/rmarkdown#383

Closed

glin mentioned this issue Apr 25, 2019

Set default UTF-8 locale rstudio/r-docker#18

Merged

remlapmot mentioned this issue Aug 3, 2022

locale is set to nothing leading to different than expected sorting of vectors of character strings in R opensafely-core/r-docker#99

Closed

vpoten mentioned this issue Oct 30, 2023

Docker build from scratch fails Ensembl/ensembl-vep#1543

Closed

florisvdh mentioned this issue Sep 4, 2024

locale problem in qgis_query_algorithms() r-spatial/qgisprocess#218

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default locale is C #19

Default locale is C #19

wch commented Oct 3, 2014

eddelbuettel commented Oct 3, 2014

cboettig commented Oct 7, 2014

wch commented Oct 8, 2014

cboettig commented Oct 8, 2014

wch commented Oct 8, 2014

eddelbuettel commented Oct 8, 2014

eddelbuettel commented Oct 8, 2014

wch commented Oct 9, 2014

eddelbuettel commented Oct 9, 2014

cboettig commented Oct 9, 2014

cboettig commented Oct 9, 2014

eddelbuettel commented Oct 9, 2014

jangorecki commented Oct 27, 2015

jangorecki commented Oct 27, 2015

cboettig commented Oct 27, 2015

eddelbuettel commented Oct 27, 2015

wch commented Oct 27, 2015

jangorecki commented Oct 27, 2015

hakanai commented Sep 13, 2018

Default locale is C #19

Default locale is C #19

Comments

wch commented Oct 3, 2014

eddelbuettel commented Oct 3, 2014

cboettig commented Oct 7, 2014

wch commented Oct 8, 2014

cboettig commented Oct 8, 2014

wch commented Oct 8, 2014

eddelbuettel commented Oct 8, 2014

eddelbuettel commented Oct 8, 2014

wch commented Oct 9, 2014

eddelbuettel commented Oct 9, 2014

cboettig commented Oct 9, 2014

cboettig commented Oct 9, 2014

eddelbuettel commented Oct 9, 2014

jangorecki commented Oct 27, 2015

jangorecki commented Oct 27, 2015

cboettig commented Oct 27, 2015

eddelbuettel commented Oct 27, 2015

wch commented Oct 27, 2015

jangorecki commented Oct 27, 2015

hakanai commented Sep 13, 2018