[R-Forge #4903] set encoding in fread

[arunsrinivasan]

Submitted by: michele de meo; Assigned to: Nobody; R-Forge link

It's useful to specify the encoding before importing a csv.
For example:

fread(... , encoding='UTF8').

In this way we can avoid the boring use of file function in read.table (not supported in fread ):

read.table(file("mycsv.csm", encoding='UTF8'), ... ).

Michele De Meo

[EDiLD2]

+1 for this

[bnbwn]

+1

[stanasa]

+1

[sirvydasdagys]

+1

[harmonica2]

+1

[zeltak]

+1

[MarcinKosinski]

+1

[ghost]

+1 Would be really usefull. I started to hack locally the code but it's too hard for me because it call a C function (readfile) and I don't know how (for the moment) how to handle encoding in C. If any one some knowledge in C...

[grgurev]

+1

[lucarno]

+1

[seyedamo]

+1

[jcizel]

+1

[Alectoria]

+1

[tophcito]

+1

[clarkdk]

+1

[JohnsonHsieh]

+1

[whizzalan]

+1

[ysgit]

+1

[ZeroStack]

+1

[rshmyrev]

+1

[dbuijs]

+1, and in the meantime, a workaround:

You need iconv available from the console. You can check this with the following command:

Sys.which("iconv")

As long as this gives you a path to a binary, the following will work:

# From ISO-8859-1, To UTF-8
new.data.table <- fread("iconv -f ISO-8859-1 -t UTF-8 mytextfile.txt")

[shrektan]

• 1

Sent from my iPhone

On Jun 16, 2015, at 15:13, dbuijs notifications@github.com wrote:

+1, and in the meantime, a workaround:

You need iconv available from the console. You can check this with the following command:

Sys.which("iconv")
As long as this gives you a path to a binary, the following will work:

From ISO-8859-1, To UTF-8

new.data.table <- fread("iconv -f ISO-8859-1 -t UTF-8 mytextfile.txt")
—
Reply to this email directly or view it on GitHub.

[panda2727]

+1

[duf59]

+1

[romunov]

+1

note however that in ?Encoding, UTF-8 is used (not UTF8).

[rentrop]

+1

[leoluyi]

+1

[arunsrinivasan]

Tl;dr

Could you please test your files with encoding = "UTF-8" or "Latin-1" and write back as to whether it solves the issue (especially on windows)? Thanks.

---

About the fix:

Looking at read.table() function, the encoding is set upfront with Encoding(..) <- .... And looking at the source of Encoding<-, it calls an internal function do_setencoding. Looking at the source of that function, https://github.com/wch/r-source/blob/ca5348f0b5e3f3c2b24851d7aff02de5217465eb/src/main/util.c#L1115, it seems quite straightforward to fix this (I think).. through the use of mkCharLenCE function that R's C-API seems to expose.

fread() gains a new argument encoding with valid values unknown, UTF-8 and Latin-1. The default behaviour hasn't changed. Windows user will have to set the encoding argument explicitly. This could change in the future (perhaps also testing for performance hit would be a deciding factor).

[clarkdk]

Arun, There's a typo in README.md

27 . fread() gains eocnding argument. ... eocnding --> encoding

[jangorecki]

if speaking about the typos there is one missing () next to flush.console in bmerge function:

data.table/R/bmerge.R

Line 102 in 7f78b1b

if (verbose) {cat("done in",round(proc.time()[3]-last.started.at,3),"secs\n");flush.console}

[dbuijs]

Thank you for this!!!!

A question of clarification: does the encoding option in fread just set the marked encoding, or does it convert from what you've declared to the system's native encoding?

[andreasio]

I have an iso-8850-1 windows file, that I can't fread correctly on linux. This workaround (#563 (comment)) works. And read.csv works (i.e. produces the correct øæå letters, instead of e.g. \xe6), but fread(... encoding = "Latin-1") does not.