Assumes files with non-utf-8 encodings are binary

[benogle]

It detects these files as binary:

https://github.com/benogle/encodings/blob/master/big5.txt
https://github.com/benogle/encodings/blob/master/bom_utf-16le.txt

And likely many others in https://github.com/benogle/encodings

[gjtorikian]

Damn, this looks like a fun problem. I wish you hadn't opened this. 😣

The problem can be traced to this line. 16/32 BE and LE were each enough to add. The Big5, GB, and KR files have some crazy byte detection, and use the 0x00 code point, so they're being unfairly flagged as binary.

[alexkozy]

https://github.com/ashtuchkin/iconv-lite/blob/master/encodings/tables/big5-added.json
There is one more file which is detected as binary. First 32 bytes of this file has 10.15625% of suspicious bytes.

[SamVerschueren]

Also a file with an emoji is flagged as binary.

This is a text file 🌈