Skip to content

Commit

Permalink
[ML] Fix character set finder bug with unencodable charsets (#33234)
Browse files Browse the repository at this point in the history
Some character sets cannot be encoded and this was tripping
up the binary data check in the ML log structure character
set finder.

The fix is to assume that if ICU4J identifies that some bytes
correspond to a character set that cannot be encoded and those
bytes contain zeroes then the data is binary rather than text.

Fixes #33227
  • Loading branch information
droberts195 committed Aug 29, 2018
1 parent c0c53cf commit fded3bb
Showing 1 changed file with 9 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,15 @@ CharsetMatch findCharset(List<String> explanation, InputStream inputStream) thro
// deduction algorithms on binary files is very slow as the binary files generally appear to
// have very long lines.
boolean spaceEncodingContainsZeroByte = false;
byte[] spaceBytes = " ".getBytes(name);
for (int i = 0; i < spaceBytes.length && spaceEncodingContainsZeroByte == false; ++i) {
spaceEncodingContainsZeroByte = (spaceBytes[i] == 0);
Charset charset = Charset.forName(name);
// Some character sets cannot be encoded. These are extremely rare so it's likely that
// they've been chosen based on incorrectly provided binary data. Therefore, err on
// the side of rejecting binary data.
if (charset.canEncode()) {
byte[] spaceBytes = " ".getBytes(charset);
for (int i = 0; i < spaceBytes.length && spaceEncodingContainsZeroByte == false; ++i) {
spaceEncodingContainsZeroByte = (spaceBytes[i] == 0);
}
}
if (containsZeroBytes && spaceEncodingContainsZeroByte == false) {
explanation.add("Character encoding [" + name + "] matched the input with [" + charsetMatch.getConfidence() +
Expand Down

0 comments on commit fded3bb

Please sign in to comment.