❇️ Small improvement over Unicode detection

[Ousret]

Tiny improvement that allow small Unicode payload to be detected with more accuracy.

import charset_normalizer

payload = "😀".encode("utf_8")

results = from_bytes(payload)

print(results.best().encoding)  # print "utf_8"!

chardet has trouble with those cases. Any Unicode payload containing emoticons makes the detection fail. Probably their prober that need to be retrained with more up-to-date stats.

>>> chardet.detect("😀".encode("utf_8"))
{'encoding': None, 'confidence': 0.0, 'language': None}

That a tiny improvement but noticeable/appreciable.

[Ousret]

This will land in 2.0.5

Prior to that, charset_normalizer answered (not correctly) with a single emoticon.

{'encoding': 'cp1125', 'language': 'Russian', 'confidence': 1.0}