Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Minor improvements for the MD #101

Merged
merged 3 commits into from
Sep 16, 2021
Merged

✨ Minor improvements for the MD #101

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ad68e8c
:sparkle: Ignore the ASCII subtitute character in the UnprintablePlugin
Ousret Sep 16, 2021
aece497
:sparkle: Ignore common ASCII sep used in all sub CP for SuspicousRan…
Ousret Sep 16, 2021
3f79dea
:art: reformat md.py
Ousret Sep 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion charset_normalizer/md.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ def feed(self, character: str) -> None:
character not in {"\n", "\t", "\r", "\v"}
and character.isprintable() is False
and character.isspace() is False
and ord(character) != 0x1A # Why? Its the ASCII substitute character.
):
self._unprintable_count += 1
self._character_count += 1
Expand Down Expand Up @@ -218,7 +219,27 @@ def eligible(self, character: str) -> bool:
def feed(self, character: str) -> None:
self._character_count += 1

if character.isspace() or is_punctuation(character):
if (
character.isspace()
or is_punctuation(character)
or character
in [
"<",
">",
"=",
":",
"/",
"&",
";",
"{",
"}",
"[",
"]",
",",
"|",
'"',
]
):
self._last_printable_seen = None
return

Expand Down