DRAFT: Expand zero-width support for ZWJ and emoji sequences #85

jquast · 2023-09-30T06:00:04Z

Bugfix: wcswidth() for zero-width join characters such as used in emoji
sequences, and variations such as emoji skin tone, as well as many other
non-printable characters now correctly identified as zero-width.
Enhancement: new function width(), a copy of wcswidth() without
the POSIX compliance to return total width of -1 for any C0 or C1 control
characters, those characters measured as width 0 by this new function.

Discussion,

This implementation of ZJW for emojis does not qualify whether the sequence is supported !! This is possible because of the publication of "recommended" sequences in emoji-zwj-sequences.txt. I experimented with creating a large optimized regex of all supported emoji sequences as part of code generation, but I refrained from implementing it -- it seems like a large cost to lookup every possible sequence, even an optimized regex (using triregex library) is many tens of thousands of characters. Whereas this implementation increments idx past the next character, assuming they are cojoined is very high-performing and easy to implement
It is possible to provide a EMOJI_ZWJ_VERSION environment variable, which when undefined is set accordingly to the same release of UNICODE_VERSION. This would require implementing "recommended sequence" checking for the given version level, and returning the length as assumed it would be rendered when supported, for example 2 wide co-joined vs. unsupported, 4-wide with two side-by-side.
I hope the results of automatic testing with popular terminals will help decide what weight the performance vs. correctness tradeoff. I predict most popular terminals or libraries do not support any ZWJ at all.

Work remaining,

Update bin/wcwidth-browser.py to use and render tables of EMOJI_ZWJ_SEQUENCES by version.
Write automatic tests for ZWJ behavior and and new width() function, returning to 100% coverage.
Write interactive program that automatically checks all of those sequences, and some other ZWJ such as Bengali phrases with the current terminal, and writes some kind of report file about current os+terminal and detected wide-character and ZWJ emoji support level, to support the activity like was done with this tool and report https://github.com/jquast/ucs-detect#wcwidth-support-in-terminals
test some terminals, the most popular, how well do they support these sequences, at what "Emoji version" level, if any?
update docs/intro.rst to suggest wcwidth.width() function in the example. Write small section about POSIX wcwidth and wcswidth functions, why they exist for function signature compatibility for POSIX, but why they are not suggested due to the final -1 return value on error.
Update docstrings of wcwidth and wcswidth with deprecation warning in docstring only, to suggest using width() instead, with reference to the POSIX section for the reason.

wcwidth/wcwidth.py

codecov · 2023-09-30T06:05:57Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Files	Coverage Δ
wcwidth/emoji_zwj_sequences.py	`100.00% <ø> (ø)`
wcwidth/table_wide.py	`100.00% <ø> (ø)`
wcwidth/table_zero.py	`100.00% <ø> (ø)`
wcwidth/unicode_versions.py	`100.00% <ø> (ø)`
wcwidth/wcwidth.py	`94.68% <83.33%> (-5.32%)`	⬇️

📢 Thoughts on this report? Let us know!.

also partial implementation of --automatic

bin/wcwidth-browser.py

        return fmt.format(name_len=style.name_len,
                          ucs_printlen=UCS_PRINTLEN,
                          delimiter=delimiter,
-                          name=name,
-                          ucs=disp_ucs,
-                          val=val)
+                          name_val=name_val,
+                          ucs=ucs)


bin/wcwidth-browser.py


 # local
-from wcwidth import ZERO_WIDTH, wcwidth, list_versions, _wcmatch_version
+import wcwidth
+from wcwidth import ZERO_WIDTH, wcwidth, list_versions, _wcmatch_version, EMOJI_ZWJ_SEQUENCES


bin/wcwidth-browser.py


 # local
-from wcwidth import ZERO_WIDTH, wcwidth, list_versions, _wcmatch_version
+import wcwidth


bin/wcwidth-browser.py


-    def run(self, writer, reader):
+    def run(self, writer, reader, automatic=False):


bin/wcwidth-browser.py

+        # sequence clears to end-of-line
+        clear_eol = self.term.clear_eol
+        # sequence clears to end-of-screen
+        clear_eos = self.term.clear_eos


bin/wcwidth-browser.py

it's too bad the tables aren't more consistent.

ZOC terminal gets nutty with vertical position instead of horizontal around some characters. I think it is some terrible mistake, but it is closed source, bugs are private, and it costs $80 !

bin/wcwidth-browser.py

+            writer(self.text_entry(ucs, name), end='')
+            ny, nx = self.term.get_location()
+            assert y == ny, ("y position is out of control on ucs value=", hex(ord(ucs)) if len(ucs) == 1 else repr(ucs))
+            if y == ny:


bin/wcwidth-browser.py

+                             attr_text=self.term.bright_red,
+                             name_len=self.screen.style.name_len)
+
+        def measure_distance(ucs: str, name: str):


tests/test_core.py

@@ -59,10 +61,11 @@
    expect_length_phrase = sum(expect_length_each)

    # exercise,
-    length_phrase = wcwidth.wcswidth(phrase, end)
+    length_each = tuple(map(wcwidth.wcwidth, phrase))


bin/wcwidth-browser.py

+            for test_type in ('narrow', 'wide', 'zero', 'emoji-zwj'):
+                self.initialize_page_data(test_type)
+                self.screen.change_test_type(test_type)
+                page_idx = page_offset = idx = offset = 0


bin/wcwidth-browser.py

+            for test_type in ('narrow', 'wide', 'zero', 'emoji-zwj'):
+                self.initialize_page_data(test_type)
+                self.screen.change_test_type(test_type)
+                page_idx = page_offset = idx = offset = 0


jquast · 2023-10-17T17:53:37Z

Sorry, no need to review, I have been using this branch for multiple ideas. I will carve them into smaller PR's, beginning with the automatic tests.

I am just performing automatic tests against popular terminals, now, and committing their results, when I'm finished with that I'll start smaller PR's, thank you!

Broaden zero-width definition and add zjw support

f03bdad

jquast changed the title ~~Broaden zero-width definition and add zjw support~~ Expand zero-width coverage, add support for ZWJ/emoji sequences Sep 30, 2023

github-advanced-security bot found potential problems Sep 30, 2023

View reviewed changes

wcwidth/wcwidth.py Fixed Show fixed Hide fixed

wcwidth/wcwidth.py Fixed Show fixed Hide fixed

wcwidth/wcwidth.py Fixed Show fixed Hide fixed

jquast added 9 commits September 30, 2023 02:18

some cleanup

fc58ea3

cleanup

2e4d9af

codefix width() and docfixes

35ec56e

remove unused import

79640b2

rollback render timestamp

33ca88b

change name 'None' => 'NULL'

e3a438f

Change None->NULL in codegened files

bebd8e9

remove unused legacy versions

dca7189

add wcwidth/emoji_zwj_sequences.py

5dcf133

jquast changed the title ~~Expand zero-width coverage, add support for ZWJ/emoji sequences~~ DRAFT: Expand zero-width coverage and support for ZWJ/emoji sequences Sep 30, 2023

jquast changed the title ~~DRAFT: Expand zero-width coverage and support for ZWJ/emoji sequences~~ DRAFT: Expand zero-width support for ZWJ and emoji sequences Sep 30, 2023

jquast marked this pull request as ready for review September 30, 2023 07:12

jquast added the draft-do-not-merge label Sep 30, 2023

jquast added 5 commits October 3, 2023 06:34

bugfix 'Zk' category, for 0-width esp.

b7c4905

also partial implementation of --automatic

remove docopt from requirements

1077b62

remove #trieregex

9e22c0c

some 'Sk' fixes and a small table size optimization

08242d0

add --automatic tests, mostly works, needs testing

2575ea2

github-advanced-security bot found potential problems Oct 3, 2023

View reviewed changes

jquast added 7 commits October 16, 2023 22:22

small refactor

36f69ae

include yaml development requirements for automatic tests

497acdb

An experimental script

5efc899

it's too bad the tables aren't more consistent.

detailed category fixes ... to be documented

723f55b

Collect data from terminals

91d2f82

Allow crazy ypos screwup for ZOC terminal

8f065c9

ZOC terminal gets nutty with vertical position instead of horizontal around some characters. I think it is some terrible mistake, but it is closed source, bugs are private, and it costs $80 !

Add some terminal reports

3443350

ren

fa5f520

github-advanced-security bot found potential problems Oct 17, 2023

View reviewed changes

jquast added 3 commits October 18, 2023 15:39

Add 'Mc' to zero-width category

bbf5bb8

Document a bit more about bin/wcwidth-browser.py

570f39d

Documentation and tests (draft)

1f1443b

jquast closed this Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRAFT: Expand zero-width support for ZWJ and emoji sequences #85

DRAFT: Expand zero-width support for ZWJ and emoji sequences #85

jquast commented Sep 30, 2023 •

edited

Loading

codecov bot commented Sep 30, 2023 •

edited

Loading

jquast commented Oct 17, 2023


		def run(self, writer, reader):
		def run(self, writer, reader, automatic=False):

DRAFT: Expand zero-width support for ZWJ and emoji sequences #85

DRAFT: Expand zero-width support for ZWJ and emoji sequences #85

Conversation

jquast commented Sep 30, 2023 • edited Loading

codecov bot commented Sep 30, 2023 • edited Loading

Codecov Report

jquast commented Oct 17, 2023

jquast commented Sep 30, 2023 •

edited

Loading

codecov bot commented Sep 30, 2023 •

edited

Loading