Incorrectly appended FE0F for some emojis

[smolijar]

Describe the bug

kitty +kitten unicode_input --emoji-variation graphic appends FE0F to emojis, where it's not supposed to be. For example 🎉 should be written as 1F389 but is appended with FE0F by said kitten.

Some symbols require the FE0F symbol to be displayed graphically (as reported in #2139) and some don't. Not sure why is that, but I guess it is solved via is_emoji_presentation_base.

To Reproduce
Steps to reproduce the behavior:

1. kitty +kitten unicode_input --emoji-variation graphic
2. search party popper
3. insert
4. (is inserted with additional FE0F character, that can be removed in seperate editor, but not in terminal)

Expected behavior
Insert emoji that don't require it without FE0F.
Though it is displayed correctly in many applications, emojis lose color for some reason in kitty for me. Removing the appended character in editor fixes the display issues for kitty.

Screenshots

(red is emoji without color, colored red by fish since it's invalid syntax)

Environment details
Linux manjaro 5.9.16-1-MANJARO #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020 x86_64 GNU/Linux

kitty 0.19.3 created by Kovid Goyal
Linux manjaro 5.9.16-1-MANJARO #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020 x86_64
Manjaro Linux \r  (\n) (\l)
DISTRIB_ID=ManjaroLinux
DISTRIB_RELEASE=20.2.1
DISTRIB_CODENAME=Nibia
DISTRIB_DESCRIPTION="Manjaro Linux"
Loaded config files: /home/smolijar/.config/kitty/kitty.conf
Your Wayland compositor does not support server side window decorations, disabling Wayland. You can force Wayland support using the linux_display_server option in kitty.conf See https://drewdevault.com/2018/01/27/Sway-and-client-side-decorations.html for more information.
Running under: X11

Config options different from defaults:
active_border_color     Color(red=136, green=136, blue=136)
active_tab_background   Color(red=156, green=100, blue=254)
active_tab_font_style   (True, False)
active_tab_foreground   Color(red=25, green=25, blue=25)
background              Color(red=34, green=45, blue=50)
bold_font               Cascadia Code Bold
bold_italic_font        Cascadia Code Bold Italic
color0                  Color(red=25, green=25, blue=25)
color1                  Color(red=243, green=0, blue=95)
color10                 Color(red=151, green=224, blue=35)
color11                 Color(red=223, green=213, blue=97)
color12                 Color(red=156, green=100, blue=254)
color13                 Color(red=243, green=0, blue=95)
color14                 Color(red=87, green=209, blue=234)
color15                 Color(red=246, green=246, blue=238)
color2                  Color(red=151, green=224, blue=35)
color3                  Color(red=250, green=132, blue=25)
color4                  Color(red=156, green=100, blue=254)
color5                  Color(red=243, green=0, blue=95)
color6                  Color(red=87, green=209, blue=234)
color7                  Color(red=196, green=196, blue=181)
color8                  Color(red=97, green=94, blue=75)
color9                  Color(red=243, green=0, blue=95)
cursor                  Color(red=246, green=246, blue=236)
font_family             Cascadia Code
foreground              Color(red=196, green=196, blue=181)
inactive_border_color   Color(red=68, green=68, blue=68)
inactive_tab_background Color(red=50, green=67, blue=74)
inactive_tab_foreground Color(red=34, green=45, blue=50)
italic_font             Cascadia Code Italic
open_url_modifiers      2
selection_background    Color(red=246, green=246, blue=236)
selection_foreground    Color(red=25, green=25, blue=25)
tab_bar_edge            1
tab_bar_margin_width    10.0
tab_bar_style           powerline
url_color               Color(red=255, green=255, blue=255)
window_padding_width    FloatEdges(left=8.0, top=8.0, right=8.0, bottom=8.0)
Added shortcuts:
	f11 KeyAction(func='toggle_fullscreen', args=())
	control+c KeyAction(func='copy_to_clipboard', args=())
	control+t KeyAction(func='new_tab_with_cwd', args=())
	control+v KeyAction(func='paste_from_clipboard', args=())
	control+w KeyAction(func='close_window', args=())
	control+enter KeyAction(func='new_window_with_cwd', args=())
	control+tab KeyAction(func='next_tab', args=())
	shift+control+tab KeyAction(func='previous_tab', args=())
Changed shortcuts:
	shift+control+c KeyAction(func='send_text', args=['all', b'\x03'])
	shift+control+u KeyAction(func='kitten', args=['unicode_input', '--emoji-variation graphic'])
	shift+control+enter KeyAction(func='new_window_with_cwd', args=())

Additional context
Related to #2139

[kovidgoyal]

Why do you say it's not supposed to be there? It's a variation selector.
Emoji's have two presentation forms, text and graphic. Different emoji's
have different default forms. When you dont use --emoji-variation, the
kitten will produce the emojis with no variation selector, so they will
have their default appearance. When you specify text or graphic the
kitten will append the specified selector, thereby giving the result
exactly that appearance. If you have a graphic variation selector
appended to an emoji whose default appearance is anyway graphic, it does
no harm, since the appearance remains graphic.

The purpose of that option is to force either text or graphical
presentation, which it does.

[kovidgoyal]

As for the symbols losing color, that is a different issue, probably relating to font selection.

[smolijar]

Why do you say it's not supposed to be there? It's a variation selector.

Based on Codepoints from emojipedia.org, see party-popper vs recycling-symbol. The same behavior is alinged with Windows emoji keybord and KDE symbol selector as well.

If you have a graphic variation selector appended to an emoji whose default appearance is anyway graphic, it does no harm, since the appearance remains graphic.

What is the meaning of this is_emoji_presentation_base(codepoint) then if you should just append no matter what?

As for the symbols losing color, that is a different issue, probably relating to font selection.

Might be, but I was not able to reproduce it in any other application and emojis throughout the system work correctly. Even when pasted with the special char. Is there an option for kitty terminal that would enable removal of individual characters from the composed symbol?

[kovidgoyal]

This is already fixed, read the commit that closed the issue.

[smolijar]

mb, I missed the commit on second close. Thanks.