Can’t type astral plane characters

[chris-morgan]

Describe the bug

Typing characters from outside the Basic Multilingual Plane produces a sequence of U+FFFD replacement characters instead.

Environment (please complete the following information):

• OS: Windows 10, build 19041.329
• Version: wezterm 20200620-160318-e00b076c-6-g076cf3d1

To Reproduce

One approach: Windows 10’s emoji picker, Windows+., and choose one, like 💤. You probably get ��.

Another approach: WinCompose, then enter an emoji like ComposeComposezzz. You get ����.

Configuration

None.

Expected behavior

The emoji should be there.

Additional information

The typical cause of this problem is the program taking the character in as the two UTF-16 code units via WM_CHAR events, and eagerly turning them into UTF-8 rather than waiting to merge the surrogate pair. I’m not sure exactly why WinCompose yields four replacement characters and the emoji picker two, though I have a vague notion of the general reason why (WinCompose emits events in an older way).

Good related reading on solutions: vim/vim#2800, and other issues that link to it also.

[wez]

If you have WSL installed and have it set eg: LANG=C.utf-8 then the emoji are successfully passed to the unix shell, even though the IME may render weird weird characters over the top of wezterm as an intermediate step.

I don't believe that Windows supports typing emoji into either powershell or cmd.exe; I see the unicode replacement character in the Microsoft Terminal Preview, cmd.exe windows and powershell.exe windows (a quick google shows lots of people having problem with this).

If I write the emoji to a file using eg: vim under wsl I can cat or type the file from both wsl, cmd.exe and powershell.exe and see the emoji in both wezterm and the Microsoft Terminal Preview. Vanilla cmd.exe doesn't like the emoji.

It's not clear to me that wezterm is doing something wrong here!

[wez]

based on the comments from #232 I'm closing this issue, as it sounds like the OP isn't in a position to actively troubleshoot this right now, but it also doesn't sound like wezterm is necessarily at fault per my comment above.
If someone else is experiencing this and has an example of a case where cmd/powershell are successfully typing emoji inputs then I'm happy to revisit and see what we might be able to do about this.

[wez]

See also: microsoft/terminal#1503

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.