Fix unicode display: CJK, powerline, tmux

[tibotiber]

Hi! This is a follow up PR from #1376, taking 2e58e56 into consideration. I have isolated the issue with braille characters for vtop by decorating the corresponding nodes with a braille-node class additionally to the existing unicode-node class. It is ready to merge to the best of my knowledge.
Cc @rauchg

WIP summary:

• powerline wide unicode glyphs display
• CJK chars display, wc-node css
• multiple rendering of CJK in some cases, see Fix display of unicode characters #1376 (comment) / Fix display of unicode characters #1376 (comment)
• tmux empty prompt width, see screenshot below:
  

[albinekb]

Wow, this is awesome! Will try it tonight 👍

[tibotiber]

Screenshots of the result here: vtop and unicode + tmux

[rauchg]

Only issue is: shouldn't we respect the grid size for emojis? Both Terminal.app and iTerm seem to overlap them:

I wonder if by taking space wider than a character we would be breaking some UIs ?

[rauchg]

Also, since other Terminals limit the width of Emojis, I worry that we'd be displaying them not as nicely in Hyper, because they'd essentially have an extra space (think Yarn)

[tibotiber]

@rauchg: great point about emojis, one important breaking UI is the cursor position. However we need to address at least powerline unicode characters (e.g. #1336 (comment)), which is handled properly by iTerm as well. Emojis being spread over codepoints, it would be easier to isolate the powerline glyphs and give them more space. We can choose to apply flexible or double spacing, I would suggest double to keep the cursor position in check. From what I can see, this is the way iTerm handles it too.

After that, I think I should remove css decoration for braille, since it will fall in the normal unicode styling (?).

Sidenote: This is getting interesting, powerline glyphs are stored in the Unicode Private Use Area, along with medieval scripts, Klingon, and various languages of the Middle-earth... Should we display that with fix width or double width like powerline glyphs? 😂

[rauchg]

@tibotiber that's a very good question. I think ultimately it would depend on each glyph in the private use area whether they want double or single width.

Let's do this: for now, let's give all private use area double (which would encompass Powerline glyphs), and then if a request comes up, we can think about adding a config flag for specifying ranges that you want to be double width

[tibotiber]

OK, I'll push changes today (it's Saturday here already). I'll remove the braille part then.

[tibotiber]

Update: The issue with powerline glyphs is not a width issue. I couldn't get this to work in a way that supports basic powerline glyphs as well as user defined ones, so I checked iTerm a bit more and realised they do not treat powerline glyphs (new or old) as wide chars. What happens is wide unicode simply overflows on the next char. See screenshots for iTerm and Hyper:


If you look at the cat above, both terminals treat the glyphs exactly in the same way. Now, the difference is in the powerline prompt line where there is a background color. See the branch sign being clipped in Hyper? Because Hyper encapsulates each unicode char in its own <span />, the wide unicode do not visibly overflow on the char to its right, since that char's background color is on top of whatever overflows. This is not an issue in iTerm where unicode is probably not encapsulated like this. Why do we encapsulate unicode in individual containers exactly?

I removed my previous changes and only left the low hanging fruit: fixing of CJK wide chars. Still trying to find a way around the powerline issue. Summary of issues i'm trying to address and the ones fixed:

• CJK chars display
• powerline wide unicode glyphs display
• multiple rendering of CJK in some cases (@BirkhoffLee report)
• tmux empty prompt width, see screenshot below:
  

[rauchg]

Why do we encapsulate unicode in individual containers exactly?

As explained above, without the <span> with a fixed width, glyphs that are not included in a monospace font (such as braille which uses the "Apple Braille" font would show up in a different width (and this width varies per-character) and break rendering)

What happens is wide unicode [on iTerm] simply overflows on the next char

That's not true for Emojis it seems? Or emojis are not "wide unicode" (why not?)

Another thing I'd like to add to your list: ensure chinese characters render correctly

[rauchg]

More food for thought: how should ﷽ be rendered? (widest known unicode character that I could Google for 😅)

[rauchg]

Personally, I like what Terminal.app does (similar to emoji handling). Take up one column, show overflow.

iTerm takes up one column, but only shows partial overflow:

[tibotiber]

OK, that's what I had gathered about the encapsulation, just wanted to make sure there were no other reasons.

That's not true for Emojis it seems? Or emojis are not "wide unicode" (why not?)

I don't understand. Emojis are wide unicode, and they do overflow on the next char in iTerm, don't they? This is what your screenshot shows with the heart overflowing from its span onto the next (behind the a). Maybe it's a vocabulary issue, I have the feeling we're saying the same thing.

Good job finding ﷽ 😂, I'll target full overflow, like in Terminal.app.

Another thing I'd like to add to your list: ensure chinese characters render correctly

Which part exactly? The wc-node behaviour is done. Are you referring to the 2 chars that displayed as 3 mentioned in #1376 (comment) / #1376 (comment)? Or anything else?

[rauchg]

Just wanted to be 100% sure! Seems like we're on the same page @tibotiber.
Unicode nirvana (for output) is within reach!

Next up once we merge this PR is to fix backspace (the behavior of the ANSI escape that is) for surrogate pairs and caret position

[rauchg]

Yep I was referring to the report by @BirkhoffLee

[rauchg]

One super tricky edge case that'll be difficult to solve (with my current knowledge of CSS):

We want it to overflow, but if it's at the edge, we would want the parent container to trigger scrolling somehow.

No terminal that i know of seems to handle this edge case currently

[rauchg]

Also not a priority for now considering our broken backspace, but thought I'd mention it

[tibotiber]

Took a while to get back to this, sorry. I pushed new changes to try and fix the display of unicode in powerline prompts:

• removed existing unicode patching that was creating new containers for unicode chars. It was making it impossible for unicode to overflow as desired when there is a background color (e.g. powerline).
• instead I override hterm's insertString to get back the node in which the string has been added and I decompose its textContent when it contains unicode to insert child nodes wrapping unicode in individual spans. Doing so we still manage to get monospaced unicode but it can now overflow as long as the background color doesn't change.

This is partially working and I can't figure out why only "partially". It basically works for general unicode display such as cat files or vtop, it also work with file editors when re-rendering rows etc, but the node's content is flattened as if nothing happened in powerline prompts, except for the last node in a row.

I think it has something to do with the way hterm handles data decoding in hterm.VT.prototype.parseUnknown_ but I can't manage to pinpoint exactly what. I could be wrong though and was hoping someone has an idea. I was originally not familiar with hterm's internals so I might miss something.

I left some debug logs that help to understand what the new insertString does.

[rauchg]

Reverting

[albinekb]

I'll have a look for a fix

[tibotiber]

Hey, sorry about this, didn't try to run dist. I'll have a look as well.

[tibotiber]

I'm not able to reproduce (macOS 10.12.4 as well). I'm only getting a warning for an unknown OSC code.

Might be coming from different prompts, what is yours like?

[albinekb]

I use https://github.com/brandonweiss/pure.fish and i think rauch is using https://github.com/sindresorhus/pure @tibotiber

[tibotiber]

Mmmmh, I'm on sindresorhus pure as well actually. I tried a few other prompts (antigen themes) and I'm getting no error. Not understanding this.

[albinekb]

🤔 I'll diag & give you something to reproduce with tomorrow, need to get some 💤

[mengzhuo]

Why don't we just let the webkit do this work instead of calculate the width of the char?

[albinekb]

@mengzhuo I'm sad to say it's a bit more complicated than that...

Sorry for not coming back yet, I'll get this sorted tomorrow 👼

[mengzhuo]

@albinekb

I use font "Cousine for Powerline", this screenshot is after I remove all unicode-node style.

You can compare with gnome-terminal at the background.

I think it's a "web-based" app, we should due with it more "web-way" by using more css instead of messing with DOM.

[tibotiber]

@mengzhuo, the issue with this approach is that you break the rendering for unicode that is not monospaced (see this comment). You can observe a bad example by running vtop after removing unicode styling. More generally, terminals expect monospace chars and handle exceptions such as CJK with "wide-chars" which are doubled-spaced.

[albinekb]

Sorry for the delay @tibotiber

the problem is that not all nodes has className, for example text-nodes (see https://developer.mozilla.org/en-US/docs/Web/API/Node/nodeType)

I tried two things:

• adding a check in hterm.Screen.prototype.wrapUnicode to only handle element nodes (type 3), no errors but emojis get scrambled.
• adding a check for node.className, same problem

even worse in fish:

here's what's in my console when entering one emoji: (fish)

this is maybe my shell being f*d up 😄

---

After testing some more, it seems to be an issue with hterm.Screen.prototype.insertString ?

because if i print emojis (cat a file with emojis in it for example), they work as expected

[tibotiber]

@albinekb thanks for posting more details. I looked into it and here are some thoughts:

1. we seem to have an issue with cursor position. Basically the cursor moves with double spacing on single spaced unicode. Since hterm.Terminal.prototype.eraseToRight uses the cursor position to calculate how many chars should be deleted, it creates an issue with unicode deletion. I'm thinking this might be the source of the unknown char that appears when we delete emojis. Note: this issue exists in master so it is not new to this PR and could be tackled in another PR. (edit: issue reported at A space before the cursor #1506)
2. nice find on "the problem is that not all nodes has className", I completely missed it. I fixed this locally and will open a new PR soon since this one is now frozen. (edit: here it is -> Fix unicode display, third attempt :\ #1873)

[tibotiber]

Quite honestly, hterm is horrible to patch because the code is extremely imperative and convoluted. Do you know of any other terminal emulator that would be more "react-ish"? That would be really cool! @rauchg?

[LeonBlade]

I stumbled upon this PR and wanted to ask here before creating an issue first, so sorry in advance for posting this here.

By any chance is this rendering issue related to the problem produced in this screenshot:

On the left is Terminal and on the right is Hyper. I'm speaking of course about the box side which seems to warp as well as contain some overlap between the characters.

[tibotiber]

@LeonBlade this will need to be checked once the follow up PR #1873 lands, but it should fix your issue.

before:

after:

[uetchy]

I've tested it with Japanese and Korean characters on macOS El Capitan and there are some visual errors appeared.
There is an iTerm2 screen next to Hyper to clarify differences.
It looks like the width of Japanese characters is too narrow and Korean characters are weirdly broken.

[ELLIOTTCABLE]

Hi! I don't necessarily have a horse in this race, besides the ones that the contributors above have (Pretty Unicode Interfaces [for english text] And Maybe Sometimes Emoji) — but I want to point out that this heavily affects users from other countries and cultures? This seems like a question that's ideal for a ‘reach out and ask users’ kind of moment. Maybe a request-for-comments notice in the README for six months, or something like that? (Also, a great way for beginners to contribute: “What does your language need from a terminal emulator?)

If nothing else, I suspect we need to get a ~Unicode expert~ to come in and answer some questions about character ranges to treat as ‘wide’ (our implementation gives them double width, automatically consuming an extra column) versus ones to ignore, treat as single-wide, and leave for userspace handling (“What? This emoji looks funny? I guess I'd better spit out an extra space after it.” aka The Yarn Approach).

[ELLIOTTCABLE]

An aside — copied from a friend, as examples of why the emoji in particular are A Hard Problem, and “display all private-use-area characters as two columns” isn't enough:

• 👨‍❤️‍💋‍👨 1f468-200d-2764-fe0f-200d-1f48b-200d-1f468 KISS: MAN, MAN
  8 codepoints, 11 UCS-2 units, one Wide character.

• 🤼🏻 1f93c-1f3fb WRESTLERS + EMOJI MODIFIER FITZPATRICK TYPE-1-2
  This used to be 1 glyph in Emoji 3.0, but 4.0 deprecated group skin tones.

[sachin21]

@tibotiber Great!! You saved my life. Thank you. I'll looking forward to be released 👍

[lucasqueiroz]

Seems like 1.4.x fixed this issue.
Vtop with 1.4.2 (PragmataPro Font):