TextEmbed Widget Discussion

[danschwarz]

Moving the discussion of TextEmbed here (from here, because it really shouldn't be in a PR)

[danschwarz]

@AnonymouX47 OK, so about TextEmbed. Now that you've basically done the impossible, I'll tell you more about my use case and we'll see how closely the widget lines up with those requirements 😄

In Toot, Mastodon status posts are displayed using a custom widget - an urwid.Pile containing, among other things, an urwid.Text widget rendering the status content with multiple lines of styled text.

Here's a good example. For now we're concerned with the urwid.Text widget rendering the area that's bracketed in red:

Mostly plain text, with styled hashtags. Also, some kind of weird glyph after #biz , but that might be some Windows Terminal weirdness so let's ignore that. There's a URL in the text as well.

This example has no embedded custom server emoji :shortcodes: , which is typical - they are more frequently used in display names, but they sometimes do appear in status content text.

Ideally, what I'd like to do is this. I'd like to be able to drop in the TextEmbed widget in place of an urwid.Text widget and have it work like an urwid.Text widget (same options for color, format, alignment). But with TextEmbed, I can also specify placeholders in the text, and drop in other widgets to render in the placeholder's location. (What you have done with the { } syntax.)

I can then use the UrwidImage widget to render custom server emoji in a placeholder location, and I can use URLText to render a hyperlink with OSC 8 codes in another placeholder location.

This means that I will be doing the following, for terminals that have pixel rendering capabilty:

1. Get content text from status
2. Escape all existing { } in text, so they don't get misinterpreted as placeholders
3. Parse out :shortcodes: in the content text and replace with placeholders
4. Parse out URLs in the content text and replace with placeholders.
5. For each shortcode, look up the appropriate image URL, load image from internet or cache, and create an UrwidImage widget matched to the corresponding placeholder.
6. For each URL, create a URLText widget matched to the corresponding placeholder
7. Create the TextEmbed widget with the content text including placeholders, plus all the UrwidImage and URLText widgets.

For terminals that can only render to cells, I will do a subset of the above - just steps 1, 2, 4, 6, 7. :shortcodes: will be left as is.

Given that the number of :shortcode: images and URLs I need to embed will vary from one status post to another, it'd be easiest for me to provide a list of widgets to TextEmbed, rather than sending named parameters.

[AnonymouX47]

I've gone through and I think I understand the use case better now.

I'll reply in a moment...

[AnonymouX47]

have it work like an urwid.Text widget (same options for color, format, alignment)

Yes, I have that in mind too. The current state is just the initial implementation to get the main thing working.

This means that I will be doing the following, for terminals that have pixel rendering capabilty:

...

For terminals that can only render to cells, I will do a subset of the above - just steps 1, 2, 4, 6, 7. :shortcodes: will be left as is.

True. re.split() with a group of OR-ed patterns comes to mind 🤔.

Just my [not so filtered] thoughts... As for the emojis, a single wrapper widget can actually be used for both graphics-capable and non-graphics-capable terminals. The widget wraps an UrwidImage widget on a graphics-capable terminal and a Text widget containing ":shortcode:" if otherwise.

it'd be easiest for me to provide a list of widgets to TextEmbed, rather than sending named parameters.

Yeah, TextEmbed supports positional replacement fields also.

[danschwarz]

As for the name, TextEmbed is fine.

[AnonymouX47]

I've opened the PR but note, it's only in the initial stage.

[danschwarz]

By the way, if you're planning an official release of term-image, I think it's reasonable to do a release without including TextEmbed, then include TextEmbed in a future release. The existing UrwidImage widget (and the others) are worthy of a release on their own.

[AnonymouX47]

I think it's reasonable to do a release without including TextEmbed, then include TextEmbed in a future release.

Never planned to include TextEmbed in term-image. 👀 I guess you assumed so because of the attention I'm giving to it but NO, i don't intend to.

There's one thing that has been bothering me though... toot is GPL-3 licensed. By merging TextEmbed into toot, would that mean even myself couldn't reuse the code in, let's say, an MIT-licensed codebase?

[GuardKenzie]

There's one thing that has been bothering me though... toot is GPL-3 licensed. By merging TextEmbed into toot, would that mean even myself couldn't reuse the code in, let's say, an MIT-licensed codebase?

Sorry I'm eavesdropping here but yes, all code contributions to GPL3-licensed works are themselves licensed under GPL3. This means that you can not reuse them in an MIT-licensed codebase. (for reference, see this thread on stack exchange)

[AnonymouX47]

Sorry I'm eavesdropping

Oh! Not at all.

---

Thanks for the clarification.

But if I were to publish the code with a more permissible but GPL-3-compatible license first and then contribute it, the license with which I published it would be retained right? (See https://www.tawesoft.co.uk/kb/article/mit-license-faq)

EDIT: @danschwarz has answered my question in another comment below. Thanks 🙏🏾

[danschwarz]

(rewriting my entire comment because I now understand that you intend for TextEmbed to be a separate entity from TermImage)

• You're the author of TextEmbed and retain copyright over it
• Any version of TextEmbed not included in the Toot source code base can be MIT licensed, dual licensed, whatever you like.
• Any version of TextEmbed included in the Toot source code base will be GPL3 licensed and any changes to the version of TextEmbed within Toot will be GPL3 licensed.

If you're willing to host TextEmbed and maintain it as a MIT licensed library, we could incorporate it into Toot that way, and direct anyone who wants to change it, to submit a PR to you. That way you would be able to get all changes under MIT license from Day 1.

[AnonymouX47]

(rewriting my entire comment because I now understand that you intend for TextEmbed to be a separate entity from TermImage)

Ouch! Sorry about that.

If you're willing to host TextEmbed and maintain it as a MIT licensed library, we could incorporate it into Toot that way, and direct anyone who wants to change it, to submit a PR to you. That way you would be able to get all changes under MIT license from Day 1.

I see. I guess I'll go with this then. To be sincere, the code seems too precious 🥲 to get locked up (since toot is not published as a "library") under GPL-3. I can imagine others or even myself who might want to re-use it in the future.

Even if it were GPL-3-licensed but packaged and distributed as a library, I wouldn't mind but this is not the case with toot. If https://github.com/tonycpsu/panwid wasn't "dead", I would've happily contributed it there but I guess I might just start a collection of non-image-related custom urwid widgets and publish it as a library.

Sorry, I have to put you through all these but I feel it's for the better.

Thanks.

[danschwarz]

No worries, it's your effort so you should decide how it gets licensed.

If https://github.com/tonycpsu/panwid wasn't "dead", I would've happily contributed it there but I guess I might just start a collection of non-image-related custom urwid widgets and publish it as a library.

Sometimes I feel like Dr. Frankenstein, raiding these "dead" projects for widgets: ☠️

[AnonymouX47]

Hello! 😃

Sorry for keeping this waiting for sooo long. Happily, there's no need for excuses anymore... It's finally here!

Source: https://github.com/AnonymouX47/urwidgets/blob/main/src/urwidgets/text_embed.py
Doc: https://urwidgets.readthedocs.io/en/latest/api.html#urwidgets.TextEmbed

I ended up implementing something even beyond my initial imagination. Here's a little demo:

simplescreenrecorder-2023-04-27_15.56.52.mp4

Lest I forget, this Hyperlink widget is what you can see in the demo above.

Also, the test script I used in the demo above: https://pastebin.com/HFknitc0 (You'll need to install term-image and might need to change the image paths)

[AnonymouX47]

With this, emojis, URLs, hashtags (hyperlinked), etc... should work just fine. All that's needed now is to parse the raw text, cut out the emojis, URLs, etc... and replace them with the appropriate widgets.

I intend to release the package as soon as you've tested the widgets and we're sure they working as intended and have all necessary features. Please, let me know if there's anything to fix, improve or add. I'll do whatever I can to get it done ASAP.

Also, in case I forgot any widget or the likes that we discussed earlier, please let me know.

Thanks.

[AnonymouX47]

All that's needed now is to parse the raw text, cut out the emojis, URLs, etc... and replace them with the appropriate widgets.

Here goes something 😃: https://urwidgets.readthedocs.io/en/latest/api.html#urwidgets.parse_text

[danschwarz]

Had to simplify your example a bit to get it working in Toot for Mastodon statuses (there's no Markdown syntax available, so I'm just recognizing embedded URLs in plaintext via regex)

Here's an example of a URL surrounded by plaintext; the TextEmbed and Hyperlink widgets work correctly, the line wrapping works, and the OSC 8 functionality gives the correct behavior in Windows Terminal! I tried half a dozen different ways to make this work by hacking the Text widget; this is much, much better. 😁

There's more work to be done before this can be submittted as a PR. I need to make it play nicely with some other features. And then I need to rewrite it entirely once PR 348 is accepted. (Then I'll be rendering real HTML anchor tags in the terminal)

Anyway, this is excellent work, thanks again!

[AnonymouX47]

Oh, glad to hear that.

By the way, have you seen my other comment above?

All that's needed now is to parse the raw text, cut out the emojis, URLs, etc... and replace them with the appropriate widgets.

Here goes something 😃: https://urwidgets.readthedocs.io/en/latest/api.html#urwidgets.parse_text

If not, I believe it'll simplify a lot of things.

[AnonymouX47]

There's more work to be done before this can be submittted as a PR. I need to make it play nicely with some other features. And then I need to rewrite it entirely once PR 348 is accepted. (Then I'll be rendering real HTML anchor tags in the terminal)

I see... that's some real work there. Nice work with that PR by the way.

Anyway, this is excellent work, thanks again!

It's a pleasure 😃.

[danschwarz]

Oh, glad to hear that.

By the way, have you seen my other comment above?

Yes- I used a simplified version of your sample code with parse_text.

[AnonymouX47]

Oh! Good.

[danschwarz]

It's so close to being 100% identical in layout / formatting to the master branch now, with one issue that I'm having a hard time figuring out.

Is it possible that TextEmbed is calculating a unicode character's width in columns, and in this one case is doing so incorrectly?

Here's how the text is rendered using standard urwid.Text:

Here's how it looks using TextEmbed:

I should note that I'm using parse_text with a regex to detect the URL and build the markup list.

With TextEmbed, everything looks fine with the camera emoji and the single space after it.
But it seems that TextEmbed is skipping a space after the World Map emoji - where the red arrow is pointing; the t character is left over from a previous status message; Urwid hasn't redrawn that cell.

The Camera emoji is U+1F4F7, introduced in Unicode 6.0.
The World Map emoji is U+1F5FA, introduced in Unicode 7.0

The Unicode strings are:

'📷 https://www.windy.com/-Webcams/webcams/1577050386'
'🗺️ https://www.google.com/maps/search/40.705222,-73.995946 '

FWIW, Toot uses the wcwidth library to calculate unicode character widths.

[AnonymouX47]

Sorry, for whatever reason unknown to me, I'm just seeing this now 🥲.

This is really surprising by the way. I don't recall either TextEmbed or Hyperlink producing any cursor control sequences 🤔, even if it's producing an extra space character or so, it should at least overwrite that cell.

The Unicode strings are:

':camera: https://www.windy.com/-Webcams/webcams/1577050386'
':world_map: https://www.google.com/maps/search/40.705222,-73.995946 '

I'll look into the issue using this for now but is this the entire content of the concerned widgets?

If not, I'll prefer the whole thing, if possible, i.e the raw markup of the TextEmbed (with the tuples, newline escapes, etc)... you can probably paste it in a code block or something else that'll preserve the string as-is.

The markup for the Text in addition will also help.

Thanks.

[danschwarz]

The above two strings are rendered as two separate Text widgets, one per string, each on their own row.
The same strings are parsed by text_parse and separated into text markup and a Hyperlink widget. I don't think you are moving the cursor in any way, but is it possible Urwid might be moving the cursor in order to accomodate what it thinks is a wider width World Map emoji, one column right, before rendering the Hyperlink widget? Just a theory.

I should build a small test app to attempt to reproduce the problem outside the entire Toot application. Will try to do that after I take care of my responsibilities today.

[AnonymouX47]

The above two strings are rendered as two separate Text widgets, one per string, each on their own row.

Oh! I see... makes the whole thing pretty much simpler.

but is it possible Urwid might be moving the cursor in order to accomodate what it thinks is a wider width World Map emoji

🤔... that's possible.

[AnonymouX47]

I've tried it on my end and couldn't reproduce this behaviour. Text and TextEmbed produced identical output (i.e everything was the same excluding the the hyperlink feature), from the character width computation, to the canvas content, down to the output drawn on-screen (on every terminal emulator I tried it on).

Though, there was one difference between your output and mine... the glyph for the U+1F5FA symbol occupied 1 cell on every terminal emulator I tested it on (probably due to my system/installed fonts) while urwid computes the width to be 2. Despite this, the output of Text and TextEmbed were still consistent with each other.

It seems whatever is responsible for the issue is specific to toot's environment.

FWIW, Toot uses the wcwidth library to calculate unicode character widths.

What exactly did you mean by this and in what way does it affect the output of Text and/or TextEmbed?
It'll also be good if you can point out to me where it's used in toot's code base.

Or better still, this:

I should build a small test app to attempt to reproduce the problem outside the entire Toot application. Will try to do that after I take care of my responsibilities today.

Thanks.

[danschwarz]

Re: wcwidth. Toot doesn't use it everywhere, just in our custom widgets, for example when setting the width of our custom button widgets that include emoji in their titles, to properly measure the width of the Unicode text, so the buttons display their titles at full width without line-wrapping.

There was a push to integrate wcwidth into Urwid itself; see this issue that didn't get resolved. That link shows a lot of references to Urwid issues where the width of certain Unicode characters - especially newer emoji, or combining characters - were being calculated differently by Urwid than the width the terminal app itself was using to draw them.

This is a nasty unsolved problem across terminal apps - basically, the low level code that a lot of software uses to calculate Unicode character widths is out of date, hasn't been updated since Unicode 5.0 days, and doesn't support combining characters well. The TUI app and the terminal software have to agree on the character widths or things start getting out of alignment.

wcwidth goes part of the way towards solving these problems; another requirement is to determine at runtime the unicode version supported by the terminal in use, so wcwidth can return values that agree with the terminal's unicode version.

Windows Terminal is pretty good with recent Unicode standards, I see some width problems with a few emoji, but not a lot. Other terminal programs like xterm.js have more problems because they've not been keeping up with Unicode standards.

Anyway, to bring it back around to the bug at hand - when I saw this I thought we might have some kind of emoji width calculation bug going on, which is why I brought up wcwidth. Based on your testing, this may not be the case.

If you can't reproduce on your side, I'll try to do so here. This is really the only glitch i've seen so far in comparing output of the old Text vs new TextEmbed widgets.

[AnonymouX47]

I see.

The TUI app and the terminal software have to agree on the character widths or things start getting out of alignment.

Very true!

If you can't reproduce on your side, I'll try to do so here. This is really the only glitch i've seen so far in comparing output of the old Text vs new TextEmbed widgets.

Ok, good. If by any chance I'm able to reproduce the issue or somehow figure out what could be wrong, I'll let you know.

By the way, have you tried this same case on any other terminal emulator?
It just might be an issue with Windows Terminal (maybe related to it's OSC 8 implementation).

[danschwarz]

OK, so I can tell you definitively that it's the fault of the terminal emulator - or, rather, a disagreement between python/Urwid and the terminal emulator about the width of certain emoji. Note that in the screenshots below, the code is slightly changed to render URLs in italics. Everything else is exactly the same and the spacing between characters should be unaffected by that change.

Here's xterm.js, which doesn't leave enough space after either emoji

Here's kitty, getting it exactly right

Xterm can't even display emojis in my configuration, but if it did, it'd have inconsistent spacing for the emoji

Alacritty gets it wrong, too

[AnonymouX47]

I guessed as much.

[danschwarz]

I'm not sure this is even fixable by anyone other than the terminal emulator authors. So, I think the best thing to do is to live with it. It's not a bug in TextEmbed.

If it is possible to fix on the TUI side, it would probably require integration of wcwidth and ucs-detect in the Urwid canvas rendering code. Ucs-detect at startup to determine the highest version of Unicode supported by the terminal, and wcwidth to provide the true width of each Unicode character as it will be rendered in the terminal.

[AnonymouX47]

If it is possible to fix on the TUI side, it would probably require integration of wcwidth and ucs-detect in the Urwid canvas rendering code.

The thing is urwid actually calculates the width of these two emojis correctly 🤔 i.e it takes them as occupying 2 cells which is correct.

It's the (most) terminal emulators that tend to use the wrong widths.

Anyways, if anything like this were to be modified in urwid, it can only be done upstream cos it'll require modifying various private interfaces, particularly in urwid.text_layout.