Handle more terminals (Wayland, multiplexers)

[Porkepix]

I discovered Ueberzug as a dependency for the Termusic player, but as I'm using Wayland sessions, I could unfortunately discover it doesn't work in many cases I'll list below. This sometimes trigger an error message, sometimes not depending on the cases.

From what I read to some places, it looks like it's currently (or actually the python project was, but this one is supposed to work pretty much the same way from what I understand) pretty tied to Xorg, and therefore bound to issues with Wayland.
On a Wayland session, I could indeed see that it doesn't work within Alacritty, nor does it in Gnome's new Console. I didn't checked the old gnome-terminal, but I guess the same issue is to be expected.
For some reason I can't explain (and I would be really curious about it), Wezterm manage to have ueberzug to work even when ran as a Wayland program (I could check it wasn't from XWayland).

Another case to consider is the one of terminal multiplexers: even in a working terminal such as Wezterm, tmux or zellij for example could be verified as having the album art not working, despite it seemed to show no error.

[jstkdng]

Under wayland the only way to support displaying images is through sixel, and that has to be supported by your terminal.
You can check here which terminals support sixel. https://www.arewesixelyet.com/

Terminal multiplexers also have to support sixel. Tmux doesn't support it, and it seems zellij does but it it's broken.

This project does have sixel support, so perhaps that's why wezterm worked.

[Porkepix]

But then if Zellij supports it, why wouldn't it appear there? Is it on Zellij's side in your opinion?

[jstkdng]

Zellij is in that list, and it works with with tools such as img2sixel. It just returns bad values when calling this code

struct winsize sz;
ioctl(this->pty_fd, TIOCGWINSZ, &sz);

Without this information ueberzug can't make the required calculations.

[Porkepix]

Zellij is in that list, and it works with with tools such as img2sixel. It just returns bad values when calling this code

struct winsize sz;
ioctl(this->pty_fd, TIOCGWINSZ, &sz);

Without this information ueberzug can't make the required calculations.

So, it should provide an information and doesn't provide that information? How would you phrase that if this is something that should be reported to Zellij?

[jstkdng]

Zellij should report non-zero values when calling the ioctl function. In this case, both ws_xpixel and ws_ypixel return 0.

[imsnif]

Hey, just a heads-up - while Zellij indeed doesn't set these values correctly (we're not the one reporting them btw, just setting them, the pty does its own thing here) - you can use XTSMGRAPHICS queries (https://invisible-island.net/xterm/ctlseqs/ctlseqs.html) to get the same info, which Zellij does support.

Thanks for the troubleshooting! I'll fix the ioctl call on the Zellij side when I have the time.

[eylles]

Ah speaking of tmux, the maintainer is open to support sixel but he doesn't want to put the work on the sixel resizer and handling of sixel sequences on non sixel capables terminal, iirc he once put out a list of requirements for anyone to submit either a library or a company binary and a guy did promised something but he went on to make his fork of tmux instead of cobtributing to the upstream, if you are already doing the sixel scaling calculations on ueb++ (i think you are but have not taken a close look at the code) perhaps u could open an issue on the tmux repo and offer your sixel handling code and ask what the needs are now and what features would u have to add or change.

[jstkdng]

do you have the link for the issue/PR where the requirements are?

[eylles]

The OG issue with the discussion:
tmux/tmux#1613

The fork i talked about:
https://github.com/csdvrx/sixel-tmux

And the standalone tool developed by that guy to create ansi colored unicode of images when sixel is not available for whatever reason:
https://github.com/csdvrx/derasterize

ncim's frustration over csdvrx didn't push forward with patches or a PR:
tmux/tmux#2759 (comment)

From his tone he would be open to go with it as long as someone submits a proposal PR and is willing to do as he asks to integrate onto tmux, tho no certainity on it landing on a release soon altho there's push from the wayland guys to have sixel or kitty supported on tmux.

[jstkdng]

well, I doubt my sixel code would help them as it depends on libsixel itself. AFAIK no terminals that support sixel depend on libsixel.

SIXEL is too slow

it is slow compared to plain X11 but compared to others (kitty/iterm) it is great so supporting it is worth it. If I got the time I'll check if I can help out on the tmux side as I use it a lot.

[eylles]

Yeh, those are some of the challenges, not depend on libsixel itself and speed, tho xterm is the slpwest renderer, nowadays there are much faster ones like vte (tho in the git master)

[Porkepix]

If SIXEL is too slow, is there other faster alternatives on wayland?

[jstkdng]

sixel is only slow on the encoding side (pixels to sixels), the interpreting done by the terminal is almost instantaenous.

Kitty and iterm are the next best options but I have not tested them. It seems kitty and iterm only need to convert pixel data to base64 so it could be faster than libsixel. Now the problem is only a few terminals have implemented the kitty and/or iterm2 protocol. Sixel is more widely used so I'd stay with it.

[Porkepix]

sixel is only slow on the encoding side (pixels to sixels), the interpreting done by the terminal is almost instantaenous.

Kitty and iterm are the next best options but I have not tested them. It seems kitty and iterm only need to convert pixel data to base64 so it could be faster than libsixel. Now the problem is only a few terminals have implemented the kitty and/or iterm2 protocol. Sixel is more widely used so I'd stay with it.

Kitty also comes with its share of issues, I've seen that the combination of kitty + zellij + termusic is even failing to draw anything and end completely stuck. I have no idea though which of these software is to blame for this…

I wish there was something more standardized; is it the case for X11?
And yeah, I wish every terminal would manage it the same way rather than implementing their own way to do things.

[jstkdng]

For X11, child windows are the fastest option, though some terminals that add custom decorations inside the terminal do mess with the window positioning. Konsole's toolbar, VTE-based terminal menu bars, wezterm in particular doesn't like x11 child windows, that could be a bug on the wezterm side since alacritty works fine (both are written in rust).

It'd be nice if wayland had a way to attach child windows to existing ones. From what I've read that is not possible. Perhaps if the terminal itself used U++ as a library and spawned wayland child windows it could work but that'd be hard to do I imagine.

[jstkdng]

ok, I have implemented the kitty protocol and it is fast than sixel only on kitty itself, on konsole is ~2x slower and I wasn't able to test on wezterm/wayst.

some times to draw a large png file

kitty + kitty => ~50ms
kitty + konsole => ~150ms
sixel + konsole => ~70ms
sixel + alacritty => ~60ms

So if you want to use the kitty protocol, use the kitty terminal.

[jstkdng]

With the zellij issue fixed, I'll be closing this.

[Porkepix]

Can you point to where was it fixed for Zellij? As for the terminals I guess it falls upon them to support sixels or other techs.

[jstkdng]

zellij-org/zellij#2212

zellij's sixel implementation is still bad though, can't handle gifs and cleanup is broken.

[xlucn]

... some terminals that add custom decorations inside the terminal do mess with the window positioning. Konsole's toolbar, VTE-based terminal menu bars ...

Does that mean ueberzugpp indeed doesn't work with VTE-based terminal emulators? I just found out that it doesn't show images on tym (a vte-based one) and bare-bone vte-2.91.

[jstkdng]

I meant that it doesn't work correctly on the VTE terminals I've tested, if there's a terminal (doesn't matter if it is VTE based) in which ueberzugpp doesn't work at all but ueberzug does then it is a bug and it should be fixed.

[xlucn]

Alright then I will report it later if I can't figure it out myself.