Kitty Image Protocol Support #986
Comments
These were parsed but swallowed. This commit expands the transitions to be able to track the APC start, data and end and then adds an `apc_dispatch` method to allow capturing APC sequences. APC sequences are used in the kitty image protocol. refs: #986
This teaches termwiz to recognize and encode the APC sequences used by the kitty image protocol. This doesn't include support for animations, just the transmit, placement and delete requests. refs: #986
This isn't complete; many of the placement options are not supported, and the status reporting is missing in a number of cases, including querying/probing, and shared memory objects are not supported yet. However, this commit is sufficient to allow the kitty-png.py script (that was copied from https://sw.kovidgoyal.net/kitty/graphics-protocol/#a-minimal-example) to render a PNG in the terminal. This implementation routes the basic image display via the same code that we use for iterm2 and sixel protocols, but it isn't sufficient to support the rest of the placement options allowed by the spec. Notably, we'll need to add the concept of image placements to the data model maintained by the terminal state and find a way to efficiently manage placements both by id and by a viewport range. The renderer will need to manage separate quads for placements and order them by z-index, and adjust the render phases so that images can appear in the correct plane. refs: #986
|
I am happy to see this, please don't hesitate to ping me if you have any questions. |
Great to hear! Do you have a set of tests or similar that could be made to run against wezterm to sanity check conformance? |
|
I do have unit tests in kitty itself, for individual bits of functionality from the protocol, but I doubt they can be easily lifted for another terminal emulator. If you wish to have a look, please see kitty_tests/graphics.py I would actually be happy to collaborate to make those (and other) tests defined in a terminal independent fashion so anyone implementing the protocol could use them. Perhaps a simple txt or json based format that specifies the input as escape codes and the output as a set of image placement data or similar. |
first off, i enthusiastically support this move. the kitty protocol is (at least for my purposes) a tremendous improvement upon both sixel and iterm. it simplifies things (no more requirement that one draw in terms of 6 rows, sane deletion behavior), improves performance in several areas (fast moves, fast changes to drawn images), and makes certain things possible that otherwise are not (text drawn atop bitmaps without killing entire cells). relative to iterm2, it's much more flexible and powerful. at the same time, it currently shows worse local performance in Notcurses due to (a) greater bandwidth demands than sixel and (b) time spent in zlib. i expect to be able to hide the latter in most applications via threading. it would be possible to reduce the local bandwidth via using the filesystem as a side channel to load images, but i have not yet embraced this, and might never do so (but might, who knows). right now Notcurses selects the Kitty protocol strictly based on heuristics, as opposed to doing the recommended query. I intend to add support for the latter (and really ought have by now), but hadn't bothered since no one else had implemented it (and also because said query frustratingly doesn't let you determine the version of the kitty protocol supported--as you note, it's a large protocol). you can ensure kitty graphics are being used by running
where it says "rgba pixel animation support", you want that. wezterm currently says "sixel graphics" or "iterm graphics", i forget which one. if it says "rgba ....", you're driving kitty graphics. i'll try to add this query soon. Notcurses uses a pretty wide subset of the protocol, and indeed motivated/proposed some of it. Among the elements it exercises are:
i do not currently exercise: sideloading images via the filesystem, managed animations, scaling, or z-indices other than 0, though i expect to start using the last Really Soon Now.
if you have any questions, don't hesitate to ask me; i reckon i know more about the kitty graphics protocol than anyone living save @kovidgoyal himself. happy hacking! |
oh and let me add that i make use of kitty's honoring of transparency values in RGBA (at at least a bimodal level), just as I do in iterm, and using unspecified pixels in sixel when P2=1 in sixel. |
|
@dankamongmen regarding using heuristics for detection, because you dont have a version. You shouldnt need a version. You can create a dummy image and try to perform every operation you want on it and see if it succeeds, i.e. the terminal does not respond with an error. Thus you can know exactly what the terminal you are running in supports. And of course as am sure you already know, if you really want a version us XTVERSION or XTGETTCAP |
this is exactly what i'm doing, see https://github.com/dankamongmen/notcurses/blob/master/src/lib/termdesc.c#L501-L527
yep, i could do that. what i've got now works for me pretty well, though. and let it be known in all lands the sun touches: i wholeheartedly affirm the use of the Kitty graphics protocol over others. indeed, i sat down a few weeks ago to describe my ideal terminal graphics protocol, and ended up with something very similar to Kitty's. i will be adding support for coarse kitty graphics support very shortly, probably tonight, in dankamongmen/notcurses#1998. |
This is a stepping stone towards dynamically allocating vertices. refs: #986
This commit removes the `Quads` struct which maintained pre-defined quad indices for each of the cells, the background image and scrollbar thumb. In its place, we now "dynamically" hand out quads to meet the needs of what is being rendered. There are some efficiency gains here with things like the selection (which can now be a single stretched quad, rather than `n` quads in width). This isn't a fully dynamic allocation scheme, as we still allocate the current worst case number of quads when resizing. A following commit will adjust that so that we allocate a ballpark and then employ a mechanism similar to OutOfTextureSpace to grow and retry a render pass when we need more quads. Futhermore, this dynamic approach may allow reducing the amount of stuff we have in the Vertex and "simply" render some quads before others so that we don't have to have so many draw() passes to build up the complete scene. refs: #986
This removes the pre-allocated (at resize) number of quads and replaces it with a dynamic mechanism that tracks how many quads are needed for a frame and then will re-allocate and re-render when there weren't enough. We start with 1024 quads and try to allocate in multiples of 1024 quads. refs: #986
Taking further advantage of dynamic quad allocation, we can now remove the multiple render passes in favor of allocating the quads and painting them from back to front. In turn, this means that we can reduce the amount of data that we store in the vertex, which simplifies the shaders a bit, at the expense of making the render code in rust a bit more complex. However, we can take advantage of stretching runs of cells with background colors in to a single quad. refs: #986
Now that we have a single pass, we don't need to "include" snippets, and that makes the shaders a lot easier to follow. refs: #986
Allows selecting a source "sprite" from the image data, and offsetting its position within the cell. refs: #986
Just use a single quad for a given split refs: #986
I noticed when running the notcurses demo that we're spending a decent amount of time decoding png data whenever we need to re-do the texture atlas. Let's avoid that by allowing for ImageData at the termwiz layer to represent both the image file format and decoded rgba8 data. This commit is a bit muddy and also includes some stuff to try to delete placements from the model. It's not perfect by any means--more expensive than I want, and there's something funky that causes a large number of images to build up during some phases of the demo. refs: #986
This adds a simple garbage collection scheme; when adding an image, check to see if we're over budget on the total amount of RAM used by the image data. If we are, remove unreferenced images (images that are not placed) until we're below the budget. refs: #986
Moves the localized hashing logic from term -> termwiz where it can be re-used. refs: #986
|
@wez 👆🏾 |
|
I saw this, and I appreciate the detail in your comments, but haven't had time to look at it. If you're eager to see this improved more quickly then I would be happy to accept a PR! Otherwise, I'd like to remind you that this is free software that I hack on in my spare time, and I have a lot of demands on my spare time. |
|
Oh! I totally understand that, I was only expecting some form of acknowledgement (of the comment). It's not urgent for me as I personally don't use Wezterm, I was only testing out my project in different terminal emulators and decided to report bugs/issues I found (just like kovidgoyal/kitty#5081). Thanks for your response. |
|
Do I remember correctly that wezterm used to have a "cat image to screen" utility? Where's a good one? |
|
The default tool is the
Also, you can use term-image which provides an interactive user interface for browsing/viewing images... still under developmenr though.
There are a number of other tools out there that also perform similar tasks e.g timg. |
|
The latest release of Kitty added an interesting feature that allow displaying images without explicit support from a program, using unicode chars, diacritics and fg color to select and position an image. |
|
A wezterm fork (by the author of the feature) that implements support for the feature: https://github.com/sergei-grechanik/wezterm/tree/unicode-placeholders Might make sense to ask the author to open a PR. |
|
Assuming that adding support for Kitty's unicode placeholder model means Wezterm + tmux can support the Kitty image protocol, can we get a todo-checkbox for it up top here? |
|
I'm trying to find a good way to display images in neovim, but most plugins use the kitty protocol, which doesn't seem to work as intended on wezterm.
wezterm.mp4Edit: Another plugin, another issue
image.nvim.mp4 |
- renamed image protocol check functions - switched back to base64.StdEncoding for Kitty format. Kitty handles Std or Raw, but wezterm only accepts Std. - wezterm is also kitty-capable now (wez/wezterm#986) - updated tests - write kitty image preamble outside of chunk writer
|
There is a bug where sending using an image in two places only renders it once. When I first send the RGBA data with 
|
This PR adds support for kitty protocol detection when you're using tmux. This means now when you're running tmux inside kitty you no longer need to explicitly configure the image protocol to be kitty-local/remote and it will be instead figured out automatically. This is done by making two kitty graphics protocol queries (and for local and one for remote support) and another widely supported query just in case the emulator doesn't support the kitty protocol and ignores our queries (this is what's suggested [here](https://sw.kovidgoyal.net/kitty/graphics-protocol/#querying-support-and-available-transmission-mediums)). The [third query](https://vt100.net/docs/vt510-rm/DA1.html) being done gets us the terminal capabilities; we figure out whether we support sixel by parsing its output. This means now we always make these queries once regardless of the protocol being used unless you explicitly configure a protocol other than `auto` (the default). The caveat here is that terminal emulators may not support [unicode placeholders](https://sw.kovidgoyal.net/kitty/graphics-protocol/#unicode-placeholders) which is a requirement for the kitty protocol to work while inside tmux. At least wezterm [does not currently support them](wez/wezterm#986) so images printed while inside tmux in it look like crap. The workaround for now is to guess that we're inside wezterm + tmux by checking an env var that wezterm sets. This won't work if you start tmux in another terminal (since it inherits _those_ env vars) but it's the best we can do. The outcome of this is if there's another terminal that does support the kitty protocol but does not have unicode placeholders implemented, images will look bad and users will be forced to [specify the image protocol](https://mfontanini.github.io/presenterm/guides/configuration.html#preferred-image-protocol) to be used by hand.
This issue is tracking the status of supporting Kitty's image protocol.
Spec at: https://sw.kovidgoyal.net/kitty/graphics-protocol/
Using it:
Enable the protocol by setting
enable_kitty_graphics=truein your config.Known conformance issues
The spec models image placements independently from the terminal cells, but wezterm maps them to terminal cells at placement time. As a result, over-writing cells with text can poke holes in wezterm which may not appear in kitty.The text was updated successfully, but these errors were encountered: