Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: sixel graphics support #448

Closed
migueldeicaza opened this issue May 7, 2019 · 116 comments
Closed

Feature Request: sixel graphics support #448

migueldeicaza opened this issue May 7, 2019 · 116 comments
Labels
Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Needs-Tag-Fix Doesn't match tag requirements Product-Conpty For console issues specifically related to conpty Product-Terminal The new Windows Terminal.
Milestone

Comments

@migueldeicaza
Copy link

Would like to see Sixel support in the Terminal, this is the standard used to show graphics in the console.

Sixel is part of the original DEC specification for doing graphics in terminals and has been re-popularized in recent years for doing graphics on the command line, in particular by Pythonistas doing data science.

The libsixel library provides an encoder but is also a great introduction to the subject (better than the Wikipedia page):

https://github.com/saitoha/libsixel

@PhMajerus
Copy link

PhMajerus commented May 7, 2019

While implementing Sixel, it is important to test with images that contain transparency.
Transparency can be achieved by drawing pixels of different colors but not drawing some pixels in any of the Sixel colors, leaving the background color as it.
I believe this is the only way to properly draw non-rectangular Sixels, and would be especially nice with the background acrylic transparency in the new Windows Terminal.

Testing using WSL with Ubuntu for example, in mlterm such images are properly rendered as having a transparency mask and the background color is kept, while in xterm -ti vt340, untouched pixels are drawn black, even though the background is white, which seems to imply they render sixels on a memory bitmap initialized as black without transparency mask or alpha before blitting them into the terminal window.

@fearthecowboy
Copy link
Member

OOh. Sixel is very cool stuff.

I've decided that I need that. NEED.

@zadjii-msft
Copy link
Member

I'll happily review a PR :)

@zadjii-msft zadjii-msft added the Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. label May 7, 2019
@zadjii-msft zadjii-msft added this to the Backlog milestone May 7, 2019
@zadjii-msft zadjii-msft added Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues labels May 7, 2019
@therealkenc
Copy link

therealkenc commented May 8, 2019

Caught the Build 2019 interview today that mentioned this request. I still maintain that Xorg on sixel is just wrong. So very very wrong.

The ffmpeg-sixel "Steve Ballmer Sells CS50" demo never gets tired tho. Gotta say, it is a little disappointing the video lacks sound (sound really makes the video). Consoles already have sound, naturally. They totally beep. Precedent set. What we really need is a new CSI sequence for the opus clips interleaved with the frames, amirite?

@DHowett-MSFT
Copy link
Contributor

DHowett-MSFT commented May 9, 2019 via email

@DHowett-MSFT DHowett-MSFT changed the title Sixel graphics support Feature Request: sixel graphics support May 9, 2019
@DHowett-MSFT
Copy link
Contributor

Related: #120

@chadbr
Copy link

chadbr commented May 9, 2019

Need.

needthis

@zadjii-msft
Copy link
Member

LOL I was watching the stream and I just thought to myself "here's my boss assigning me work live in front of a studio audience".

@WSLUser
Copy link
Contributor

WSLUser commented May 10, 2019

Please make this a priority for v1.0!

@WSLUser
Copy link
Contributor

WSLUser commented May 10, 2019

3d animations can be v1.5 😛

@bef178
Copy link
Contributor

bef178 commented May 15, 2019

OMG

@ghost ghost added the Needs-Tag-Fix Doesn't match tag requirements label May 17, 2019
@miniksa miniksa added Product-Terminal The new Windows Terminal. and removed Mass-Chaos labels May 17, 2019
@ghost ghost removed the Needs-Tag-Fix Doesn't match tag requirements label May 18, 2019
@miniksa miniksa added Needs-Tag-Fix Doesn't match tag requirements Product-Conpty For console issues specifically related to conpty labels May 18, 2019
@ghost ghost removed the Needs-Tag-Fix Doesn't match tag requirements label May 18, 2019
@lofcz
Copy link

lofcz commented Jun 7, 2019

Upvoting this request, Sixels would be such an amazing thing to have in the Terminal.

@ghost
Copy link

ghost commented Aug 5, 2019

This weekend I finished implementing sixel read support for my MIT-licensed Java-based TUI library, and it was surprisingly straightforward. The code to convert a string of sixel data to a bitmap image is here, and the client code for the Sixel class is here.

I have done very little for performance on the decoder. But when using the Swing backend, performance is still OK, as seen here. (The snake image looks bad only because byzanz used a poor palette creating the demo gif.) I was a bit taken aback how quickly it came together. It's very fair to say that the "decode sixel into bitmap" part is the easy bit, the hard bit is the "stick image data into a text cell, and when that is present blit the image to screen rather than the character".

Just want to mention it to other folks interested in terminal support for sixel, and hoping it could help you out.

@kfarmer-msft
Copy link

I'll upvote if someone else writes a Jupyter notebook client ;)

@WSLUser
Copy link
Contributor

WSLUser commented Aug 6, 2019

We already have an example of Sixel support in mintty which is written in C (vice java). Only thing needed is a refactor to C++ (at least for initial support). Still always good to see how it's been implemented in other projects.

@ofek

This comment was marked as spam.

@zadjii-msft

This comment was marked as off-topic.

@Erquint

This comment was marked as off-topic.

@artcancrotp
Copy link

sixel support is rapidly becoming as mandatory as ANSI escape sequence support. Please do not allow Terminal to become as obsolete as the old Windows console host by not implementing sixel.

@anderspitman
Copy link

As far as I know, terminal emulators are the only platform capable of running cross platform, statically linked, mouse driven apps that can even be run remotely over ssh.

The main feature missing from this platform is a universally implemented graphics API. I've spent a bit of time researching this, and I believe implementing at least one of sixel, kitty, or iTerm2 would be well worth the effort.

@piranna
Copy link

piranna commented May 6, 2024

IS there something similar to sixel, but for audio? I think it would makes sense over SSH connections...

@DHowett
Copy link
Member

DHowett commented May 6, 2024

IS there something similar to sixel, but for audio? I think it would makes sense over SSH connections...

Similar, insofar as that representing anything even moderately complex will require extreme downsampling? Absolutely there is: DECPS.

@rgpublic
Copy link

rgpublic commented May 6, 2024

@piranna : Off-topic here, but apart from what @DHowett mentioned, you probably have more success to stream audio via a pulseaudio TCP connection which you can tunnel via SSH. 💡

@piranna
Copy link

piranna commented May 6, 2024

@rgpublic I know it's offtopic, sorry for that, but hoped you could guide me. Good to know something exists, thank you to both of you :-)

@marcIhm
Copy link

marcIhm commented May 28, 2024

I hope this i the central issue for image display, so I am following this. Unfortunately there seem to be many other issues which essentially request the same feature, so that its relevance might be underestimated ...

@zadjii-msft
Copy link
Member

@j4james How stale is this: #448 (comment) branch? With eyes on #1173 in the immediate term, I wanna see just how far away that would be from merging in a full passthrough world

@j4james
Copy link
Collaborator

j4james commented May 31, 2024

@zadjii-msft I actually just got my sixel branch merged with main in the past week or so. And with the latest progress on #1173 I was going to ask whether it might be a good time to create a PR for it. Worst case I thought we could put it behind a feature flag until the Windows Terminal side is ready.

That said, I've still got a lot of cleaning up to do before the code is presentable, so it may be a while before I can create a PR. Also, I should note that it only has a GDI renderer - I was hoping somebody else would handle the Atlas implementation, because I know very little about how that works.

@zadjii-msft
Copy link
Member

NOW IS THE TIME

@zadjii-msft
Copy link
Member

Sorry I got a little excited but yes - I think this would be the perfect time to start merging that in. We can absolutely merge this all piece by piece. We can work on merging the conhost sixel implementation while passthrough is getting ready. Then we can get the Atlas sixel implementation in in post.

@lhecker
Copy link
Member

lhecker commented May 31, 2024

I was hoping somebody else would handle the Atlas implementation, because I know very little about how that works.

Once I have an idea of what you need, I'll give you an implementation the same day. :)
FYI ideally, I need a per-sixel-texture ID so that I can cache the CPU-side texture as a GPU-side one between frames. But it will also work sufficiently well without that, at least in the beginning.

@TheXenocide
Copy link

This is my favorite GitHub notification in today's catch-up. So excited to see this make it in 😁

@eabase
Copy link

eabase commented Jun 5, 2024

And this FFmpeg-SIXEL is crazy cool. IDK why I am so excited for running videos in the terminal... or maybe because then we can get graphics output directly from python scripts in both REPL and from posh CLI.
What's the ETA for this?

@kasper93
Copy link
Contributor

kasper93 commented Jun 5, 2024

I would rather recommend mpv. And you can already play movies in the terminal with mpv --vo=tct --really-quiet <file>, obviously this is not what can be achieved with --vo=sixel or --vo=kitty, they are both supported in mpv and will just work ones they hit Terminal.
image

github-merge-queue bot pushed a commit that referenced this issue Jul 1, 2024
## Summary of the Pull Request

This PR introduces basic support for the Sixel graphics protocol in
conhost, limited to the GDI renderer.

## References and Relevant Issues

This is a first step towards supporting Sixel graphics in Windows
Terminal (#448), but that will first require us to have some form of
ConPTY passthrough (#1173).

## Detailed Description of the Pull Request / Additional comments

There are three main parts to the architecture:

* The `SixelParser` class takes care of parsing the incoming Sixel `DCS`
  sequence.
* The resulting image content is stored in the text buffer in a series
  of `ImageSlice` objects, which represent per-row image content.
* The renderer then takes care of painting those image slices for each
  affected row.

The parser is designed to support multiple conformance levels so we can
one day provide strict compatibility with the original DEC hardware. But
for now the default behavior is intended to work with more modern Sixel
applications. This is essentially the equivalent of a VT340 with 256
colors, so it should still work reasonably well as a VT340 emulator too.

## Validation Steps Performed

Thanks to the work of @hackerb9, who has done extensive testing on a
real VT340, we now have a fairly good understanding of how the original
Sixel hardware terminals worked, and I've tried to make sure that our
implementation matches that behavior as closely as possible.

I've also done some testing with modern Sixel libraries like notcurses
and jexer, but those typically rely on the terminal implementing certain
proprietary Xterm query sequences which I haven't included in this PR.

---------

Co-authored-by: Dustin L. Howett <dustin@howett.net>
@mominshaikhdevs
Copy link

since #17510 is merged, maybe close this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Needs-Tag-Fix Doesn't match tag requirements Product-Conpty For console issues specifically related to conpty Product-Terminal The new Windows Terminal.
Projects
None yet
Development

No branches or pull requests