Fixes #1265 - Adds Sixel rendering support

[tznind]

Sixel

See also this discussion: #1265

Sixel is a graphics rendering protocol for terminals. It involves converting a bitmap of pixels into an ASCII sequence.

Sixel is the most widely supported graphic protocol in terminals such as xterm, mlterm, windows terminal (support still in pre-release - see v1.22.2362.0) etc. To see what terminals support sixel you can go to: https://www.arewesixelyet.com/

Quantization

Sixel involves a palette of limited colors (typically 256). This means we need to do the following:

• Create a palette of 256 colors from an input bitmap
• Map colors during rendering to the palette colors

The class that handles this is ColorQuantizer. For extensibility I have created interfaces for the actual algorithms you might want to use/write. These are IPaletteBuilder and IColorDistance.

Color Distance

Color distance determines how closely two colors resemble one another. I included the fastest algorithm in Terminal.Gui and left the other one in UICatalog

Distance Metric	Speed	Description
Euclidean	Fast	Measures the straight-line distance between two colors in the RGB color space.
CIE76	Slow	Computes the perceptual difference between two colors based on the CIELAB color space, aligning more closely with human vision.

Palette Building

Deciding which 256 colors to use to represent any image is interesting problem. I have added a simple fast algorithm, PopularityPaletteWithThreshold. It sums all the pixels and merges those that are similar to one another (based on IColorDistance) then takes the most common colors.

I have also included in UICatalog a more standard algorithm that is substantially slower called MedianCutPaletteBuilder. This creates a color palette by recursively dividing a list of colors into smaller subsets ("cubes") based on the largest color variance, using a configurable color distance metric IColorDistance.

Encoding

Once the palette is built we need to encode the pixel data into sixel.

Sixel encoding involves converting bitmap image pixels 6 rows at a time (called a 'band'). Each band of vertical pixels will be converted into X ASCII characters where X is the number of colors in the band:

Setup the palette

First we need to describe all the colors we are using. This takes the format:
#0;2;100;0;0

The first number is the palette index i.e. 0 (first color). Next is the color space e.g. RGB/HSL - 2 is RGB. Then comes the RGB components. These are 0-100 not 0-255 as might be expected.

Pick the drawing color

The next step is to specify the first color you painting with, e.g. this orange:

We pick the color by outputting the index in the palette e.g. #1 (pick color 1)

Then we look at the current band and see what pixels match that color, for example the the second and third pixels in this case (highlighted in pink below):

You convert this into a 6 digit binary array (bitmask)

       A sixel is a column of 6 pixels - with a width of 1 pixel

    Column controlled by one sixel character:
      [0]  - Bit 0 (top-most pixel)
      [1]  - Bit 1
      [1]  - Bit 2
      [0]  - Bit 3
      [0]  - Bit 4
      [0]  - Bit 5 (bottom-most pixel)

This is binary 011000 is then converted to an int and add 63. That number will be a valid ASCII character - which is what will be added to the final output.

For example the bitmask is 011000 is 24 in decimal. Adding 63 (the base-64 offset) gives 24 + 63 = 87, which corresponds to the ASCII character W (upper case w).

0x3F in hexadecimal corresponds to 63 in decimal, which is the ASCII value for the character ? [...] sixels use a base-64 encoding to represent pixel data.

After encoding the whole line in this way you have to 'rewind' to the start of the line and do the next color. This is done with $ and picking another color (e.g. #2 - select color 2 of the palette). You only have to paint the colors that appear in the band (not all the colors of the image/palette).

Finally after drawing all the 'color layers' you move to the next band (i.e. down) using - (typically also with $ to go to the start of the row).

Run length encoding

If you have the same ASCII character over and over you can use exclamation then the number of repeats e.g. !99 means repeat 99 times. Then the thing to repeat e.g.
!12~ is the same as ~~~~~~~~~~~~.

Transparency

You can achieve transparency by not rendering in a given area. For example if every 'color layer' has a 0 for a given entry in the bitmask it will not be drawn.

This requires setting the background draw mode in the header to 1

\u001bP0;1;0
instead of
\u001bP0;0;0

Not all terminals support transparent sixels.

Determining Support

It is possible to determine sixel support at runtime by sending ANSII escape sequences. We don't currently support these sequences although @BDisp is working on it here: #3768

So I have added the ISixelSupportDetector interface and SixelSupportResult (includes whether transparency is supported, what the resolution is etc).

I've also left the ANSI escape sequences detector in but commented out. Let me know if you want that removed.

Fixes

• Fixes Supercharged graphics. #1265

Proposed Changes/Todos

• Adds Sixel rendering support

Pull Request checklist:

• I've named my PR in the form of "Fixes #issue. Terse description."
• My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
• My code follows the Terminal.Gui library design guidelines
• I ran dotnet test before commit
• I have made corresponding changes to the API documentation (using /// style comments)
• My changes generate no new warnings
• I have checked my code and corrected any poor grammar or misspellings
• I conducted basic QA to assure all features are working

[dodexahedron]

Hey I had a thought to bounce off of ya:

This seems like something that would be particularly well suited as a pluggable component, in whatever form that may take, be it a satellite assembly/module, dynamically loaded libraries in a configurable path, straight c# code compiled at run-time, or whatever.

In any of them, the glue would be essentially the same - an interface any plugin must implement, with them then being free to do whatever they like beyond that in their own code.

Any thoughts on that?

The work I'm doing on the drivers will make that kind of thing a lot easier for us to provide, since I'm pulling out interfaces for the public API.

[tznind]

Could be an option 🤔. At the moment I am still in the exploration phase.

The driver level bit is basically

• move to x,y
• output sixel

Currently I'm doing this every render pass of driver which is very slow. But I'm not sure how much of that is down to the pixel encoded instructions being unoptimised.

There are 3 areas I'm working on at the moment

• better color palette picking
• understanding and optimising sixel pixel data encoding algorithm
• integrating with driver

If outputting frames is just inherently slow then some kind of 'reserve area' method might be required to allow a single render to persist through multiple redrawing of main ui

But for now I think it is too early to think about plugin - it needs to work first!

Also down the line it might be nice to do more with GraphView e.g. output sixel if available or fallback to existing ASCII

[tznind]

Looking good but for some reason the colors are off, specifically the dark colors. I thought at first it was the pixel encoding that was redrawing over itself with wrong colors or the palette was not having the dark colors or something but after completely replacing the pixel encoding bit I'm pretty sure that is now correct.

Maybe I can improve situation with some more buttons in scenario e.g. to view the palette used.

Test can be run on a sixel compatible terminal with

dotnet run -- Images -d NetDriver

Image encoding (one off cost) is slow, image rendering is relatively fast (but done every time you redraw screen).

I've included a few algorithms because I thought color issue was bad palette generation or bad color mapping. Might scale it back a bit or provide 1 fast implementation in core and the slow ones in UICatalog as examples.

Looking at this its also possible the color structs are off somewhere such that RGB is interpreted as ARGB and so the blue element is missing or something.

Also haven't explored dithering yet. Which seems to be another big area of sixel image synthesis.

[tznind]

Success, bug was indeed just creating the image wrong at the start

Literally the first step in image generation and all because in TG the A is on the right instead of left of the arguments ><.

public static Color [,] ConvertToColorArray (Image<Rgba32> image)
{
    int width = image.Width;
    int height = image.Height;
    Color [,] colors = new Color [width, height];

    // Loop through each pixel and convert Rgba32 to Terminal.Gui color
    for (int x = 0; x < width; x++)
    {
        for (int y = 0; y < height; y++)
        {
            var pixel = image [x, y];
-            colors [x, y] = new Color (pixel.A, pixel.R, pixel.G, pixel.B); 
+            colors [x, y] = new Color (pixel.R, pixel.G, pixel.B); // Convert Rgba32 to Terminal.Gui color
        }
    }

    return colors;
}

[dodexahedron]

Have you tested out how it behaves if you've altered your terminal color settings/environment variables? Like...do you get predictably ruined colors, if in an indexed color mode, or does it do its best to try to force "correct" colors?

My assumption would be that the output will depend on color depth, with only ANSI or other indexed color schemes being subject to any silliness from that, and consoles capable of true color looking right no matter how ugly one's terminal color scheme may be. But that's just conjecture based on how I'd expect other things to work in most terminals without monkey business going on under the hood. 🤷‍♂️

[dodexahedron]

Success, bug was indeed just creating the image wrong at the start

Literally the first step in image generation and all because in TG the A is on the right instead of left of the arguments ><.

Color can be constructed as ARGB or RGBA. Just pass it the bytes as an int or uint, or if sixel exposes the raw value as an RGBA or ARGB value, use that directly for the constructor.

If it does not expose the whole 32-bit value and you can only get to the bytes, here is each way of doing it in one line and all on the stack:

// The int constructor is RGBA
new Color (BitConverter.ToInt32 ([pixel.R, pixel.G, pixel.B, pixel.A]));
// The uint constructor is ARGB
new Color (BitConverter.ToUInt32 ([pixel.A,pixel.R, pixel.G, pixel.B]));

I could add a direct implicit cast if you like, to make life easier while using it. 🤷‍♂️

IIRC, the uint vs int decision was based on the same or very similar design with System.Drawing.Color, for consistency.

[tznind]

Color can be constructed as ARGB or RGBA

Yup, wasn't meaning that there was a problem with the constructor param order just that I made mistake right at the start and kept thinking issue was with palette generation.

sixel exposes the raw value as an RGBA or ARGB value

Sixel exposes RGB only (no A) and it is on a scale of 0-100. You can define up to x colors (typically 256) and those can be any RGB values you like.

For example

#0;2;100;0;0

The # indicates that we are declaring a color. The 0 is the index in the palette we are setting (i.e. the first color). The 2 indicates Type (RGB) - its basically always going to be a 2. Then you have RGB as 0-100 scaled.

So the above declares the color red (255,0,0) as palette entry 0.

The above text string is the the pure ASCII that you would output to the console when redering the sixel.

You use the palette colors when you encode the pixel data. Rendering pixels involves selecting a color index (from palette) then filling along the band (6 pixels high) with it. Then either 'rewinding' to start of band and drawing with a different color or moving to next band.

Have you tested out how it behaves if you've altered your terminal color settings/environment variables?

At this stage I am making something that works and then writing tests and documenting. I have tested in Windows Terminal Preview (the one that supports sixel) and ML Term (on linux).

There are some terminals that support limited palette sixel (e.g. 16 colors instead of 256). But I think generally if a terminal supports sixel it probably supports true color too.

Once it is done it can be tested for compatibility under corner cases like color setting changes. I think compatibility will have to be left to the user i.e. user can set a config value to support sixel rather than trying to dynamically detect it based on environment vars etc.

[dodexahedron]

Interesting.

As for the color values for conversion purposes, I'd suggest an extension method on Sixel colors in the spirit/convention of the common ToBlahBlah or FromBlahBlah methods color types often have. If there's value to you in doing that, of course.

I wouldn't suggest any modifications to Color that directly depend on any types from Sixel, for sake of separation and not building in too deep a dependency, though. 🤷‍♂️

[tznind]

This is starting to come together.

I now have a pretty firm understanding of sixel encoding and can write tests explaining the expected encoded pixel data.

TODO:

• more tests
• screen positioning
• sixel to screen coordinates measuring
• potentially some resizing logic in UICatalog (I.e. to show how you ensure sixel doesn't spill out of a View)
• refactor and finalise the 'out of the box' algorithms and move rest to UICatalog

So far I have not really touched the console drivers. Only hooking in via static to NetDriver - where outputting sixel encoded image is 2 lines of code (console move then console write).

Probably need some guidance on how best to implement in drivers once I've done the above

[BDisp]

@tznind the error is related with this #3784. Can you help me on this, please. I'm not very good with scripts. My intended is in using a local Terminal.Gui package in release mode instead of the latest version in the nuget.org. But it must build without errors locally and in the github actions. Thanks.

[tznind]

@tznind the error is related with this #3784. Can you help me on this, please. I'm not very good with scripts. My intended is in using a local Terminal.Gui package in release mode instead of the latest version in the nuget.org. But it must build without errors locally and in the github actions. Thanks.

Hmn interesting. Yes it should be possible to build the output to a local folder with publish -o and then have that be the referenced package source (i.e. local source path).

Something like

dotnet pack --configuration Release --output ./nupkg

Can try and take a look later but have ran out of time today.

[tznind]

@tig this is now ready for review. Happy to discuss any design changes. I've tried to keep it a light touch in terms of interacting with the core drivers etc.

[tig]

Any way you can detect sixel support?

I forgot I turned off the WT preview and coulnd't figure out why things weren't working.

[tznind]

Any way you can detect sixel support?

I forgot I turned off the WT preview and coulnd't figure out why things weren't working.

Yes you can send Device Attributes Request and look for 4 in the response. This is what SixelSupportDetector does but it is commented out for now as it's dependent on #3768

[BDisp]

The screen is always flickering on only mouse moves because this code doesn't check for RunState state is dirty or not, as before, and thus always draw the screen unnecessarily. This image is using CursesDriver.

    public static void RunIteration (ref RunState state, ref bool firstIteration)
    {
        if (MainLoop!.Running && MainLoop.EventsPending ())
        {
            // Notify Toplevel it's ready
            if (firstIteration)
            {
                state.Toplevel.OnReady ();
            }

            MainLoop.RunIteration ();
            Iteration?.Invoke (null, new ());
        }

        firstIteration = false;

        if (Top is null)
        {
            return;
        }

        Refresh ();

        if (PositionCursor ())
        {
            Driver!.UpdateCursor ();
        }

    }

[tznind]

I agree the unnecessary redraw is the problem but i dont think its caused by changes in this pr. I raised it as issue here #3761

I think it happens in v2 develop branch too- you just don't notice it because drawing ascii is fast.

Sixel flashes because you draw the ansi view characters then draw sixel over the top.

I put a check in manually for outputting identical content in windows driver in this pr but really we should find source of bug and fix to how it was when it was working correctly I think

[tig]

@tznind do you want me to merge this now or wait until #3768 and #3791 are done?

[tznind]

Hmn it's technically ready but maybe a little tweak to scenario to make it more 'opt in' . At the moment it looks like it is checking for support when it isn't.

[tznind]

@tznind do you want me to merge this now or wait until #3768 and #3791 are done?

Ok we are good to go.

Theres a wierd thing with TabView not refreshing/redrawing properly until changing tabs. I have put a hack in this scenario to work around it but needs to be a seperate issue if it is not already fixed in your draw/layout code changes @tig

[tig]

@tznind do you want me to merge this now or wait until #3768 and #3791 are done?

Ok we are good to go.

Theres a wierd thing with TabView not refreshing/redrawing properly until changing tabs. I have put a hack in this scenario to work around it but needs to be a seperate issue if it is not already fixed in your draw/layout code changes @tig

TabView is horribly messed up. It completely conflates draw/layout. I've given up trying to fix it in #3798 and have just been "Skip"ing unit tests.

Can you rework the scenario to not use it?

[tznind]

Can you rework the scenario to not use it?

I could... but the 'tab switch' hack works and I assume that the tab view issue is temporary.

Or do you mean view is fully broken in the drawing branch?

Would be a shame to rewrite scenario just to change it back after draw issue is fixed no?

I can take a look at tab view if it would help.

[tig]

Can you rework the scenario to not use it?

I could... but the 'tab switch' hack works and I assume that the tab view issue is temporary.

Or do you mean view is fully broken in the drawing branch?

Would be a shame to rewrite scenario just to change it back after draw issue is fixed no?

I can take a look at tab view if it would help.

I would really appreciate it. If you did it as a branch of the drawing branch so you could also review and give feedback on the new drawing and layout api design that would be extra saucy.