Spec for shell integration marks

[zadjii-msft]

Summary of the Pull Request

Abstract

"Shell integration" refers to a broad category of ways by which a commandline
shell can drive richer integration with the terminal. This spec in particular is
most concerned with "marks" and other semantic markup of the buffer.

Marks are a new buffer-side feature that allow the commandline application or
user to add a bit of metadata to a range of text. This can be used for marking a
region of text as a prompt, marking a command as succeeded or failed, quickly
marking errors in the output. These marks can then be exposed to the user as
pips on the scrollbar, or as icons in the margins. Additionally, the user can
quickly scroll between different marks, to allow easy navigation between
important information in the buffer.

Marks in the Windows Terminal are a combination of functionality from a variety
of different terminal emulators. "Marks" attmepts to unify these different, but
related pieces of functionality.

Detailed Description of the Pull Request / Additional comments

*^*_* read the spec _*^**

In all seriousness, I've already implemented a pile of this. This is just putting the finishing touches of formalizing it.

PR Checklist

• This is a spec for megathread: Scrollbar Marks #11000 and everything linked to that.

[KalleOlaviNiemitalo]

How does the terminal know what input it needs to send in order to rerun the command?

• user presses Ctrl+V Escape to input an ESC character and the shell displays it as ^[
• user presses Ctrl+V Ctrl+J to input an LF character and the shell displays it as a line break
• user presses Enter within a quoted string and the shell displays a continuation prompt
• user types a command including an exclamation point, and the shell invokes history expansion and echoes the result of expansion before it runs the command

[DHowett]

All excellent questions :)

[lhecker]

I mostly reviewed the technical aspects of the spec. There's just one inaccuracy that I think should be addressed. It might also help us confidently pick alternative ways of storing marks without worrying about performance issues.

I also noticed that while the spec discusses storage and reflow of the marks, I personally think that the hardest aspect is how to draw potentially thousands of marks in the scrollbar. The way it's currently done, Windows Terminal starts to lag around quite a bit during lengthy sessions with large scrollback buffer, because it draws so many rectangles and WinUI and its overhead makes it 100x worse on top (unfortunately). As such I think we need to consider algorithms to downsample marks to match the expected number of marks on the scrollbar. Since iterating through marks is supposedly very fast, such an algorithm will likely end up being very simple but make the UI less laggy. I think we might want to consider this at least by adding such a section as a placeholder.

[lhecker]

Reflow should not be a (realistic) performance problem either IMO. The problem is that we need to associate marks with rows during the reflow, which either downright forces us to store marks in our rows (instead of a separate vector) or store them sorted. This is because in the future we'll copy entire strings of text from the old to the new buffer, which means that all we have is a row/y coordinate and two x coordinates: Where the copied text started and ended. From these 3 numbers we need to get the range of matching marks and translate them to the new text buffer, just like how we translate and copy TextAttributes.

[zadjii-msft]

I also noticed that while the spec discusses storage and reflow of the marks, I personally think that the hardest aspect is how to draw potentially thousands of marks in the scrollbar. The way it's currently done, Windows Terminal starts to lag around quite a bit during lengthy sessions with large scrollback buffer, because it draws so many rectangles and WinUI and its overhead makes it 100x worse on top (unfortunately).

I'm also open to alternative ways of drawing the marks, vs just.... 9000 Rectangles (that I destroy and recreate each time).

[lhecker]

I think the most performant solution would be for us to draw our own scrollbar as part of AtlasEngine. It'd be fairly simple to replicate the exact look that the Win32/WinUI scrollbars have. That way we could efficiently overlay them with a bitmap of marks. This is approximately how Chromium does it (which also draws its own scrollbar with Skia).

Alternatively, we could write a custom WinUI control, create a simple, 1px wide bitmap with Direct2D and stretch it onto our WinUI control. Then we overlay that control on top of the scrollbar. That's about 30 lines of code - if you were to use Direct2D and DirectComposition directly. I have no idea however how to actually create a custom control for WinUI. There's no documentation for it and all the issues over on the repository that did talk about it found no answer either. Does WinUI even support using the underlying Direct2D? I know that Win2D exists, but that can't possibly be the answer, right? That one creates an entire swap chain per surface. Compared to that I think 9000 rectangles are faster and simpler. 🫤

So, I think we should just focus on downsampling the marks. For instance, even on my 4K display my 30 row window is only about 900px tall and thus we could reduce the cost of marks by up to 10x. If the marks are sorted by scrollbar position, the downsampling could occur with a simple loop where we ignore marks if they touch the same pixel as a previous mark (for instance).

[j4james]

I'm not sure this needs to go in the spec, but just FYI, there were a few other operations supported by Final Term that aren't listed here.

For example, E and F were used to indicate things like file names and folders in the output. I only have a vague idea of how they worked, but I think the terminal possibly provided aliases for certain system commands like ls that would "mark up" the output with the appropriate sequences.

So from the user's perspective, you would go into a directory, execute an ls, and the terminal would then be aware of all the files and folders in that directory. That way it could assumedly do fancy things with popup autocomplete menus.

There was also a progress operation, G, which I'm guessing would be similar to what we're currently doing with the ConEmu OSC 9 sequence.

Various other terminals have added their own proprietary OSC 133 extensions as well.

[DHowett]

We can also do a mini spec for "precmd" and "postcmd" CMD hooks if you want ;)

[zadjii-msft]

• HKCU\Software\Microsoft\Command Processor\precmd
• HKCU\Software\Microsoft\Command Processor\postcmd

*shudder*

[DHowett]

Eh. I don't mind them being similar. As long as we refer to them consistently as "Prompt Marks"?

[DHowett]

Since they're really not bookmarks in all cases (since they can be emitted by the app)

[DHowett]

ooh, very interesting

[DHowett]

I think it should, why not

[carlos-zamora]

There's a few comments from last time that still aren't resolved.

For the default value discussions, I'm open to figuring that out when the PRs are being reviewed. Just let me know if that's what you prefer and I'll resolve them.

[carlos-zamora]

Idea: could we add a theme (or maybe a setting in the appearance config?) that defaults the category to a color?

For example:

• "prompt" --> white
• "error" --> red
• "warning" --> yellow
• "success" --> green
• "info" --> white

It'd be really annoying for the user to have to update each addMark action separately.

[zadjii-msft]

Right now, I just use the color values from the color scheme, so that they feel seamless. So error is the red from the color palette. Does that seem good?