Allow mouse clicks to reposition cursor by emitting cursor key sequences

[sosnik]

Description of the new feature/enhancement

Windows Terminal version: 1.4.3243.0
Windows Version: Microsoft Windows [Version 10.0.19042.630]
Also using:
    GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin) under cygwin
    GNU bash, version 4.4.23(2)-release (x86_64-pc-msys) under msys2
    GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu) under Ubuntu 20.04 WSL 2

It is not possible to place the cursor with the mouse in any of these shells under Windows Terminal. It is, however, quite possible to do this when running bash with msys2, cygwin or wsl on mintty (or WSL terminal).

This issue was discussed in #4268, where it was dismissed because I'm not sure I know of any shell application that actually supports that tbh.. This is patently not the case as it is supported in the integrations between bash and mintty (and the same bash as used by mintty is being loaded in Windows Terminal).

Mouse support was tracked in #376, #545 and #5177. There was a pull request in #/pull/4859 to close some of these issues, however cursor placement still doesn't appear to work in the applications I listed.

[zadjii-msft]

Okay which terminal emulator are you seeing this behavior in? I just tried a plain old conhost with Ubuntu20.04, and I'm not seeing bash request mouse mode.

Same thing with Msys

Both these have pretty vanilla setups.

Is there something that you have to set in your bashrc to configure bash to request mouse mode? Mouse input is a little complicated here. Win32 mouse input doesn't work in the Windows Terminal, but VT-style mouse input should work.

[sosnik]

cygwin, msys, msys2, and wsltty all use mintty as the terminal emulator. As far as I am aware, there's nothing special in my bash dotfiles. Here is a gif of mouse positioning in cygwin:

Here's one for msys2:

And one for Ubuntu WSL running under mintty:

mintty itself has a setting to place mouse cursor with clicks:

To access this settings pane: Click Title bar of application running under mintty (e.g. msys) -> Go to Properties > Go to Mouse Tab

I am assuming this is a feature provided by mintty and not necessarily by bash (compiled for these platforms) because opening bash.exe from within cmd (conhost) or through the Windows Terminal application does not produce this result. One would assume that if this was functionality internal to bash, it would work in those other requirements.

Therefore, I believe this should be a feature that can be implemented within Windows Terminal.
This is the mintty repo: https://github.com/mintty/mintty

[skyline75489]

Yeah the "Clicks place command line cursor" is the key. I can confirm this with wsltty.

[skyline75489]

In Terminal.app & iTerm2 on macOS, you can hold down option while clicking to jump in the prompt line. I just found out this and I think this is actually a helpful feature.

@sosnik I appreciate the effort for mentioning the implementation of mintty. But due to the licensing difference(WT being MIT and mintty being GPL), it is not legal or ethical for us to use the actual code of GPL-licensed projects. Still I like the feature. I'd like this added to the backlog.

[sosnik]

I appreciate the effort for mentioning the implementation of mintty. But due to the licensing difference(WT being MIT and mintty being GPL), it is not legal or ethical for us to use the actual code of GPL-licensed projects.

I understand the legal and ethical limitations. I merely wanted to point out the mintty implementation as proof that 'it can be done'.

Nevertheless, let's see what comes of it. I use a mixture of keyboard and mouse for terminal navigation (sometimes mouse is just more convenient when you're using it with a screen magnifier).

[zadjii-msft]

Ah okay. So this still isn't a shell feature, it's a terminal one. Glad I wasn't totally off base in #4268.

I'll keep this open to be the "add a setting to allow mouse clicks to send arrow keypresses issue". It's a bit of a wonky setting (IMO), but hey there's prior art we can use for it. I'd wonder how this would interact with something like vim's visual selection mode with the mouse.

I'd probably end up sticking this under the experimental. namespace, mostly because "this may not work the way you'd expect"

[malxau]

I'm really curious how this works. Using mintty in git bash, I can see that the mouse click can move the cursor across lines (which blows my mind.) When we'd spoken about this before, the big question I had is how to decide when to emit vertical cursor keys vs. assume a line has wrapped and emit many more horizontal keys. The "correct" answer is application dependent. (Is readline handling mouse clicks itself for this?)

Using SSH so I can have Win32 console processes under mintty, the experience degrades somewhat, because the mouse click can only move the cursor along the same line that it's currently on. It's clearly trying to guess the number of keystrokes to emit and send them to the application though - I used tabs so when it calculated the number of cells to move it didn't match the number of characters seen by the application, and the cursor moved to the wrong place.

[DHowett]

Yeah, I am deeply conflicted about this feature. Cool and magical when it works, “something is a bit off” when it doesn’t.

Also, can’t msys2 host win32 processes natively now? They added ConPTY support recently. Why use SSH for that? This may warrant a side thread.

[malxau]

@DHowett My fault, I was using an old version of Git for Windows that I had lying around. Using the current version and enabling ConPTY during setup means I don't need to SSH. Without SSH though, results are the same - it only emits keystrokes for navigating around the current line.

[sosnik]

mintty manages to incorporate a lot of mouse support indeed. It also intercepts scroll events and will scroll within a console window (like more or man) instead of scrolling the terminal output (current behaviour with Windows Terminal). However there are some variations in how interactive console applications are handled by mintty and applications running with it. For instance, cygwin running in mintty works with nodejs repl, msys2 running under mintty doesn't, but msys2 running under Windows Terminal does.

There are probably different builds of mintty at work, and perhaps some custom interactions with the shell/rest of the system.

I'd wonder how this would interact with something like vim's visual selection mode

I haven't had occasion to use it before, vim is one place where I'm fully keyboard-based.

[DHowett]

intercepts scroll events

at least that one is standardized and an application requests it via DECSET.

[dkaloc]

Hi.

Not sure how much relevant this is, but I wrote some tiny shell scripts implementing placing the CLI text cursor on mouse clicks for Zsh running in an xterm-compatible terminal.

The solution is yet undocumented (sorry for that :-)) and kind of brute-force, but as an exchange, you get a script supporting multiline, glyphs of variable widths and prompts (left and right) of any complexities.

Usage. If you are a Prezto user, do the following:

1. Enable additional module locations:

# Add additional directories to load prezto modules from
zstyle ':prezto:load' pmodule-dirs $HOME/.zprezto-contrib

2. Download the 'mouse-toolkit' module:

mkdir $HOME/.zprezto-contrib/mouse-toolkit && \
curl https://raw.githubusercontent.com/dkaloc/misc/main/mouse-toolkit.zsh > $HOME/.zprezto-contrib/mouse-toolkit/init.zsh && \
chmod o+x $HOME/.zprezto-contrib/mouse-toolkit/init.zsh

3. Download the 'zle-cursor-placement' module:

mkdir $HOME/.zprezto-contrib/zle-cursor-placement && \
curl https://raw.githubusercontent.com/dkaloc/misc/main/zle-cursor-placement.zsh > $HOME/.zprezto-contrib/zle-cursor-placement/init.zsh && \
chmod o+x $HOME/.zprezto-contrib/zle-cursor-placement/init.zsh

4. Enable and configure the modules in your zpreztorc.

...
# Set the Prezto modules to load (browse modules).
# The order matters.
zstyle ':prezto:load' pmodule \
  ... \
  'mouse-toolkit' \
  'zle-cursor-placement'
...
#
# Mouse Toolkit
#
zstyle ':prezto:module:mouse-toolkit' zle-mouse-mode 'buttons-only'
zstyle ':prezto:module:mouse-toolkit' capture-mouse-in-zle 'yes'
...

5. And finally, make use of these modules by appending this to your zshrc:

###########################################################################
########### PLACE CURSOR on MOUSE CLICK ###################################
###########################################################################

_place-cursor-on-mouse-click() {
  ((MOUSE_TOOLKIT_BUTTON == 1)) && \
    zle zle-cursor-placement-place-cursor-to-cell \
      $MOUSE_TOOLKIT_COLUMN \
      $MOUSE_TOOLKIT_ROW
}
zle -N _place-cursor-on-mouse-click

mouse_toolkit_add_handler \
  $MOUSE_TOOLKIT_EVENT_BUTTON_CLICKED _place-cursor-on-mouse-click

Cheers,
Dusan

PS: Tried hard to also support bash and fish shells, but unfortunately they lack the right tools in their shell program-facing APIs.

[dkaloc]

Forgot to mention that the code "steals" your mouse when ZLE is active.

But if it doesn't work for you (and I guess it doesn't), then set the capture-mouse-in-zle to 'no' and:

1. Define some "mouse enablers". For example:

_enable-mouse-for-placing-cursor() {
  mouse_toolkit_enable_sgr_encoded_buttons_only_mode
  zle mouse-toolkit-start-listening-to-sgr-encoded-events-in-zle
}
zle -N _enable-mouse-for-placing-cursor

_disable-mouse-for-placing-cursor() {
  zle mouse-toolkit-stop-listening-to-sgr-encoded-events-in-zle
  mouse_toolkit_disable_sgr_encoded_buttons_only_mode
}
zle -N _disable-mouse-for-placing-cursor

2. And bind them to some key sequences:

bindkey '^M' _enable-mouse-for-placing-cursor
bindkey '^m' _disable-mouse-for-placing-cursor

3. [optional] But best combined with some AutoHotkey script. For example:

#If WinActive("ahk_exe WindowsTerminal.exe")

^LButton::
BlockInput On
SendInput ^[:M ; bindkey '^[:M' _enable-mouse-for-placing-cursor
SendEvent {Click}
SendInput ^[:m ; bindkey '^[:m' _disable-mouse-for-placing-cursor
BlockInput Off
return

#If

[christianparpart]

After reading through almost all of this, because i initially also liked that idea and considered implementing it on my end too, i now think it is absolutely wrong to implement that on the terminal emulator end.

That is what the mouse reporting protocols are there for and it is the applications duty to make use of them instead of having the terminal emulator trying to brute force through them. I think it should rather be sent out a feature request to the shells/libs (fish, bash, libreadline, libedit,, maybe zsh for first-class support) to make use of the existing mouse reporting VT sequences to get that. :-)

[malxau]

I think it should rather be sent out a feature request to the shells/libs (fish, bash, libreadline, libedit,, maybe zsh for first-class support) to make use of the existing mouse reporting VT sequences to get that. :-)

I may be missing something, but can you elaborate a bit more on how this would work?

AFAIK, the issue is that once an application requests mouse sequences, the terminal stops processing mouse sequences. That includes things like selection and scrolling. In the past (with conhost) I went down the "boil the ocean" route of trying to re-implement these features, which isn't ideal and leaves a real tension between two components trying to do the same thing and doing them slightly differently. With Terminal that isn't possible, because it doesn't expose the viewport (so no scrolling, see #10191 ), and there's no ability to read back buffer contents outside of the viewport, so selection doesn't work well either.

I think making this work would require fine grained mouse control specification, where an application can say things like "hey terminal, please tell me about mouse clicks, but please continue to handle mouse drag or scroll the way you previously did."

[zadjii-msft]

FWIW, we take comments like the above into account, but hide them because of the visual noise they add to the thread. IMO, in the middle of the technical discussion of how to implement this, it's best to collapse comments reiterating a desire for the requested feature. Sorry, we certainly meant no offense!

[softworkz]

Sometimes, dev discussions need a little tear-down "back to earth and reality". That was my impression while following the discussion and my motivation for posting the above. I removed my last sentence and you might want to hide this and your previous comment to avoid "pollution" :-)

[DHowett]

Sometimes, dev discussions need a little tear-down "back to earth and reality".

Naw, you're totally right. Thanks for bearing with us!

[zadjii-msft]

x-linking a thread that looks relevant: "Passive Mouse Tracking & Text Selection Tracking VT extensions.", terminal-wg/specifications#31

[christianparpart]

@zadjii-msft this is exactly what I was aiming for, while the spec I wrote it as general purpose as it can get, I also wrote a small sample CLI app to demo the event reporting. If you find design flaws, please /cc me in any way. :)

[zadjii-msft]

huh, just noticed this block on invisible-island

CSI ? Pm h
      DEC Private Mode Set (DECSET).
        Ps = 2 0 0 1  ⇒  Enable readline mouse button-1, xterm.
        Ps = 2 0 0 2  ⇒  Enable readline mouse button-2, xterm.
        Ps = 2 0 0 3  ⇒  Enable readline mouse button-3, xterm.
        Ps = 2 0 0 4  ⇒  Set bracketed paste mode, xterm.
        Ps = 2 0 0 5  ⇒  Enable readline character-quoting, xterm.
        Ps = 2 0 0 6  ⇒  Enable readline newline pasting, xterm.

and also on https://github.com/mintty/mintty/wiki/CtrlSeqs#readline-mouse-modes

These three mode settings, switched by DECSET/DECRST sequences (xterm 379) enable mouse-controlled editing on the command line (as detected by the cursor position) by sending virtual cursor or erase keystrokes.

sequence	mode
^[[?2001h	left button places cursor on command line
^[[?2002h	middle button pastes at current mouse position
^[[?2003h	double right-click deletes selection until mouse position

did this get added to xterm while no one was looking?

[j4james]

did this get added to xterm while no one was looking?

I think that stuff has existed for years - it's only just been documented recently. And I think it was (possibly still is) behind a build flag, so maybe wasn't enabled by default.

[GaryAtlan82]

This has been a long issues of mine, my sentiments are the same as @softworkz.
I am still fairly new to this opposing Terminal vs GUI world. I really dont like seeing them as two diametrically oppossed approaches to using a PC and have been trying to reduce this friction as much as I can.

This feature is really at the core of this friction. A universal means of setting cursor position and even selecting words, regardless of the underlying client, would be such a watershed momment.

You guys have done some amazing stuff with windows terminal, I know you will achieve this too.

[zadjii-msft]

The dumbest possible solution: Only emit keys WHEN:

• the setting is on
• The click is somewhere after the "active command" mark.¹
• You clicked on the line WITH THE CURSOR.

and even then just blindly emit one arrow key per cell. Literally as dumb as possible.

Does it work for wrapped lines? Nope.
Does it work for vim? Nope.
Does it work for emoji/wide glyphs? I mean, emoji input is fucked anyways, right?
Does it do selections? Nope.
Lines with continuation prompts? As long as you don't click on the > , sure?

In VT mouse mode? Don't do it. The app wanted mouse.
Pressed shift? Don't do it. Make a selection.

---

I hate that I feel okay with this. Like, yes, yes, the shell should handle this, there's so many ways for it to not work, but like... idk. Now that I see it in action... it seems more okay. We could probably round out some of the edge cases to make it a little more accurate.

Maybe just cause I'm high on shell integration so anything that looks like an application for it, I'm like YEA LET'S DO IT. I need an unbiased opinion.

Footnotes

1. This means it requires shell integration to be enabled. ↩

[j4james]

The dumbest possible solution: Only emit keys WHEN:

* the setting is on

* The click is somewhere _after_ the "active command" mark.[1](#user-content-fn-1-bf5dd1751f3b9b514bf4e8568fc9c371)

* You clicked on the line WITH THE CURSOR.

and even then just blindly emit one arrow key per cell. Literally as dumb as possible.

I think this is perfectly reasonable. Tying it to the active command makes it feel a lot safer than just triggering it on arbitrary clicks. I'd even go so far as to say that we could probably accept clicks on any line from the active command mark and below (assuming the cursor is also past the active command mark).

[softworkz]

If it doesn't work for wrapped lines, it's not much useful, because it's those wrapped lines where it 's most tedious to navigate to a specific position via cursor keys.

[zadjii-msft]

I'm tempted to call this closed by #15758 - I'm not sure what more we'd do here.

I'm also not sure I'd know when to call this not experimental. It's like, pretty definitively not going to work all the time, so I want that to be clear. But it does work most of the time?

[christianparpart]

@zadjii-msft while I am aware that this might be the most straight forward solution in terms of getting it quickly done, it comes with its caveats, as very well known.

Some months ago, I was drafting a potentially better solution and also gathered some feedback from @jerch, textshell and others directly or over there on IRC.
What would you think about a solution such as:

• Passive Mouse Tracking Extension
• Text Selection Tracking Extension

This should probably not change the decision on closing this ticket here, but I'd like to raise your interest in thinking more forward for a potential better solution.

NB: I've experimentally implemented the above proposals on my TE end along with a simple CLI example. But shells so far (the one I asked for) is still a little conservative on change, but that's nothing new to me. Looks like I'll also end up implementing a shell, eventually :)

[softworkz]

I'm tempted to call this closed by #15758 - I'm not sure what more we'd do here.

I think there's a bunch of things left to do:

• Add it to the schema
  (it's currently neither in in https://aka.ms/terminal-profiles-schema nor in https://aka.ms/terminal-profiles-schema-preview)
• Stop calling it experimental
• Add it to the config UI
• Make it a single-click activation
  After half an hour, I still haven't figured out how to enable it. The docs are pointing to "How to enable shell integration marks", but this page is only talking about "marrks on scrollbar", "right-click menu" and "automarkPrompts".
  Seriously: I want none of that. I also don't want to modify the prompts, nor even deal with this in any way. I just want to be able to "click to position the cursor" - that's all. How to do it without messing around with those things?
• Move it from "advanced" to a "general" feature
  It is currently hidden deeply amongst a hundred of features, most of which I had never thought of anybody would ever want or need, while this is the single most important one of all (IMHO)

Note: I got it working a while ago on a different machine with the preview, but it took me a long time to get there and I don't remember what I did. It's incredibly painful, having to deal with changing the prompts (even though I don't want to change them actually).
But what I can say from the time where I had it working: It's all fine they way it is and how it works. None of those cases you are frequently talking about - where it doesn't work - is of any interest or will ever be of any interest to me with regards to this feature.
IMO it's all good, except the way for enabling it.

[sarim]

• Stop calling it experimental

It is called experimental because it is experimental. Few weeks ago I tried to use it, but its unstable for daily usage. It frequently hanged the terminal by putting itself in infinite loop of sending left arrow keystroke, activate itself while trying to copy text by left click and dragging etc... I didn't had the time to debug so stopped using it for now. Will try to debug and report issue when I get the time.

• Add it to the schema

Should be fixed when #16652 lands.

[zadjii-msft]

Add it to the schema

(as mentioned above: #8573 (comment))

Stop calling it experimental

Yea, I think I've lined out at length in this thread why I believe that this needs to remain an "experimental" feature. A permanent one, but one that's got enough edge cases that the average user will likely run into one and not understand why.

Add it to the config UI

Already done in #16809

Make it a single-click activation

I've got designs of my own to make "automatic shell integration" work, in the same vein as the way VsCode does. That however, is far beyond the scope of this thread. #13445 is probably the closest in spirit, though I'll probably fork off a dedicated thread when that gets closer to landing. I'd much rather keep threads more tightly coupled with atomic bits of work, and I'm not about to hold this one open, waiting on some piece of work that can be done independently.

---

@christianparpart Sorry I missed that comment! I don't even think I was on vacation or anything, so I have no idea how it slipped by. Passive mouse tracking looks great for shells that can support it. I think this is a good example of "yes, and" - passive mouse tracking is better for the shells that can support that. For crochety old men like me who like CMD, then I'll have to make do with this experimental mode

wait

can we have OpenConsole just opt in to passive tracking? Like, always send openconsole passive events, if we're not in VT mouse input mode? This is a half backed thought. But maybe there's a way for us to know that

• the client app didn't actually ask for mouse input
• and it's doing a cooked read
• but OpenConsole can handle mouse input to move the cursor within the cooked read

I'd need think on that for a while. Feels like the dough is under-proved on that thought.

[softworkz]

Add it to the config UI

Already done in #16809

Awesome!

Make it a single-click activation

I've got designs of my own to make "automatic shell integration" work, in the same vein as the way VsCode does. That however, is far beyond the scope of this thread. #13445 is probably the closest in spirit, though I'll probably fork off a dedicated thread when that gets closer to landing. I'd much rather keep threads more tightly coupled with atomic bits of work, and I'm not about to hold this one open,..

Yea sure, I didn't mean to say this needs to be kept open, I was merely afraid that this might remain an unknown and forgotten feature, which is hardly accessible and lastly will make me hate all this each time I'm on a different machine ;-)

Would you kindly help me out with what exactly I need to do to enable this for cmd and PowerShell without changing anything else (it's all at defaults)?
And maybe add it to the docs page to which the text about the setting is linking to but which doesn't even mention it and neither discriminates which modification is needed for which feature exactly.

[cow1337killer3]

Wow it's 2024 and we still can't move a terminal cursor up and down. Welcome to the future, it's mind blowing.

[zadjii-msft]

@cow1337killer3 If you read the thread, you'll probably learn how that's a responsibility of the shell, not the terminal emulator itself. Up is pretty universally "go to the previous command in the history" across shells. And as discussed, we'd all love for the shells to support clicking to move the cursor in the prompt themselves, but there's myriad reasons why that's Hard. That's why the terminal added this feature to attempt to work around that issue.

[vblazhkun]

I guess the following behaviour is related to this feature. When I click with the LMB in any console app waiting for an input (ssh-keygen, cat, etc.) the whole screen is filled with ^[[D aka \x1b5b44:

And it infinitely fills up with just 2-3 consecutive clicks.

[DHowett]

Yes. If you don't have shell integration enabled to indicate where your prompt starts/ends and you are trying to use this feature (which is not enabled by default! for this reason!) you may see an unexpected spew of cursor positioning input.

[vblazhkun]

I have the shell integration enabled (all 4 escape sequences implemented) and its markers are visible on the right side of the screenshot.