Feature request: mouse mode 1016 (SGR-Pixels)

[ghost]

Abstract

I would like to support pixel-based mouse position in my application.

Two existing methods to do this are DEC Locator Input (DECLRP) which according to that doc is only supported for REGIS and Tektronix; and SGR-Pixels which is supported by xterm and works in normal cases.

Motivation

With SGR-Pixels, one could feasibly make terminal-based applications for image editing, support a custom bitmap/animated mouse pointer, and I'm sure plenty other things.

Other applications desire this too, example: dankamongmen/notcurses#2326

I would try my hand at a PR for you, but I cannot seem to get contour to compile on my system (Debian 11.1 bullseye) :-(

Specification

I would like SGR-Pixels (mouse mode 1016) to be supported. The difference between this and 1006 mode which is already supported are:

• Mouse movement by pixel, not cell, triggers an event.
• The rows/cols reported are in pixels, not cells.
• DECRQM (CSI ? 1016 $ p) produces a corresponding DECRPM so that applications can detect it. (...and BTW thank you for teaching me about DECRQM/DECRPM :-) )

Checking xterm via:

echo -ne '\033[?1003;1016h' ; cat

...shows that click-drag outside the windows does continue to produce SGR-Pixels mouse events, i.e. xterm continues reporting for the entire mouse grab.

[christianparpart]

Cool idea @klamonte. Many thanks for the inspiration.

DECRQM (CSI ? 1016 $ p) produces a corresponding DECRPM so that applications can detect it. (...and BTW thank you for teaching me about DECRQM/DECRPM :-) )

You're welcome, man :)

I quickly tried xterm with this to get a first impression. Workable. But it seems like xterm sometimes returns negative numbers when close to the top/left border?

Also, I was super surprised to not know about this xterm feature, especially since - when I started with this project - I literally implemented as many xterm VT sequences as possible and never read about this mouse mode.

Patch #359 - 2020/08/17
...

• add a new mouse mode 1016, which uses the same format as mode 1006, but sends the mouse's position in pixels (suggested by Igor van den Hoven).

Turns out to be added like one and half a year ago, I should have refreshed my ctlseqs.txt.

p.s.: why bug label?

EDIT: I'm currently super busy on master (and dev branches to be merged to master), so that's not as stable as 0.2.x branch and its releases, working on better resource management / perf. When that's done pretty soon, I'll certainly want to pick this one up. :)

[ghost]

I have no idea how the bug label got there. I swear I pressed the feature request button. 🙂

Interesting that xterm reports negative, but makes sense. It keeps the grab as long as the mouse button is down. I didn't check to see what it did with no-button mouse motion.

[j4james]

DEC Locator Input (DECLRP) which according to that doc is only supported for REGIS and Tektronix

@klamonte I think you might be misreading that. REGIS and Tektronix have their own locator input modes which are different from the Text Locator extension. All the terminal emulators I know that support DECLRP do so independent of REGIS and Tektronix. This includes XTerm, MLTerm, Mintty, RLogin, and VTStar.

@christianparpart I noticed Contour is claiming to support the Text Locator extension (29 in the DA1 report), but you don't seem to implement any of the locator sequences?

[christianparpart]

@j4james that is indeed a bug of wrong understanding back in the time I implemented mouse support. Will remove that. Thanks.

[christianparpart]

Do we know any app already using SGR-Pixels mode so I can verify my implementation isn't bogus? (apart from looking at xterm output and reason about the comparison?) :)

[christianparpart]

..shows that click-drag outside the windows does continue to produce SGR-Pixels mouse events, i.e. xterm continues reporting for the entire mouse grab.

Oh heck! I was reading this just now! Sorry.

I would consider this a bug. Because the mouse pointer is outside the window. The xterm dev maybe had a reason for this. But I consider this bad / dies not Sound right.

What do you think?

(I just remembered, xeyes knows about the mouse location outside is window too, but still, little to no use)... Hmmm

[ghost]

I could see it both ways. Reporting negative coordinates could be handy for something like an image cropper where you have a selection rectangle/rubberband going always to the mouse location (even when outside the window) when being dragged. But it could also just be a bug. SGR 1006 (normal coordinates) clamps everything to positive integers only. It could be that SGR-Pixels was supposed to do that too, and forgot. Or it could be we needed a few more TE's to implement to get some apps that can play with it and really nail it down.

You had asked if an application can test it. Jexer 1.5.0, released last week, can test it. Just open the Pixels window in the demo and either mouse over the field (which will show a custom i-bar icon tracking the mouse by pixel) or click the "Cute mouse" button.

( If you want to have more fun, pull from my git head, same pixels demo button: Xterm animations ;) )

[j4james]

Negative numbers in a CSI-based response are almost certainly a bug, because you can't have negative parameter values. The - character is an intermediate according to ANSI/ECMA rules, and thus would indicate the end of the parameter values and the beginning of the intermediate/final control identifier.

[christianparpart]

I now want a simple terminal drawing demo. Haha. Well. I know that GUIs are better suited. But knows what usernames people might come up with. I am curious.