status: draft
The goal of the vt-input-mode
protocol is to make command line interactivity cross-platform.
- No TTY required.
- No OS-level signal tracking required.
Anyone who wants to:
- Operate without TTY.
- Share applications on LAN (using inetd, netcat, etc).
- Track every key press and key release.
- Track position dependent keys such as WASD.
- Distinguish between Left and Right physical keys.
- Get consistent output regardless of terminal window resize.
- Track mouse on a pixel-wise level.
- Track mouse outside the terminal window (getting negative coordinates).
- Take advantage of high-resolution wheel scrolling.
- Track scrollback text manipulation.
- Track application closing and system shutdown.
- Be independent of operating system and third party libraries.
Existing approaches have the following drawbacks:
- There is no uniform way to receive keyboard events.
- Window size tracking requires platform-specific calls with no way to synchronize the output.
- Mouse tracking modes lack support for negative coordinates, high-resolution wheel scrolling, and have a limited set of buttons.
- Bracketed paste mode does not support the transfer of binary data and data containing sequences of bracketed paste mode itself.
All numeric values used in this protocol are decimal and zero-based.todo: Should we use HEX-form of the uint32 (IEEE-754 32-bit binary float, Little-Endian) for the floating point value representation?- Space characters are not used in sequence payloads and are only used for readability of the description.
- All unescaped symbols outside of this protocol should be treated as clipboard pasted data.
Signaling uses APC ESC _ <payload> ESC \
with an event-specific payload syntax.
The payload consists of a list of attributes in the following format:
<attr>=<val>,...,<val>; ...; <attr>=<val>,...,<val>
todo: representation of the floating point values: float32 -> uint32 IEEE-754 form -> hex? or just round it up to 0.00001?
Field | Descriprtion |
---|---|
<attr> |
Attribute name. |
<val>,...,<val> |
Comma-separated value list. |
Set: ESC _ events=<Source0>,...,<SourceN> ESC \
Reset: ESC _ events ESC \
Source | Events to track |
---|---|
keyboard |
Keyboard. |
mouse |
Mouse. |
focus |
Focus. |
format |
Line format. |
clipoard |
Clipboard. |
window |
Window size and selection. |
system |
System signals. |
This sequence enables vt-input-mode
and event tracking for the specified event Source
s. The vt-input-mode
is deactivated if none of the Source
s is specified.
Note: By enabling vt-input-mode
, all current terminal modes are automatically saved (to be restored on exit) and switched to something like "raw" mode, in which input is available character by character, echoing is disabled, and all special processing of terminal input and output characters is disabled (except for LF
to CR+LF
conversion).
- Keyboard
ESC _ event=keyboard ; kbmods=<KeyMods> ; keyid=<KeyId> ; pressed=<KeyDown> ; scancode=<ScanCode> ; chord=<KeyId0>,...,<KeyIdN> ; print=<C0>,...,<Cn> ESC \
- Mouse
ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ; wheel=<DeltaY>,<DeltaX> ESC \
- Focus
ESC _ event=focus ; state=<FocusState> ESC \
- Format
ESC _ event=format ; wrapping=<State> ; alignment=<State> ; rtl=<State> ESC \
- Clipboard
ESC _ event=clipoard ; format=<ClipFormat> ; security=<SecLevel> ; data=<Data> ESC \
- Window
ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \
- System
ESC _ event=system ; signal=<Signal> ESC \
ESC _ event=keyboard ; kbmods=<KeyMods> ; keyid=<KeyId> ; pressed=<KeyDown> ; scancode=<ScanCode> ; chord=<KeyId0>,...,<KeyIdN> ; print=<C0>,...,<Cn> ESC \
Q: Do we need to track scancode chord?
scanchord=<Code0>,...,<CodeN>
?
Attribute | Description |
---|---|
kbmods=<KeyMods> |
Keyboard modifiers. |
keyid=<KeyId> |
Physical key ID. |
pressed=<KeyDown> |
Key state: <KeyDown>=1 - Pressed. <KeyDown>=0 - Released. |
scancode=<ScanCode> |
Scan code. |
chord=<KeyId0>,...,<KeyIdN> |
Simultaneously pressed key id's in ascending order. |
print=<C0>,...,<Cn> |
Codepoints of the generated string. |
In response to the activation of keyboard
tracking, the application receives a vt-sequence containing keyboard modifiers state:
ESC _ event=keyboard ; kbmods=<KeyMods> ESC \
The full sequence is fired after every key press and key release. The sequence can contain a string generated by a keystroke as a set of codepoints: C0 + ... + Cn
. The string can be fragmented and delivered by multiple consecutive events.
The chord=<KeyId0>,...,<KeyIdN>
attribute contains the set of simultaneously pressed key id's in ascending order and is used to track key combinations. It is possible to track both chord presses +
(e.g. Ctrl+F1
) and chord releases -
(e.g. Ctrl-F1
or Ctrl-Alt
):
- Pressed chord: The
pressed
attribute has the value1
. - Release chord: The
pressed
attribute has the value0
.
The state kbmods=<KeyMods>
of keyboard modifiers is the binary OR of all currently pressed modifiers and enabled modes.
Bit | Side | Modifier Key | Value |
---|---|---|---|
0 | Left | ⌃ Ctrl | 0x0001 |
1 | Right | ⌃ Ctrl | 0x0002 |
2 | Left | ⎇ Alt ⌥ Option |
0x0004 |
3 | Right | ⎇ Alt ⌥ Option ⇮ AltGr |
0x0008 |
4 | Left | ⇧ Shift | 0x0010 |
5 | Right | ⇧ Shift | 0x0020 |
6 | Left | ⊞ Win ⌘ Command ◆ Meta ❖ Super |
0x0040 |
7 | Right | ⊞ Win ⌘ Command ◆ Meta ❖ Super |
0x0080 |
8 | reserved | 0x0100 |
|
9 | reserved | 0x0200 |
|
10 | reserved | 0x0400 |
|
11 | reserved | 0x0800 |
|
12 | ⇭ NumLock Mode | 0x1000 |
|
13 | ⇪ CapsLock Mode | 0x2000 |
|
14 | ⇳ ScrollLock Mode | 0x4000 |
|
15 | reserved Mode | 0x8000 |
Scan codes are usually useful for applications that need to know which key is pressed, regardless of the current keyboard layout. For example, the WASD (Up, Left, Down, Right) keys for games, which ensure a consistent key formation across QWERTY, AZERTY or Dvorak keyboard layouts.
QWERTY: "W" = 11 "A" = 1E "S" = 1F "D" = 20
AZERTY: "Z" = 11 "Q" = 1E "S" = 1F "D" = 20
Dvorak: "," = 11 "A" = 1E "O" = 1F "E" = 20
Scan codes for the keys on a standard 104-key keyboard:
┌────┐ ┌────╥────╥────╥────┐ ┌────╥────╥────╥────┐ ┌────╥────╥────╥────┐ ┌────╥────╥────┐
| 01 │ | 3B ║ 3C ║ 3D ║ 3E | | 3F ║ 40 ║ 41 ║ 42 | | 43 ║ 44 ║ 57 ║ 58 | |E037║ 46 ║E045|
└────┘ └────╨────╨────╨────┘ └────╨────╨────╨────┘ └────╨────╨────╨────┘ └────╨────╨────┘
┌────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────────┐ ┌────╥────╥────┐ ┌────╥────╥────╥────┐
| 29 ║ 02 ║ 03 ║ 04 ║ 05 ║ 06 ║ 07 ║ 08 ║ 09 ║ 0A ║ 0B ║ 0C ║ 0D ║ 0E | |E052║E047║E049| | 45 ║E035║ 37 ║ 4A |
╞════╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══════╡ ╞════╬════╬════╡ ╞════╬════╬════╬════╡
| 0F ║ 10 ║ 11 ║ 12 ║ 13 ║ 14 ║ 15 ║ 16 ║ 17 ║ 18 ║ 19 ║ 1A ║ 1B ║ 2B | |E053║E04F║E051| | 47 ║ 48 ║ 49 ║ |
╞══════╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩══════╡ └────╨────╨────┘ ╞════╬════╬════╣ 4E |
| 3A ║ 1E ║ 1F ║ 20 ║ 21 ║ 22 ║ 23 ║ 24 ║ 25 ║ 26 ║ 27 ║ 28 ║ 1C | | 4B ║ 4C ║ 4D ║ |
╞═══════╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩══════════╡ ┌────┐ ╞════╬════╬════╬════╡
| 2A ║ 2C ║ 2D ║ 2E ║ 2F ║ 30 ║ 31 ║ 32 ║ 33 ║ 34 ║ 35 ║ 36 | |E048| | 4F ║ 50 ║ 51 ║ |
╞══════╦══╩═╦══╩═══╦╩════╩════╩════╩════╩════╩════╩╦═══╩╦═══╩╦═════╦══════╡ ┌────┼────┼────┐ ╞════╩════╬════╣E01C|
| 1D ║ 5B ║ 38 ║ 39 ║E038║ 5C ║ 5D ║ E01D | |E04B|E050|E04D| | 52 ║ 53 ║ |
└──────╨────╨──────╨───────────────────────────────╨────╨────╨─────╨──────┘ └────┴────┴────┘ └─────────╨────╨────┘
The <KeyId>
is incremented by 2 for each generic key, providing two <KeyId>
placeholders for each physical key to distinguish between Left and Right (or Numpad) in the last bit. Ignore the last bit of <KeyId>
for tracking generic keys.
Key ID | Name | Physical Key | Scan Code | Notes |
---|---|---|---|---|
0 | undef | |||
2 | LeftCtrl | Left Ctrl | 29 0x1D |
|
3 | RightCtrl | Right Ctrl | 57373 0xE01D |
|
4 | LeftAlt | Left Alt | 56 0x38 |
|
5 | RightAlt | Right Alt | 57400 0xE038 |
|
6 | LeftShift | Left Shift | 42 0x2A |
|
7 | RightShift | Right Shift | 54 0x36 |
|
8 | LeftWin | Left Win | 91 0x5B |
|
9 | RightWin | Right Win | 92 0x5C |
|
10 | Apps | Apps | 93 0x5D |
|
12 | NumLock | Num Lock | 69 0x45 |
|
14 | CapsLock | Caps Lock | 58 0x3A |
|
16 | ScrollLock | Scroll Lock | 69 0x45 |
|
18 | Esc | Esc | 1 0x01 |
|
20 | Space | Space | 57 0x39 |
|
22 | Backspace | Backspace | 14 0x0E |
|
24 | Tab | Tab | 15 0x0F |
|
26 | Break | Break | 57414 0xE046 |
Ctrl + Pause |
28 | Pause | Pause | 57412 0xE045 |
|
30 | Select | Select | ||
32 | SysRq | SysRq | 84 0x54 |
Alt + PrintScreen |
34 | PrintScreen | Print Screen | 57399 0xE037 |
|
36 | Enter | Enter | 28 0x1C |
|
37 | NumpadEnter | Numpad Enter | 57372 0xE01C |
|
38 | PageUp | PageUp | 57417 0xE049 |
|
39 | NumpadPageUp | Numpad PageUp | 73 0x49 |
|
40 | PageDown | PageDown | 57425 0xE051 |
|
41 | NumpadPageDown | Numpad PageDown | 81 0x51 |
|
42 | End | End | 57423 0xE04F |
|
43 | NumpadEnd | Numpad End | 79 0x4F |
|
44 | Home | Home | 57415 0xE047 |
|
45 | NumpadHome | Numpad Home | 71 0x47 |
|
46 | LeftArrow | Left Arrow | 57419 0xE04B |
|
47 | NumpadLeftArrow | Numpad Left Arrow | 75 0x4B |
|
48 | UpArrow | Up Arrow | 57416 0xE048 |
|
49 | NumpadUpArrow | Numpad Up Arrow | 72 0x48 |
|
50 | RightArrow | Right Arrow | 57421 0xE04D |
|
51 | NumpadRightArrow | Numpad Right Arrow | 77 0x4D |
|
52 | DownArrow | Down Arrow | 57424 0xE050 |
|
53 | NumpadDownArrow | Numpad Down Arrow | 80 0x50 |
|
54 | Key0 | 0 | 11 0x0B |
|
55 | Numpad0 | Numpad 0 | 82 0x52 |
|
56 | Key1 | 1 | 2 0x02 |
|
57 | Numpad1 | Numpad 1 | 79 0x4F |
|
58 | Key2 | 2 | 3 0x03 |
|
59 | Numpad2 | Numpad 2 | 80 0x50 |
|
60 | Key3 | 3 | 4 0x04 |
|
61 | Numpad3 | Numpad 3 | 81 0x51 |
|
62 | Key4 | 4 | 5 0x05 |
|
63 | Numpad4 | Numpad 4 | 75 0x4B |
|
64 | Key5 | 5 | 6 0x06 |
|
65 | Numpad5 | Numpad 5 | 76 0x4C |
|
66 | Key6 | 6 | 7 0x07 |
|
67 | Numpad6 | Numpad 6 | 77 0x4D |
|
68 | Key7 | 7 | 8 0x08 |
|
69 | Numpad7 | Numpad 7 | 71 0x47 |
|
70 | Key8 | 8 | 9 0x09 |
|
71 | Numpad8 | Numpad 8 | 72 0x48 |
|
72 | Key9 | 9 | 10 0x0A |
|
73 | Numpad9 | Numpad 9 | 73 0x49 |
|
74 | Insert | Insert | 57426 0xE052 |
|
75 | NumpadInsert | Numpad Insert | 82 0x52 |
|
76 | Delete | Delete | 57427 0xE053 |
|
77 | NumpadDelete | Numpad Delete | 83 0x55 |
|
78 | Clear | Clear | 76 0x4C |
|
79 | NumpadClear | Numpad Clear | 76 0x4C |
Numpad 5 |
80 | Multiply | * | ||
81 | NumpadMultiply | Numpad * | 55 0x37 |
|
82 | Plus | + | ||
83 | NumpadPlus | Numpad + | 78 0x4E |
|
84 | Separator | Separator | ||
85 | NumpadSeparator | Numpad Separator | ||
86 | Minus | - | 12 0x0C |
|
87 | NumpadMinus | Numpad - | 74 0x4A |
|
88 | Period | . | 52 0x34 |
|
89 | NumpadDecimal | Numpad . | 83 0x53 |
|
90 | Slash | / | 53 0x35 |
|
91 | NumpadSlash | Numpad / | 57397 0xE035 |
|
92 | BackSlash | \ | 43 0x2B |
|
94 | OpenBracket | [ | 26 0x1A |
|
96 | ClosedBracket | ] | 27 0x1B |
|
98 | Equal | = | 13 0x0D |
|
100 | BackQuote | ` | 41 0x29 |
|
102 | SingleQuote | ' | 40 0x28 |
|
104 | Comma | , | 51 0x33 |
|
106 | Semicolon | ; | 39 0x27 |
|
108 | F1 | F1 | 59 0x3B |
|
110 | F2 | F2 | 61 0x3C |
|
112 | F3 | F3 | 62 0x3D |
|
114 | F4 | F4 | 63 0x3E |
|
116 | F5 | F5 | 64 0x3F |
|
118 | F6 | F6 | 65 0x40 |
|
120 | F7 | F7 | 66 0x41 |
|
122 | F8 | F8 | 67 0x42 |
|
124 | F9 | F9 | 68 0x43 |
|
126 | F10 | F10 | 69 0x44 |
|
128 | F11 | F11 | 87 0x57 |
|
130 | F12 | F12 | 88 0x5B |
|
132 | F13 | F13 | ||
134 | F14 | F14 | ||
136 | F15 | F15 | ||
138 | F16 | F16 | ||
140 | F17 | F17 | ||
142 | F18 | F18 | ||
144 | F19 | F19 | ||
146 | F20 | F20 | ||
148 | F21 | F21 | ||
150 | F22 | F22 | ||
152 | F23 | F23 | ||
154 | F24 | F24 | ||
156 | KeyA | A | ||
158 | KeyB | B | ||
160 | KeyC | C | ||
162 | KeyD | D | ||
164 | KeyE | E | ||
166 | KeyF | F | ||
168 | KeyG | G | ||
170 | KeyH | H | ||
172 | KeyI | I | ||
174 | KeyJ | J | ||
176 | KeyK | K | ||
178 | KeyL | L | ||
180 | KeyM | M | ||
182 | KeyN | N | ||
184 | KeyO | O | ||
186 | KeyP | P | ||
188 | KeyQ | Q | ||
190 | KeyR | R | ||
192 | KeyS | S | ||
194 | KeyT | T | ||
196 | KeyU | U | ||
198 | KeyV | V | ||
200 | KeyW | W | ||
202 | KeyX | X | ||
204 | KeyY | Y | ||
206 | KeyZ | Z | ||
208 | Sleep | Sleep | ||
210 | Calculator | Calculator | ||
212 | ||||
214 | MediaVolMute | Media Vol Mute | ||
216 | MediaVolDown | Media Vol Down | ||
218 | MediaVolUp | Media Vol Up | ||
220 | MediaNext | Media Next | ||
222 | MediaPrev | Media Prev | ||
224 | MediaStop | Media Stop | ||
226 | MediaPlayPause | Media Play/Pause | ||
228 | MediaSelect | Media Select | ||
230 | BrowserBack | Browser Back | ||
232 | BrowserForward | Browser Forward | ||
234 | BrowserRefresh | Browser Refresh | ||
236 | BrowserStop | Browser Stop | ||
238 | BrowserSearch | Browser Search | ||
240 | BrowserFavorites | Browser Favorites | ||
242 | BrowserHome | Browser Home |
ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ; wheel=<DeltaY>,<DeltaX> ESC \
Attribute | Description |
---|---|
kbmods=<KeyMods> |
Keyboard modifiers (see Keyboard event). |
coord=<X>,<Y> |
Pixel-wise coordinates of the mouse pointer. Each coordinate is represented in the form of a floating point value of the sum of the integer coordinate of the cell in the terminal window grid and the relative offset within the cell in the range [0.0f, 1.0f) . |
buttons=<ButtonState> |
Mouse button state. |
wheel=<DeltaY>,<DeltaX> |
Vertical and horizontal wheel high-resolution delta represented as floating point value. |
In response to the activation of mouse
tracking, the application receives a vt-sequence containing current mouse state:
ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ESC \
The mouse tracking event fires on any mouse activity, as well as on keyboard modifier changes.
Bit | Active button |
---|---|
0 | Left |
1 | Right |
2 | Middle |
3 | 4th |
4 | 5th |
Note: Mouse tracking will continue outside the terminal window as long as the mouse button pressed inside the window is active. In this case, coordinates with negative values are possible.
ESC _ event=focus ; state=<FocusState> ESC \
Attribute | Description |
---|---|
state=<FocusState> |
Terminal window focus: <FocusState>=1 - Focused. <FocusState>=0 - Unfocused. |
In response to the activation of focus
tracking, the application receives a vt-sequence containing current focus state.
ESC _ event=format ; wrapping=<State> ; alignment=<State> ; rtl=<State> ESC \
Attribute | Description |
---|---|
wrapping=<State> |
Line wrapping: <State>=0 - Unwrapped. <State>=1 - Wrapped. |
alignment=<State> |
Line alignment: <State>=0 - Left. <State>=1 - Right. <State>=2 - Center. |
rtl=<State> |
Text flow direction: <State>=0 - Left to Right. <State>=1 - Right to Left. |
In response to the activation of format
tracking, the application receives a vt-sequence containing current line format.
ESC _ event=clipoard ; format=<ClipFormat> ; security=<SecLevel> ; data=<Data> ESC \
Attribute | Description |
---|---|
format=<ClipFormat> |
Clipboard data format. |
security=<SecLevel> |
Security level. |
data=<Data> |
Base64 encoded data. |
Format | Description |
---|---|
text |
Plain text. |
rich |
Rich text format. |
html |
HTML format. |
ansi |
ANSI/VT format. |
Bit | Description |
---|---|
0x01 |
Exclude clipboard content from monitor processing. |
0x02 |
Can include in clipboard history. |
0x04 |
Can upload to cloud clipboard. |
ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \
Attribute | Description |
---|---|
size=<Width>,<Height> |
Terminal window size in cells. |
cursor=<X>,<Y> |
Current text cursor position. |
region=<Left>,<Top>,<Right>,<Bottom> |
Scrolling region margins. |
selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> |
Coordinates of the text selection start/end (half-open interval) and text selection mode: <Mode>=0 - Line-based. <Mode>=1 - Rect-based. |
In response to the activation of window
tracking, the application receives a vt-sequence containing current window state.
When the terminal window is resized, the changes are only applied after a handshake between the terminal and the application. Successive multiple resizing of the window initiates the same number of handshakes.
Handshake steps:
- Terminal requests a new size, omitting all other parameters.
- Application receives the request and prepares for resizing, e.g. by updating, cropping or deleting visible lines.
- Application replies with the same message as the request.
- Terminal applies the new size and sends the changes.
- Application updates its UI.
Terminal: ESC _ event=window ; size=<Width>,<Height> ESC \
Application: ESC _ event=window ; size=<Width>,<Height> ESC \
Terminal: ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \
Note that the terminal window resizing always reflows the scrollback, so the window size, cursor position, scrolling regions, and selection coordinates are subject to change during step 3. Upon receiving the resize request (step 1), a fullscreen application can prepare a scrollback by cropping visible lines to avoid unwanted line wrapping or line extrusion, then send a resize confirmation (step 2). In case the aplication's output is anchored to the current cursor position or uses scrolling regions, the application should wait after step 2 for the updated values before continuing to output.
Hypothetical case with Far Manager (FM):
- FM saves visible original scrollback.
- FM draws its UI on top of the visible original scrollback.
- Terminal sends a resize request.
- FM receives request.
- FM restores the original scrollback.
- FM replies on resize request.
- Terminal receives reply.
- Terminal resizes the window and reflows the scrollback.
- Terminal sends modified window parameters.
- FM receives modified window parameters.
- FM saves the modified scrollback.
- FM draws its UI on top of the scrollback.
The window size tracking sequence is fired after every scrollback text selection changed. In the case of using the mouse for selection, a single left click is treated as a special case of selection when the start and end are the same (empty selection).
Note that selected text in the scrollback above the top index row will produce negative Y-coordinate values.
ESC _ event=system ; signal=<Signal> ESC \
Signal | Description |
---|---|
0 |
Window closing. |
1 |
System shutdown. |
The application must respond to the terminal within 5 seconds with the same message confirming that it will close itself without being forced. After a response to Signal=0, the application can continue running and closing the terminal window will be silently aborted. In the absence of confirmation, and also in the case of Signal=1, the application will be forced to close.
#include <iostream>
int main()
{
std::cout << "test" << '\n';
}
#!/bin/python
while True:
print('test\n')
while ($True)
{
"test\n";
}