ConPTY: transmit DECSET/DECRST state when client application enters/exits ENABLE_VIRTUAL_TERMINAL_INPUT mode

[DHowett]

Certain applications (mainly those using the Cygwin/MSYS runtime) will attempt to change VT input parameters while VT input is technically off. Because we do nothing to communicate the expected VT input parameters to a connected terminal before we turn on VT input passthrough, it's possible for the connected terminal to disagree with the conhost acting as its pty host.

When the application enters ENABLE_VIRTUAL_TERMINAL_INPUT mode, ConPTY should send a string of DECSET/DECRST sequences to bring the connected terminal in line with it to keep up the illusion of passthrough.

Notes from #6737:

From the Terminal's perspective, ENABLE_VIRTUAL_TERMINAL_INPUT is always on. It can't be lost/forgotten, because the terminal only generates VT. In general, it always has conhost sitting between it and the application to do translation from VT input. There are cases when the conhost sitting in the middle enters passthrough mode and dumps input straight out of terminal into the application.
Therefore, Terminal needs to be aware of DECCKM. Whether it receives DECCKM and therefore whether it generates the right cursor keys, however, is based on whether the application has requested ENABLE_VIRTUAL_TERMINAL_INPUT.
(There are applications that ask for DECCKM or other VT input format changes but are not in VT input mode, and sending them that data will freak them out.)

For WT 1.0, here's the matrix of behaviors.

Action	VT Output Mode	VT Input Mode	Notes
Application prints \e[?1h	❌	❌	VT output is off, request ignored
	❌	✅
	✅	❌	conhost (PTY) (only) enables DECCKM
	✅	✅	conhost (PTY) _and Terminal_ enable DECCKM

Action	VT Output Mode	VT Input Mode	PTY in DECCKM mode	Terminal in DECCKM mode	Notes
User presses arrow key	❌	❌	❌	❌	Terminal emits \e[A or \eOA conhost (PTY) translates it into KEY_EVENT_RECORD
	✅	❌	✅	❌	Terminal emits \e[A conhost (PTY) translates it into KEY_EVENT_RECORD
	✅	❌	✅	✅	Terminal emits \eOA conhost (PTY) translates it into KEY_EVENT_RECORD
	✅	✅	passthrough mode	❌	Terminal emits \e[A conhost (PTY) passes it through unmodified
	❌	✅		❌
	✅	✅		✅	Terminal emits \eOA conhost (PTY) passes it through unmodified
	❌	✅		✅

BEHAVIORS BEFORE TERMINAL 1.0

Action	VT Output Mode	VT Input Mode	Notes
Application prints \e[?1h	❌	❌	VT output is off, request ignored
	❌	✅
	✅	❌	conhost (PTY) (only) enables DECCKM
	✅	✅

Action	VT Output Mode	VT Input Mode	PTY in DECCKM mode	Notes
User presses arrow key	❌	❌	ignored	Terminal emits \e[A conhost (PTY) translates it into KEY_EVENT_RECORD
	✅	❌
	❌	✅	❌	Terminal emits \e[A conhost (PTY) translates it into \e[A
	✅	✅	❌
	❌	✅	✅	Terminal emits \e[A conhost (PTY) translates it into \e[OA
	✅	✅	✅

@DHowett

Okay, I'm literally seeing this behavior at the PTY:

• Input Mode = 0x9
• Input Mode = 0x209
• Input Mode = 0x9
• Print out DECCKM
• Input Mode = 0x209
• Input Mode = 0x9

This happens with printf, this happens with less's startup (though it prints more than just DECCKM)`, this happens pretty much every time I launch a process.

It's quite literally disabling VT input mode only while it is changing the input configuration. If we'd ever made the console more strict ("VT input mode changes only apply when VT input mode is on"), this would break horribly. I wonder why they'd do this?
@DHowett

Whether it receives DECCKM and therefore whether it generates the right cursor keys, however, is based on whether the application has requested ENABLE_VIRTUAL_TERMINAL_INPUT.

I totally did not expect that. If ENABLE_VIRTUAL_TERMINAL_INPUT is designed to affect which control sequences a pseudoconsole forwards from WriteConsole to the terminal, it is worth documenting.

This regressed with #6485

Why doesn't Windows Terminal Preview 1.1.1812.0 have the same problem, then? 6ff8ba7 was cherry-picked from 5e2c4c6, which is included in that version.
@KalleOlaviNiemitalo

[DHowett]

Oh, sorry! Windows Terminal Preview 1.1+, which should be released to the stable channel soon, doesn’t exhibit this issue. Note if you switch today that it won’t migrate your settings from the Release branch. It’s still pretty stable; we all use it every day to make sure we’re the most exposed to any weird issues it might have. 😄

[DHowett]

This is wild!

cygwin/msys2 sets and resets ENABLE_VIRTUAL_TERMINAL_INPUT not on every process launch, but for literally every character that's typed.

When we emit DECSET/DECRST for every state transition (to make sure the connected terminal does the right thing even outside of win32 input mode), each byte input into a msys2 application results in about 200 bytes of control sequences just to maintain terminal state.

(@KalleOlaviNiemitalo might find this interesting, even if just because they've been helping out with some VT_INPUT issues 😄)

[lazka]

(@tyan0 is working in this area in cygwin, mentioning here in case he wants to chime in)