A minor ConPTY refactoring: Goodbye VtEngine Edition

[lhecker]

The idea is that we can translate Console API calls directly to VT at
least as well as the current VtEngine setup can. For instance, a call
to SetConsoleCursorPosition clearly translates directly to a CUP
escape sequence. Effectively, instead of translating output
asynchronously in the renderer thread, we'll do it synchronously
right during the Console API call.

Most importantly, the this means that any VT output that an
application generates will now be given to the terminal unmodified.

Aside from reducing our project's complexity quite a bit and opening
the path towards various interesting work like sixels, Device Control
Strings, buffer snapshotting, synchronized updates, and more, it also
improves performance for mixed text output like enwik8.txt in conhost
to 1.3-2x and in Windows Terminal via ConPTY to roughly 20x.

This adds support for overlapped IO, because now that output cannot
be "skipped" anymore (VtEngine worked like a renderer after all)
it's become crucial to block conhost as little as possible.

⚠️ Intentionally unresolved changes/quirks:

• To force a delayed EOL wrap to wrap, WriteCharsLegacy emits a
  \r\n if necessary. This breaks text reflow on window resize.
  We cannot emit \r the way readline does it, because this would
  overwrite the first column in the next row with a whitespace.
  The alternative is to read back the affected cell from the buffer
  and emit that character and its attributes followed by a \r.
  I chose to not do that, because buffer read-back is lossy (= UCS2).
  Unless the window is resized, the difference is unnoticeable
  and historically, conhost had no support for buffer reflow anyway.
• If ENABLE_VIRTUAL_TERMINAL_PROCESSING is set while
  DISABLE_NEWLINE_AUTO_RETURN is reset, we'll blindly replace all
  LF with CRLF. This may hypothetically break DCS sequences, but it's
  the only way to do this without parsing the given VT string and
  thus the only way we can achieve passthrough mode in the future.
• ENABLE_WRAP_AT_EOL_OUTPUT is translated to DECAWM.
  Between Windows XP and Windows 11 21H2, ENABLE_WRAP_AT_EOL_OUTPUT
  being reset would cause the cursor position to reset to wherever
  a write started, if the write, including expanded control chars,
  was less than 100 characters long. If it was longer than that,
  the cursor position would end up in an effectively random position.
  After lengthy research I believe that this is a bug introduced in
  Windows XP and that the original intention was for this mode to be
  equivalent to DECAWM. This is compounded by MSDN's description
  (emphasis mine):

  If this mode is disabled, the last character in the row is
  overwritten with any subsequent characters.

⚠️ Unresolved issues/quirks:

• Focus/Unfocus events are injected into the output stream without
  checking whether the VT output is currently in a ground state.
  This may break whatever VT sequence is currently ongoing.
  This is an existing issue.
• VtIo::Writer::WriteInfos should properly verify the width of
  each individual character.
• Using SetConsoleActiveScreenBuffer destroys surrogate pairs
  and extended (VT) attributes. It could be translated to VT pages
  in the long term.
• Similarly, ScrollConsoleScreenBuffer results in the same and
  could be translated to DECCRA and DECFRA in the near term.
  This is important because otherwise vim output may loose
  its extended attributes during scrolling.
• Reflowing a long line until it wraps results in the cooked read
  prompt to be misaligned vertically.
• SCREEN_INFORMATION::s_RemoveScreenBuffer should trigger a
  buffer switch similar to SetConsoleActiveScreenBuffer.
• Translation of COMMON_LVB_GRID_HORIZONTAL to SGR 53 was dropped
  and may be reintroduced alongside UNDERSCORE = SGR 4.
• Move the OSC 0 ; P t BEL sequence to WriteWindowTitle
  and swap the BEL with the ST (ESC \).
• PowerShell on Windows 10 ships with PSReadLine 2.0.0-beta2
  which emits SGR 37/40 instead of 39/49. This results in black
  spaces when typing and there's no good way to fix that.
• A test is missing that ensures that FillConsoleOutputCharacterW
  results in a CSI n J during the PowerShell shim.
• A test is missing that ensures that PtySignal::ClearBuffer
  does not result in any VT being generated.

Closes #262
Closes #1173
Closes #3016
Closes #4129
Closes #5228
Closes #8698
Closes #12336
Closes #15014
Closes #15888
Closes #16461
Closes #16911
Closes #17151
Closes #17313

[lhecker]

Most of the files in this PR will either be deletions or minor modifications. I recommend just ticking them off as "viewed" first. Afterwards it'll still be a large PR, but it'll be more "manageable", I think.

[DHowett]

notes inline

[j4james]

Two big issues I've noticed so far (just testing in WSL on this branch by itself).

• printf "\e[1;1;1,~" will kill the app.
• printf "\e[6n"; read will generate two CPR reports (same thing happens with any query sequence).

The underlying problem in both cases is that we're letting the sequences be processed by conhost in addition to passing them through, and I think maybe in some cases we shouldn't be doing that, i.e. we need to retain some IsConsolePty tests in AdaptDispatch to suppress certain operations or at least handle them differently.

Edit: Actually it's probably easier to put these checks in ConhostInternalGetSet rather than AdaptDispatch, at least for these two cases.

[lhecker]

(FYI the old code is crashing VS 17.10 preview 2, so I changed this a bit.)

[lhecker]

I went through the PR and ticked off all files that are either wholly deleted or trivialities and... look y'all........ It's only 40 modified files:

No biggie. 👍 /s

[lhecker]

(The compiler was complaining otherwise.)

[j4james]

It seems like query responses aren't working at all now. With this test case: printf "\e[6n"; read I see nothing until I press a key.

It looks like that broke in commit 78ae6dd. Prior to that I got the double response immediately. After that I only got the one response immediately, and had to press a key to see the second response. Then once the double-response was fixed, I got nothing until I pressed a key.

[vladimirshefer]

@lhecker hi! do you know when this will be released?

[lhecker]

It is available in Windows Terminal Preview 1.22 which can be found on our release page (among others). Is that what you meant or are you asking when it will be available in the stable version?

[vladimirshefer]

Yes, stable release.
Btw, Thank for this link.

[lhecker]

The stable version always trails the preview version by 1 release. As such, we'll release 1.22 as stable in about 2-3 months.