Control+Space not sent to program running in terminal

[jfedorkiw]

Windows Terminal version

1.17.11461.0

Windows build number

10.0.22621.2134

Other Software

NeoVim v0.9.1

Steps to reproduce

1. Inside NeoVim remap to Control + Space

Inside init.lua:

vim.keymap.set({'i', 'n', 'v', 'o'}, '', '', {noremap=true})

2. Open NeoVim and enter insert mode.
3. Enter Control + Space and notice it doesn't send the Escape command

NOTE: This Is also the case in VIM as well. It doesn't seem like the Control+Space character is making it to whatever is running in the terminal

Expected Behavior

I expect Control+Space character to make it to the program running in the terminal

Actual Behavior

When you hit Control+Space there is no reaction to the program running in the terminal. The command does not appear to be sent.

[jfedorkiw]

Note, I see this issue was brought up before (and apparently solved): #2865. Either this is a slightly different issue or perhaps it broke again.

[zadjii-msft]

Are you running a Windows version of neovim, or in WSL/cygwin/msys/git bash, or over ssh of some sort/? Those can all be complicating factors here.
Can you try repro'ing this with the debug tap to get a trace of all the input and output? Once the bug starts occurring, send us a screenshot and we might be able to figure out what the Terminal thinks it's getting here.

[237dmitry]

In pwsh (PSReadline module), for example, Ctrl-Space works. This combination opens command completion variants.

[zcobol]

@jfedorkiw inside nvim session type :map * to check if the keymapping is what you expect, or :map <c-space> for the ctrl+space mapping only, and post the output.

[sid-6581]

I'm having the same problem with several CTRL combinations. In testing neovim with the exact same configuration in WSL2, Powershell Core, and nvim-qt:

WSL2: CTRL-space works, CTRL-[ (remapped, not used as escape) doesn't work, CTRL-9 doesn't work (seems to just come through as 9)
Powershell Core: CTRL-space doesn't work, CTRL-[ acts like escape even though it's remapped, CTRL-9 just outputs 9
nvim-qt: All work as expected

I'm on the latest WT preview and Windows 11 insider beta.

[zadjii-msft]

@sid-6581 same question to you: Can you try repro'ing this with the debug tap to get a trace of all the input and output?

[jfedorkiw]

Are you running a Windows version of neovim, or in WSL/cygwin/msys/git bash, or over ssh of some sort/?

I'm just running nvim through PowerShell inside the terminal app. The problem is also the same when using "Command Prompt" inside terminal.

Note that it if I use nvim-qt, which launches NeoVim in it's own window, Ctrl-Space works as expected (sending the escape character). Also - the same problem exists in Vim.

Can you try repro'ing this with the debug tap to get a trace of all the input and output?

In this case I opened up the terminal and hit Control+Space to see what was being sent:

Here is the text copy & pasted for your debugging pleasure:

␛[?9001h␛[?1004h␛[1t␛[?25l␛[2J␛[m␛[HMicrosoft␣Windows␣[Version␣10.0.22621.2134]␍␊
(c)␣Microsoft␣Corporation.␣All␣rights␣reserved.␛[4;1HC:\Users\jfedo>␛]0;Command␣Prompt␇␛[?25h␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[I␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;0;33;1_␛[18;56;0;0;32;1_␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[17;29;0;1;40;1_␛[17;29;0;1;40;1_␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣␛[17;29;0;0;32;1_␛[O

As a Breakdown here is what I see happen as I execute the key combination:

Press Control:

␛[17;29;0;1;40;1_

Press Space:

␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣

And then when I release Control you get the trailing

␛[17;29;0;0;32;1_

On the left side of the screen, when I hit Control+Space it results in a Space character showing up (as if I had just pressed space)

Interestingly, if I hit Control+Space on the RIGHT side of the window, where it outputs the debug text, it sends the NUL (␀) character (which I believe is what Control+Space is supposed to send).

@zcobol
Here is the output of :map .

v  <C-Space>    @<Plug>VimwikiToggleListItem
n  <C-Space>    @<Plug>VimwikiToggleListItem
o  <C-Space>   * <Esc>
v  <C-Space>   * <Esc>
n  <C-Space>   * <Esc>
Press ENTER or type command to continue

Note the mapped vim-wiki functionality doesn't work. Also, as mentioned above, these key combinations do work as expected when running nvim-qt.

Thank-you all for your support in this matter,

John

[zadjii-msft]

Vk	Sc	Uc	Kd	Cs
VK_CONTROL	29	0	down	40
VK_CONTROL	29	0	down	40
VK_SPACE	57	' '	down	40
VK_SPACE	57	' '	up	40
VK_CONTROL	29	0	up	32

that all looks right

[zadjii-msft]

This might be neovim/neovim#19575

[jfedorkiw]

I believe the problem is that when you hit Control+Space nvim is expecting a different character.

As a similar example, in the terminal when you hit Control+F you get a different set of characters:

Control+ F: 
    ␛[17;29;0;1;40;1_␛[70;33;6;1;40;1_␛[70;33;6;0;40;1_^F␛[17;29;0;0;32;1_
f:
    ␛[70;33;102;1;32;1_␛[70;33;102;0;32;1_f


␛[70;33;6;1;40;1_
vs
␛[70;33;102;1;32;1_

I believe it expects the "Nul" character, but I'm not sure 100% sure about this. Im not sure i there is an easy way to verify this (is it possible to emulate sending a specific character to the terminal ?).

As an interesting side note (and as noted above) when you type Control+Space in the right side of the split debug terminal you actual do get the Nul character (␀). For some reason that side of the terminal does something different when you hit space while having control pressed.

[elsaco]

@jfedorkiw <C-Space> mapping behaves differently depending on the terminal used. This is what I've noticed:

init.lua used for testing:

vim.keymap.set('i', 'A', 'foo', {noremap=true})
vim.keymap.set('i', '<C-Space>', 'bar', {noremap=true})

Only insert mode was mapped since the bug was filed against <C-Space> not working in this mode.

For each A the string foo is substituted and for each ctrl-space the string bar.

When using the wt.exe only A works. There's no action when ctrl-space is used:

The mapping is correct and it does come from init.lua

When using Alacritty a space is inserted when pressing ctrl-space:

The mapping seems to work okay when using the GUI version of neovim:

[zadjii-msft]

Here's a thought - what keyboard layout are you using?

[jfedorkiw]

I'm using a QWERTY Keyboard.

It's not clear to me there is agreement on what the core of the bug is. It might be useful to clarify.

I'm under the impression the core problem is that the terminal isn't passing along a special key-character when the Control+Space combination is pressed. The confusion then comes from the fact that presumably this is working on some folks setups but not mine.

It could also be the case that you don't agree the terminal should be converting Control+Space into a special key press (which seems reasonable as this clearly doesn't seem to be well defined behaviour). In that case the "fix" may be more of a feature request to allow users to bind custom key-combinations to specific key codes.

John

[zadjii-msft]

There's definitely no disagreement there. The trick here is trying to find what layer in particular is losing the keystroke.

What we established up in #15939 (comment) is

• The Terminal app is getting the ctrl+space
• The Terminal is sending sequences corresponding to a ctrl+space to the connection (this is the ␛[17;29;0;1;40;1_␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣␛[17;29;0;0;32;1_ goop)
• (not necessarily established) the console converts that goop to internal KEY_EVENT_RECORDs, then into a NUL for VT input mode
• (on my machine), a showkey -a in WSL does show NUL bytes coming through.

Now, part of the problem here is probably just that I don't totally get nvim, sorry about that. I've put the following in .config/nvim/init.lua, but I honestly don't know if that's working or not. Doesn't seem like it is?

" vim.keymap.set({'i', 'n', 'v', 'o'}, '', '', {noremap=true})
vim.keymap.set('i', 'A', 'foo', {noremap=true})
vim.keymap.set('i', '<C-Space>', 'bar', {noremap=true})

Though, when I do a C-h,k,C-Space in emacs, I do see it report the help for C-@ (as I'd expect).

Maybe another part of the problem that could be complicating this is using WSL nvim, versus a Windows build of nvim. If it's the later, that's a good hypothesis as to the earlier comment about ctrl+space not working in PowerShell.

So I'd love to find out is where in the stack the key sequence is getting lost. Is the key making it all the way through conhost? Is conhost sending something to nvim that they're throwing away?

[j4james]

So I'd love to find out is where in the stack the key sequence is getting lost. Is the key making it all the way through conhost?

@zadjii-msft I believe this is a conhost issue. I can reproduce the problem with a simple app that reads characters with ReadConsoleA while ENABLE_VIRTUAL_TERMINAL_INPUT is set.

When I press Ctrl+Space I can see it generates the NUL character here:

terminal/src/terminal/input/terminalInput.cpp

Lines 623 to 629 in cc2ba53

	// Currently, when we're called with Ctrl+@, ch will be 0, since Ctrl+@ equals a null byte.
	// VkKeyScanW(0) in turn returns the vkey for the null character (ASCII @).
	// -> Use the vkey to alternatively determine if Ctrl+@ is being pressed.
	if (ch == UNICODE_SPACE || (ch == UNICODE_NULL && vkey == LOBYTE(OneCoreSafeVkKeyScanW(0))))
	{
	return _makeCharOutput(0);
	}

But that NUL gets converted back into a key event in the InputBuffer class here:

terminal/src/host/inputBuffer.cpp

Lines 817 to 820 in cc2ba53

	for (const auto& wch : text)
	{
	_storage.push_back(SynthesizeKeyEvent(true, 1, 0, 0, wch, 0));
	}

That key event then gets process by the GetChar function here:

terminal/src/host/stream.cpp

Lines 234 to 244 in cc2ba53

	const auto zeroKey = OneCoreSafeVkKeyScanW(0);
	if (LOBYTE(zeroKey) == Event.Event.KeyEvent.wVirtualKeyCode &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, ALT_PRESSED) == WI_IsFlagSet(zeroKey, 0x400) &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, CTRL_PRESSED) == WI_IsFlagSet(zeroKey, 0x200) &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, SHIFT_PRESSED) == WI_IsFlagSet(zeroKey, 0x100))
	{
	// This really is the character 0x0000
	*pwchOut = Event.Event.KeyEvent.uChar.UnicodeChar;
	return STATUS_SUCCESS;
	}

But the zeroKey it's trying to match here is Ctrl+Shift+OEM_KEY_3 (at least on my machine), while the key event that was synthesized has a zero virtual key code and no modifiers. As a result, it just gets dropped.

Two possible solutions:

1. In the InputBuffer method where we synthesize the key event, check for a null character as a special case and convert that to Ctrl+Shift+OEM_KEY_3 (or whatever OneCoreSafeVkKeyScanW(0) translates to).

2. In the GetChar function, where we're looking for a match with OneCoreSafeVkKeyScanW(0), also consider a zero virtual key with no modifiers to be a valid match.

Frankly both of those options seem a bit messy, so I'm hoping someone that understands the code base might have a better suggestion, but at least that's a starting point.

Btw, I'm not absolutely certain this is the same issue that the OP is seeing in vim, but it seems likely.

[lhecker]

Ah I see, that makes sense, since I made the change to the /src/host/stream.cpp code very recently. I made it because:

In addition, this commit fixes at least one bug: The previous approach to detect the null key via DoActiveModifierKeysMatch didn't work. As it compared the modifier keys as a bitset with == it failed to match whenever the numpad key was set, which it usually is.

As the culprit, I'll try to fix this issue then. I think the 2nd suggestion is better, because there's no 0 value for the virtual key on any keyboard (the minimum is 1 with VK_LBUTTON) and so I think the detection is safe.

It might be interesting how it used to work in conhost v1:

#define EITHER_CTRL_PRESSED (LEFT_CTRL_PRESSED | RIGHT_CTRL_PRESSED)
#define EITHER_ALT_PRESSED (LEFT_ALT_PRESSED | RIGHT_ALT_PRESSED)
#define MOD_PRESSED (SHIFT_PRESSED | EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED)

DWORD ConsKbdState[] = {
    0,
    SHIFT_PRESSED,
    EITHER_CTRL_PRESSED,
    SHIFT_PRESSED | EITHER_CTRL_PRESSED,
    EITHER_ALT_PRESSED,
    SHIFT_PRESSED | EITHER_ALT_PRESSED,
    EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED,
    SHIFT_PRESSED | EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED
};

#define KEYEVENTSTATE_EQUAL_WINMODS(Event, WinMods)\
    ((Event.Event.KeyEvent.dwControlKeyState & ConsKbdState[WinMods]) && \
    !(Event.Event.KeyEvent.dwControlKeyState & MOD_PRESSED & ~ConsKbdState[WinMods]))

NTSTATUS GetChar(...)
{
    // ...
    SHORT sTmp = VkKeyScanW(0);
    // ...
    if ((LOBYTE(sTmp) == Event.Event.KeyEvent.wVirtualKeyCode) && KEYEVENTSTATE_EQUAL_WINMODS(Event, HIBYTE(sTmp)))
    {
        // This really is the character 0x0000
        *pwchOut = Event.Event.KeyEvent.uChar.UnicodeChar;
        return STATUS_SUCCESS;
    }
    // ...
}

As far as I can tell the intention of the original code is identical to what I now wrote. But the original code was flawed as far as I can tell, because the KEYEVENTSTATE_EQUAL_WINMODS macro did a "'has one of the modifiers pressed' and 'has none of the other modifiers pressed'" check. I believe that's wrong because you'd want to test if all the individual modifiers are matched exactly no? At a minimum I would've expected the former check to be 'has all of the modifiers pressed', but it doesn't do that.

Was the original v1 behavior correct?

[j4james]

Ah I see, that makes sense, since I made the change to the /src/host/stream.cpp code very recently.

@lhecker FWIW, I don't think this was broken by any of your recent changes. I've tested in my Window 10 inbox conhost (which I think is several years old), and it was already not working in that version.

[brookst]

Just to note that I rechecked this behaviour and it seems to have started working for me. As noted on the older issue for this, I saw no keycode for ctrl+space. Now both ctrl+space and ctrl+single quote generate a null which maps to ^@.

The only thing that seems to have changed since I last tested is the Windows version number: 10.0.22621.1778 didn't work, 10.0.22621.2283 does.

[lonnywong]

@zadjii-msft I believe this is a conhost issue. I can reproduce the problem with a simple app that reads characters with ReadConsoleA while ENABLE_VIRTUAL_TERMINAL_INPUT is set.

When I press Ctrl+Space I can see it generates the NUL character here:

terminal/src/terminal/input/terminalInput.cpp

Lines 623 to 629 in cc2ba53

	// Currently, when we're called with Ctrl+@, ch will be 0, since Ctrl+@ equals a null byte.
	// VkKeyScanW(0) in turn returns the vkey for the null character (ASCII @).
	// -> Use the vkey to alternatively determine if Ctrl+@ is being pressed.
	if (ch == UNICODE_SPACE || (ch == UNICODE_NULL && vkey == LOBYTE(OneCoreSafeVkKeyScanW(0))))
	{
	return _makeCharOutput(0);
	}

@j4james I SetConsoleMode to ENABLE_VIRTUAL_TERMINAL_INPUT, it still goes to _makeWin32Output:

terminal/src/terminal/input/terminalInput.cpp

Lines 488 to 494 in cc2ba53

	// GH#4999 - If we're in win32-input mode, skip straight to doing that.
	// Since this mode handles all types of key events, do nothing else.
	// Only do this if win32-input-mode support isn't manually disabled.
	if (_inputMode.test(Mode::Win32) && !_forceDisableWin32InputMode)
	{
	return _makeWin32Output(keyEvent);
	}

Press ctrl + space gets �[17;29;0;1;40;1_�[32;57;32;1;40;1_�[32;57;32;0;40;1_�[17;29;0;0;32;1_ from

terminal/src/terminal/input/terminalInput.cpp

Line 740 in cc2ba53

return fmt::format(FMT_COMPILE(L"\x1b[{};{};{};{};{};{}_"), vk, sc, uc, kd, cs, rc);

[j4james]

@lonnywong If you're testing in Windows Terminal rather than conhost/openconsole then you're dealing with conpty, which adds another layer of redirection. Windows Terminal forwards the individual keyup/keydown events over conpty as win32-input sequences. Those sequences are later received by conhost and converted back into keyboard events. And those keyboard events are then processed by the conhost version of the TerminalInput class as described above.

[lonnywong]

@j4james Thanks. Pressing keys in OpenConsole.exe, I got a break in GetChar:

terminal/src/host/stream.cpp

Lines 234 to 244 in cc2ba53

	const auto zeroKey = OneCoreSafeVkKeyScanW(0);
	if (LOBYTE(zeroKey) == Event.Event.KeyEvent.wVirtualKeyCode &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, ALT_PRESSED) == WI_IsFlagSet(zeroKey, 0x400) &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, CTRL_PRESSED) == WI_IsFlagSet(zeroKey, 0x200) &&
	WI_IsAnyFlagSet(Event.Event.KeyEvent.dwControlKeyState, SHIFT_PRESSED) == WI_IsFlagSet(zeroKey, 0x100))
	{
	// This really is the character 0x0000
	*pwchOut = Event.Event.KeyEvent.uChar.UnicodeChar;
	return STATUS_SUCCESS;
	}

How do you get a break in InputBuffer::_HandleTerminalInputCallback?
I SetConsoleMode to ENABLE_VIRTUAL_TERMINAL_INPUT, and got a break in InputBuffer::_HandleTerminalInputCallback:

terminal/src/host/inputBuffer.cpp

Lines 817 to 820 in cc2ba53

	for (const auto& wch : text)
	{
	_storage.push_back(SynthesizeKeyEvent(true, 1, 0, 0, wch, 0));
	}

[lonnywong]

@zadjii-msft @lhecker @j4james Can you help to review #16298 ? Thanks.
It works for me. Probably not a great solution, but it should be harmless.