"Chunked" soft fonts are no longer working correctly

[j4james]

Windows Terminal version

1.19.2682.0

Windows build number

10.0.19045.3448

Other Software

No response

Steps to reproduce

1. Download the VT220 soft font from this gist.
2. Open a cmd shell in Windows Terminal.
3. TYPE the soft font file downloaded in step 1.

Expected Behavior

The next prompt that is output should be rendered with that low-res VT220 font, looking something like this:

Actual Behavior

It appears that some of the characters are only partially downloaded, and some haven't been downloaded at all, so I'm seeing this:

If you look at the contents of the font file, you'll see that it's defined as a series of DECDLD sequences, so it's downloading 24 sets of 4 characters, rather than a single 94 character set. I think that's part of the reason for the failure, because fonts that aren't chunked like that don't seem to have the problem.

I also noticed that it only happens in a cmd shell, but not in PowerShell or a WSL bash shell. My guess is that the other shells are buffering their output in a way that hides the problem. It's also possible there is a timing factor, but the cmd shell failure is easily reproducible for me.

And as far as I can tell, the problem was introduced in PR #15991. The commit prior to 41f7ed7 was working as expected.

[j4james]

I think I know what the problem is. The new "cork" algorithm causes the VT renderer to flush when the end of the output buffer is reached. This assumes that the VT output is in a ground state, but that's not the case when we're processing a DCS sequence and are in the middle of passing it through to the conpty client. So what's happening is we're flushing a mode ?25 cursor update in the middle of the DCS stream, and that ends the stream early (you can see this in the debug tap).

In theory this could potentially affect all soft fonts, but I think it's more easily triggered by the "chunked" examples I have because they have a line break after each chunk, and the cmd shell's TYPE command probably flushes its output after every line break. Fonts that are written out in a single block will likely only be affected if they're over a certain size. DECCTR palette updates might be affected too, because they also rely on DCS passthrough.

[lhecker]

I can't figure out how this worked before... VtEngine is still bound into the ~60 "FPS" rendering loop, so why didn't this still randomly happen previously?

[j4james]

I assumed there was a lock of some sort that prevented the VtEngine render thread from doing anything while the state machine was busy processing output. If that's not the case then maybe it was a matter of luck that it worked before, and PR #15991 just made the issue more noticeable. But I wasn't able to reproduce the issue prior to that commit, and it happens all the time now.

[lhecker]

This assumes that the VT output is in a ground state, but that's not the case when we're processing a DCS sequence and are in the middle of passing it through to the conpty client.

Edit: I see now (and remember). The problem is not that we flush with the state machine is not in ground state, but rather that VtEngine inserts random stuff into ongoing sequences.

My hope was I can remedy #16769 by doing this, but that doesn't work because it's in ground state during newlines:

void VtIo::CorkRenderer(SCREEN_INFORMATION& screenInfo, bool corked) const noexcept
{
    if (!corked && screenInfo.GetStateMachine().IsGroundState())
    {
        LOG_IF_FAILED(ServiceLocator::LocateGlobals().pRender->PaintFrame());
    }
    _pVtRenderEngine->Cork(corked);
}

Edit 2: It's not that type does line-wise writes, no, it writes lines in chunks of 80 characters. cmd folks were once again clearly ahead of the the game.