Reported incompatibility with GNU Octave

[DHowett]

The GNU Octave is a cross platform project built to use the default OS terminal. Currently we would love to be able to identify if that is the new Windows Terminal. There is some incompatibility between conhost and the new Windows Terminal that is causing strange artifacts and program crashes for users of the new Windows Terminal that became very apparent when win11 made it the default. Octave is primarily developed on linux and crossbuilt for windows users, and we have few windows developers working that part of the program.

For now we wanted to create a script to try to identify if Octave was started in the Windows Terminal to prompt the user to switch, but so far have just settled on checking if the the registry key was set for having Windows Console Host be system default, and prompt the Windows users to switch the system default to that if it's not. Yes, it is a rather poor workaround. but it's effective. Until we have the developer resources to better troubleshoot the conhost / windows terminal incompatibility (or finish developing a separate, cross-platform compatible terminal widget to bundle with Octave), that change solves the problem for the users. But it does come with some false positives for the prompt because it gets set off when the windows setting is "let windows decide" even if that wouldn't use the new terminal, causing some other user confusion. So, it would at least be cleaner if at startup we could positively identify the Windows terminal.

Originally posted by @NRJank in #7434 (comment)

[DHowett]

This as well - thanks for the comment! I hate for us to be incompatible with anything unless we've decreed it so (and mostly that's just "we decree it so when we think 35-year-old Windows-specific apps have had it too good")

[NRJank]

hah, understood, and thanks again. had looked at creating an issue when we had a better idea of what to report was the issue, but useful troubleshooting has been limited to date. If i can find some useful snippets of that I'll link and/or summarize. I think at the moment that it's related to different terminal buffer behavior, which I have read is a known issue between old and new terminals.

[vefatica]

I have been using Octave as a WT profile for a few weeks and I have had no problems. Are particular incompatibilities noted anywhere?

[lhecker]

Out of the links above, this one seems to have the most information: https://octave.discourse.group/t/octave-crashing-with-weird-characters-and-not-loading/4951

[vefatica]

Thanks! What's shown there is bizarre. As I said I have had no problems. I have seen no evidence that Octave uses VT control sequences (but coming from UNIX, it might).

[NRJank]

apologies for letting this sit, seems i didn't catch the later comment notifications.

a small amount of debugging was done and described in this Octave developer comment/thread

Summary description:

The garbled output in the command window probably happens because escape sequence processing gets turned on even when we don’t want it and the scrolling problem happens because the new console doesn’t respect the buffer size that we want to set.

Whether it was reasonable for us to expect the previous behavior or we were just “lucky” that it worked for us, I don’t know (see comment #45 for some more about that).

[NRJank]

coming back to this again, didn't notice the mention in the other thread. I see this comment over there:

Yea... that thread doesn't really have anything valuable in it. https://octave.discourse.group/t/cli-is-open-together-with-gui-in-octave-7-1-on-windows-11/2753/14 has some additional info, but it sounds kinda like they just don't have the support to deal with this case.

Reading between the lines, I'm guessing they were using ShowWindow(GetConsoleWindow(), SW_HIDE) to hide a console they were spawning, instead of just requesting a hidden one to begin with.

I'd much rather help them sort that out on their side, than offer a bodgy opt-out on our side ☺️
-- zadjii-msft

Didn't want to lose that nugget above as we've periodically come back to this. the first suggestion of insufficient win gui devs to deal with WT changes to things that worked under conhost is correct. whether they were 'supposed to work' is another issue. as we're still considering bundling another terminal widget to sidestep the problem, i'd personally prefer the 'sorting it out on our side' approach

[NRJank]

this did trigger a bit more digging,

1. two issues - we do use ShowWindow(GetConsoleWindow(), SW_HIDE), and it does prevent the old windows console from appearing (removing it makes it reappear), but removing that and attempting other ways of suppressing the duplicate Windows Terminal were unsuccessful. Would be curious what 'just requesting a hidden one to begin with' is. several rounds of playing around with something like https://stackoverflow.com/a/780512, made no difference in the visibility of that terminal.

2. there is still the second, and more pressing issue for us which is that whether or not the window is suppressed, we lose ability to scroll back through the screen buffer, and attempting to do so causing either 'odd' behavior or garbled screen data, for example: https://octave.discourse.group/t/octave-crashing-with-weird-characters-and-not-loading/4951
   there appears to be some change to how we were accessing the screen buffer with conhost, and what works now, and that's not clear.