WT should set COLORTERM

[heaths]

Description of the new feature/enhancement

Related to #1040, WT should set the COLORTERM environment variable to "truecolor" since WT does support truecolor. I can understand how changing TERM to "xterm-truecolor" (from "xterm-256color") could be a breaking change, but that's what COLORTERM is for.

Some tools like with cli/cli#4079 use this to determine if 24-bit colors should be emitted. Easy enough for users to define - either as Windows environment variables or update their shell profile e.g., .bashrc, but users who don't know what won't enjoy a wash of color by default like with gh issue list starting with v2.0.0.

Proposed technical implementation details (optional)

WT should ideally define the environment variable COLORTERM=truecolor within its own process environment block by default.

/cc @mislav

[WSLUser]

Related: #8303. Might get duped into that one. Also related but was closed as something they're not willing to do as infocmp doesn't exist nor planned to exist: #8336

[zadjii-msft]

Discussed this with Dustin last week - I think we're both cool with this now. I think we were both more reluctant to do it in the past because it's just not a great solution, but it works for most people. So we may as well, same with TERM_PROGRAM and TERM_PROGRAM_VERSION. If people really want to take dependencies on these three, then why not?

[heaths]

@DHowett, @mislav for truecolor-detection support, is this really the right detection then? There's mention of many different TERM values that wouldn't match these checks. Seems TERM has become grossly overloaded and it's hard to glean anything consistent from it.

From a quick search for COLORTERM, it seems its usage is just as convoluted as TERM. Perhaps the only reliable way is to do something like this, which I've seen mentioned elsewhere.

[zadjii-msft]

Perhaps the only reliable way is to do something like this, which I've seen mentioned elsewhere.

Woah, I've never seen that particular sequence (\eP$qm\e) used before. I don't think we support it, so that definitely won't work right on Windows (conhost or Terminal). We also don't support the colon syntax for RGB colors currently, #4321. (aside: when did that version get so popular? I swear I only ever saw the semicolon version years ago when I first implemented RGB support).

You might be able to get away with just adding a if runtime.GOOS == "windows" if you can be relatively certain that the code is running on Windows 10. RGB support was added in RS2, so there's only 3 releases of Windows 10 that didn't support it (TH1, TH2, RS1), and even then VT support was only added in TH2 in the first place. Granted, I'm no golang user myself so if you need to be more precise than that, I won't be any help :P

[WSLUser]

https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences exists and notcurses is using it for the new experimental support of Windows. Notcurses in facts manipulates the terminfo entry to grab what it needs for the various terminals. There's no denying that terminfo isn't the greatest solution but no consensus appears to have been reached for implementing a standardized query method for terminals to use so any terminal that does have a query detection feature implements it in different ways, which does in the end rely on terminfo at some point.

(This all being said, WT is certainly able to create it's own query method that reflects that doc, but since there are other terminals on Windows to account for, the env vars would still be needed for them)

[heaths]

With that in mind, it really seems like there's no good solution since the CLI supports pre-Windows 10 as do many apps out there.

So would just trying to emit an RGB VT sequence be best? If a terminal doesn't support it, it should just ignore the sequence and emit text with the default background and foreground colors, right? So there's really no point in trying to detect if truecolor is supported?

/cc @mislav

[mislav]

If a terminal doesn't support it, it should just ignore the sequence and emit text with the default background and foreground colors, right?

It should but doesn't always. Terminal.app does not support RGB colors and yet outputs strange formatting when you try to use RGB sequences. As long as the default terminal emulator on one of major operating systems can't get this right, authors of CLI apps such as myself will rather want to operate on detection rather than blind faith. Plus, we might want to output 256-color to clients that do not support truecolor, rather than outputting truecolor and have it be rendered with default foreground+background.

for truecolor-detection support, is this really the right detection then?

I don't know! That's the best (read: easiest, most straightforward) that I saw various CLI tools come up with.

[heaths]

As long as the default terminal emulator on one of major operating systems can't get this right, authors of CLI apps such as myself will rather want to operate on detection rather than blind faith.

Totally makes sense and I agree. It's just strange that, at least when it comes to colors/VT support, writing console apps seems to be harder/more involved than writing windowed apps. 🙄

Still, the crux of this bug: if COLORTERM is meant to detect color support - though what the value is set to doesn't seem well-defined - perhaps the more oft-supported TERM=xterm-256color is appropriate here for WT while COLORTERM=xterm-truecolor is appropriate in addition. TERM seems to be more general-purpose from various reads, while COLORTERM seems to be more specific to color support, as the name also implies.

[j4james]

Woah, I've never seen that particular sequence (\eP$qm\e) used before.

I'm sure I have mentioned this before somewhere, but I'm convinced that just supporting DECRQSS and DECRQM would solve 99% of feature detection issues. Admittedly it's not always that straightforward to use, and I know a lot of people don't like terminal queries in general, but they're at least an existing standard that has a reasonable level of support amongst other terminals.

I actually have a partial DECRQSS implementation in a stash somewhere which might be worth resurrecting. I think I put it on hold because we didn't have the APIs in place to query all the settings that were needed, but maybe a partial implementation is still better than nothing.

That said, I am also in favor of a COLORTERM env, just because it's quite widely used, and I don't see any harm in it. I'm not so sure about TERM_PROGRAM, though, because that encourages apps to do the same stupid things that web developers did with user-agent strings, like writing code that refuses to run if the detected terminal isn't one that they recognise (this is not just hypothetical - I've seen it happening already).

[DHowett]

I'm not so sure about TERM_PROGRAM, though, because that encourages apps to do the same stupid things that web developers did with user-agent strings, like writing code that refuses to run if the detected terminal isn't one that they recognise (this is not just hypothetical - I've seen it happening already).

I felt like this for a while, but then I realized that we're fighting a losing battle against people who are looking for WT_SESSION or checking the process tree for WindowsTerminal.exe and lighting up features that would otherwise work fine in conhost only when they find it. For all the issues TERM_PROGRAM has, I suspect that it is only as bad as what folks are already doing. I don't know.

[heaths]

TERM_PROGRAM would create the mess browsers had (have, really) for decades with UAs. Feature-level detection would be ideal, and without queuing up xkcd's strip on standards, TERM=xterm-256color seems pretty standard across most terminals while COLORTERM seems to be akin to a "v2" for TERM. The bigger question is what it should be set to. In some discussions, it seems having it set to anything at all should indicate at least 256 color support if not truecolor support.

If you'd set TERM_PROGRAM=WindowsTerminal or something like that, as a dev I'd still be tempted to use WT_SESSION because it's been around longer and supports more customers.

[Diablo-D3]

Side note for those who are following this ticket: https://github.com//termstandard/colors is the best documentation out there for what supports COLORTERM. Windows Terminal is already on this list, even though it isn't emitting the variable yet.

[PennRobotics]

oh-my-zsh changed their update script to use truecolor if supported. I had replaced the omz ASCII art and color definitions, so having to merge this linked commit made me realize $TERM is xterm-256color and $COLORTERM is undefined in Windows Terminal.

Until this issue changes (subscribe for issue updates) the update script will detect full color and choose the right palette after adding export COLORTERM=truecolor near the top of .zshrc

The new oh-my-zsh commit uses the XVilke gist that @heaths linked—although not the query method—so any other script using the variable check from this gist should also get support after defining COLORTERM. I imagine one could simply add this in .bashrc as well (or whichever dotfile is shell-relevant).

[j4james]

Just FYI, that COLORTERM test in oh-my-zsh is essentially useless, because any terminal that doesn't support the RGB color sequences is also not likely to support the fallback 256-color sequences. They'll potentially just end up with a bunch of blinking text as a result of the 5 in the SGR. It's possible they just don't care about supporting older terminals, but then they might as well use the RGB colors for everyone and not bother with the COLORTERM test.

[jredfox]

ok after looking $COLORTERM is to be the only color. if it's the same name as $TERM then that's a bug on the terminal. tested on linux terminals some do contain the bug others do not but it wouldn't make sense why $COLORTERM would be equal to $TERM

[Diablo-D3]

COLORTERM only has one meaningful value: truecolor. If a terminal isn't true color, the variable should remain unset.

[jredfox]

No I seen lots of Linux terminals that return xterm-256 from that variable. If it’s unset that means it’s an older terminal then you should also look for $TERM but $TERM a lot of times gets set to an older color value such as xterm not 256 and xterm-256 even if it supports true color that value will remain non true color regardless. Example easily produced. `gnome-terminal` latest linux mint distro. print `$TERM` and `$COLORTERM` it will give you xterm-256color or was it xterm-256, fallowed by truecolor

[Diablo-D3]

TERM should return xterm-256color if it is a modern term that chooses to implement xterm as-is; gnome-terminal uses libvte, and libvte-based terminals, as far as I know, do not implement COLORTERM incorrectly. What you are describing sounds like a bug with gnome-terminal if you are accurately describing it.

Software out there that implements truecolor either check if COLORTERM contains truecolor and/or 24bit, or in some cases, is simply set at all. I have not seen one that checks for equals exactly 'truecolor', which allows what I think you described to work (COLORTERM=xterm-truecolor).

[eggbean]

Is there a way to make Windows Terminal set $COLORTERM in the settings file? If not, how can I make a bash conditional statement to determine if the terminal is Windows Terminal so I can set it in ~/.bash_profile?

[heaths]

Is there a way to make Windows Terminal set $COLORTERM in the settings file? If not, how can I make a bash conditional statement to determine if the terminal is Windows Terminal so I can set it in ~/.bash_profile?

@eggbean you can check for $WT_SESSION - if it's defined at all - and conditionally set $COLORTERM. But, yeah, I agree: it'd be nice if WT just did this automatically.

[eggbean]

@heaths Thanks.

[jredfox]

Nah tested on 40 Linux terminals colorterm not being set doesn't mean it's going to be true color

[mikehearn]

Unfortunately $WT_SESSION turns out to not be reliable (sigh). Here's a funny trick:

• Open Explorer
• Right click on a folder whilst holding the shift key
• Select "Show more options"
• click "open powershell window here"
• Observe that although what you get is Windows Terminal, WT_SESSION does not exist in the environment.

Why not? I have no idea and don't wish to spend time finding out.

Please please please just implement TERM_PROGRAM and other user agent strings. Yes, feature detection, I know. It's a lovely idea but it will never be enough by itself. In our case we're trying to detect WT specifically because there are bugs in the old legacy terminal, maybe in old versions of Windows I don't know, for example setting the console to UTF-8 will change the font in every single legacy console window. That annoys people who for whatever reason prefer to use that vs the new WT (they exist, I've worked with them) and so we just say oh whatever, if you use old stuff you get old stuff. That's why we want to detect WT, it's more like a negative detection and thus feature detection won't ever work because old conhost will gamely advertise that it has these features and then do the wrong thing.

Yep, it does create other problems. Non-WT terminals that use the same codepaths, for example. Fine, a problem for another day. They can advertise themselves as "Windows Terminal/5 (compatible; FooTerm)" and that's OK.

[zadjii-msft]

@mikehearn what you're seeing there is #13006 and is a Hard problem - in that scenario, powershell.exe was launched outside of the Terminal, then later attached to it. WT was not involved in launching it at all, so it had no opportunity to add any environment variables. Not even conhost would have a chance to say "I'm conhost, NOT Terminal" here!

[mikehearn]

Huh, OK, that's an interesting problem indeed. I guess ANSI escapes to do feature detection and/or getting a user-agent equivalent would be the only way forward there.

[heaths]

Much of this issue has talked about corner cases where setting TERM=xterm-256 and COLORTERM=truecolor may not be 100% accurate, but as has come up many times, it seems no terminal is perfect. Yet, the computing world carries on. Terminal programs are being run in mind-boggling numbers every minute of every day, and the overall experience is still fine. Sometimes those programs will add workarounds themselves to odd but significant enough problems.

Is it really worth not setting the variables as describe above in spite of all the programs that would work just fine? Feature detection based on ANSI sequences is burdensome for every program to implement when a couple of env vars are much easier to check and correct in the vast majority of cases. And feature detection via ANSI escape sequences don't give you the full picture anyway. Just because it reports truecolor doesn't mean it supports hyperlinks. And it's been discussed how different terminals respond to : or ; separators while some support either. It's already a grab bag of different behaviors. Why make it harder?

[CannibalVox]

What is the actual status of this ticket? I see earlier in the ticket that there was some resistence to using $COLORTERM, but then broadly there seems to be support, even from maintainers, throughout the rest of the ticket. The ticket is tagged with "Help Wanted". Is this ticket really waiting for someone to come along and add a single environment variable in a PR?