Language specific resource settings are not resolved for files opening right on startup

[Raydir]

I've made a language extension, that sets "files.encoding" to cp850 and has a definition for which file-extensions this extension should be used.

It seems everything is working fine as long i open my files from within VSC or if VSC is already running and i double click on a file e.g. from desktop.

If VSC is NOT running and i open the same file from e.g. desktop, it sets the language mode (lower right) correct, but the encoding is not set to cp850.

I also tried to overrule the settings with customer-specific settings for my language

but when i reopen vsc by double clicking on a file this wont work either - if i see it right, vsc seems to "think" there is a comma at the end of my string..

...but there is no comma...

How can i bring VSC to open a file with the correct encoding used by my extension? I dont want to set it by default, because i want to use VSC also for javascript with UTF8...

Thanx
Roman

• VSCode Version: 1.19.0-Insider
• OS Version: Windows 10

[bpasero]

At first I thought this was a general problem with our config, but it is not (removing the "important" label again).

The issue is that textResourceConfigurationService.getValue is called early enough on startup that the extensions are not being parsed yet and as such the markdown extension is not contributed yet.

I verified that this issue exists in stable, so it is NOT a regression from us moving the extension host starting to a later phase.

I am not sure about how to fix this unfortunately.

[Raydir]

@bpasero & @sandy081 the strange thing on this topic is also, the point with the "comma" when i try to set "language related settings" and overrule the encoding this way. it seems those settings are also not working.

Looking forward to hear from you guys.

[sandy081]

@bpasero Yes, by the time we get the encoding value extensions are not read as a result languages are not registered. Hence non-language overridden values is being read.

I see listening to configuration change event is also not helpful here.

[sandy081]

@Raydir A trailing comma should not have any impact. Please let us know the scenario that is not working with trailing comma. Also, please file a new issue for that. Thanks.

[Raydir]

@sandy081 i'm not sure if the trailing comma is a problem - i just tried to force the encoding by calling the encoding for my language by that setting. I see that you an @bpasero, found a reason why the encoding isn't used, so i forget about the "comma issue" since that seems to be related.

[bpasero]

I think this issue is not specific to file-encoding but will occur for any resource setting potentially simply because by the time we resolve the settings we do not have any language specific setting resolved.

[Raydir]

I agree @bpasero

[sandy081]

by the time we resolve the settings we do not have any language specific setting resolved.

Just to be clear, It's languages that are not registered.

[Raydir]

what you mean by "registred"?
My language is shown in the lower right panel... so it looks like the "language itself" is registred at that moment...

[bpasero]

Just to be clear, It's languages that are not registered.

Yes. Because that only happens when the extension host has scanned all extensions and for performance reasons we do not want to wait on that. I think for some settings we might be able to mitigate this issue by updating the config after the languages have been resolved and we find out that the opened editor matches a language with specific settings. But for things like the encoding it would not help because the file is already opened...

[Raydir]

@bpasero i did think about this issue again during xmas time a bit. Main problem is, that "autoguessencoding" does not work for me correct, since it supposes that my file is CP866 what would be cyrillic...

is there a reason why cyrillic is "guessed"?

As i see it, there is a relation between the place where you really "are" aka "regional settings"
.NET class "RegionInfo" and the corresponding codepage for DOS.

If you type chcp in DOS you'll also would get the "normaly" correct DOS codepage.

Could this be a workaround to get autoguessencoding get working?

If so we could close this issue here and open another for autoguess DOS-codepages

[fmonroe]

Any news on this issue?

[ghost]

@fmonroe Try setting files.autoGuessEncoding to false. I thought I was hitting this issue, but it turns out to be #67303 instead.

[vscodebot]

This issue is being closed to keep the number of issues in our inbox on a manageable level, we are closing issues that are not going to be addressed in the foreseeable future: We look at the number of votes the issue has received and the number of duplicate issues filed. More details here. If you disagree and feel that this issue is crucial: We are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding and happy coding!

[Fred-Vatin]

Is this a joke ? vscodebot closes serious issues and all its related like #122632 ? Does it sound normal to ignore bugs during four years ?

One can mess with big problems if he begins to write and save a file with the wrong encoding when previously saved with another encoding.

For instance. These settings ↓ are not enforced. When a bat file encoded with cp850 is opened while VSCode is not currently running, the file is opened with encoding tis620. It seems it tries to autoguess encoding while I specifically tell it not to do so.

	"[bat]": {
		"files.autoGuessEncoding": false,
		"files.encoding": "cp850",
		"files.eol": "\r\n",
	},

[VisionXi]

How do we get priority on this issue? It's really annoying.
If it's really because the extension host has to scan all extensions and for performance reasons we do not want to wait on that, then why not trying to solve this by a configurable setting. I could live with a small delay when opening a file with vscode. This delay would only occur if vscode was not opened before.

[haudan]

@bpasero is it possible to look into this issue again? This is a blocker that keeps VSCode from being adopted at our company and we would really love to get away from SciTE. I'm happy to answer any questions.

[bpasero]

Yeah, one question would be how it is possible that this bug blocks a company from using VSCode? What is the scenario that fails and requires devs to use a different editor or IDE?

[Raydir]

@Lisoph a possible workaround:
open the folder that contains the files and not a specific file.

This way the settings are already loaded when you open a specific file within by pressing e.g. "F1".

[haudan]

@bpasero thanks for the quick reply. I'm aware this isn't typical nowadays, but here it goes:

We're working with ERP software that always encodes source files in Windows-1252 (for historical reasons). In conventional software projects, you have a source directory from which you build or run your artifacts. This ERP however treats source code as an asset that is attached to some component - kind of like how scripts are attached to GameObjects in Unity. Unlike Unity, to edit you click a button and it invokes your editor by launching a process with a temporary copy of the file (which is beamed back to the mothership after you're done with it). We're not the vendor of the ERP, so we can't change this behavior. (@Raydir your workaround doesn't work for us for this reasons - thanks for the help).

The problem stems from our developers not necessarily having VSCode up and running when beginning to edit code. In that case, VSCode is freshly launched and inevitably the file opens in UTF-8. If the developer doesn't catch this, they're going to destroy the source on save - documentation, string literals and some identifiers. This has happened before and this is what blocks adoption.

Currently I'm the only dev using VSCode and, like a maniac, I make sure that I've manually launched it before clicking the edit button of death. As you can imagine, this is error prone. I've had conversations with consultants (from other companies) who would loved to switch to VSCode, but I had to turn them off for this reason.

Another workaround was to set Windows-1252 as the global default. This does work, but then we have the same problem when editing UTF-8-without-BOM files. Yet another workaround was to turn our temp source directory into a workspace and to set Windows-1252 as a workspace setting, but sadly this didn't work either, VSCode defaulted to UTF-8.

To summarise our problem:

• VSCode is always invoked for every edit with a temporary Windows-1252 file.
• VSCode is not guaranteed to be up and running (manaually launched). It might be a fresh start, which enables the bug described in this issue.

[bpasero]

@Lisoph for me to understand: when the file opens you are basically in a "empty" VSCode window with just the file right? Empty is indicated in a different background color of the status bar:

I am not sure how even the current way settings work would help here because in this model:

• you are not in a workspace, so workspace settings do not apply
• there is no way to globally configure an encoding for a specific file path (files.encoding is always global for all or workspace specific)
• you cannot really tell VSCode on startup which encoding to use

Is that accurate? My gut feeling is that even with a possible fix for this issue, the experience would still not be nice unless you force users into a workspace all the time.

[bpasero]

Btw there are lots of duplicates and some also talk about the reasoning the way it works today. Please take a look. It is somewhat of a chicken and egg problem: we don't want to prolong opening a file until we know the encoding but we cannot pick the right encoding from the beginning in some cases. As a stretch, we could automatically reopen in the right encoding later, with the risk of the user having made changes already...

To be fair, this issue is still active in #127936 and I would say #23570 is also related as a possible solution.

[haudan]

for me to understand: when the file opens you are basically in a "empty" VSCode window with just the file right? Empty is indicated in a different background color of the status bar:

Yes, VSCode is launched as a "blank" window, just with the requested file loaded - same status bar color.

• you are not in a workspace, so workspace settings do not apply
• there is no way to globally configure an encoding for a specific file path (files.encoding is always global for all or workspace specific)
• you cannot really tell VSCode on startup which encoding to use

Is that accurate?

Sounds about right.

1: All files land in the same dir, which we can turn into a VSCode workspace, that was a workaround we tried, but the last time I tested this it did not change VSCode's behavior. Because workspace or not, VScode is invoked the same (just for the file and not the folder). Or am I wrong and should that work?

2-3: Yes, unless VSCode has some settings that can do this, that I'm not aware of.

I should've mentioned I'm using the following settings, sorry about that:

"window.openFilesInNewWindow": "off",
"window.restoreWindows": "none",

This is to reuse the already open window when editing multiple files and to not reopen closed (dead) temporary files.

VSCode is invoked as code -w -r <the_file>

My gut feeling is that even with a possible fix for this issue, the experience would still not be nice

Besides the encoding, it's not that bad actually. Our ERP's language and tooling is so bizarre that VSCode just acts as a "dumb" text editor with syntax highlighting anyway. We have what we need (in-house extension) and what we're missing is fundamentally impossible, because the ERP doesn't support it. For all our other VSCode needs we're using it in a sane way, so that works out well.

Since the VSCode extension for the ERP language is developed in-house (should've mentioned that too, sorry), is there a way to programmatically switch to Windows-1252? This is something I've researched but I couldn't find an API for it at the time. I assume this hasn't changed?

I'll look into the issues you've linked.

[bpasero]

@Lisoph if you can control how VSCode is opened, you can force it to open the workspace and then the file via:

code -w -r <the_folder> <the_file>

Then when you can configure workspace settings for that folder, it should work.

is there a way to programmatically switch to Windows-1252

No, see #824

[bpasero]

I have pushed an attempt to resolve this, see #127936 (comment)

[bpasero]

@Lisoph can you maybe give your scenario a test from the newly released insiders version of today where a fix is included:
You can give our preview releases a try from: https://code.visualstudio.com/insiders/

To clarify: it should work when an encoding is configured in settings for the language of the file in question.

[haudan]

@Lisoph if you can control how VSCode is opened, you can force it to open the workspace and then the file via:

code -w -r <the_folder> <the_file>

Then when you can configure workspace settings for that folder, it should work.

Wow! This did it, it's working now. I would've never thought of this, yet it's so obvious! Thanks so much!

I have pushed an attempt to resolve this, see #127936 (comment)
@Lisoph can you maybe give your scenario a test from the newly released insiders version of today where a fix is included:
To clarify: it should work when an encoding is configured in settings for the language of the file in question.

I'll give it a shot (without the workspace) - will let you know.

[bpasero]

Thanks!

[haudan]

@bpasero Sorry for the wait, been busy with work.

So I've tested your changes with 1.67.0-insider (e7c6ad7) and it looks like it's working!

I tried out two methods for providing the language specific encoding:

1. I copied over our ERP-extension, which contributes some default settings for the file type, including encoding. Besides some workspace trust things, I did not change or add any settings.
2. Not using our extension this time (empty extensions folder), I added files.associations and the lang-specific "files.encoding": "windows1250" to settings.json. Besides workspace trust, no other settings were touched.

For each method, I tested the following two scenarios and I made sure the files were not opened with the workspace "trick" you mentioned last time:

1. Invoke VSCode with a file when it is already running (empty window, no folder / workspace).
2. Invoke VSCode with a file when it is not yet running (launch VSCode in the process).

The ERP files were always correctly opened in Windows-1252. Other files retained the default encoding and were opened in UTF-8.

Looks like a job well done!

[bpasero]

Thanks!