VS Code extension secrets when default keyring is unavailable/locked

[cgrama]

When default keyring is unavailable or locked, extensions are able to store/get/delete extension secrets using VSCode secret storage apis with NO errors.
We get a vscode notification (in the notification windos) "You're running in a GNOME environment but the OS keyring is not available for encryption. Ensure you have gnome-keyring or another libsecret compatible implementation installed and running.".
There is NO error returned warning that secrets are NOT stored on the disk.
The extension's secrets stored in this scenarios are available for use by the extension in the same vscode session. If you close and reopen vscode the extension's secrets are no longer available.
This makes it very difficult for secrets to be used by vscode extensions as the extension does not know if the secret was stored in memory or on the disk.
Questions:
(1) How can a vscode extension check if the extension secret is stored in-memory or in the vscode secret storage ?
(2) Can the extension secret store apis be enhanced to return an error or return value which informs the api caller that the secret is stored in-memory and not in vscode secret storage ?
(3) Also from the secret storage info shared at #748, is the default keyring used only for encryption OR is the default keyring used for storing the extension secrets also ?

Included below is a prt of the vscode console output from "code --user-data-dir=/scratch/vscode2 code --verbose --vmodule="/components/os_crypt/=1" | tee verbose_upgrade.log":
app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003951.673893:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Setting password for vscextn.extn1 extension: key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003951.675050:VERBOSE1:key_storage_util_linux.cc(52)] Password storage detected desktop environment: GNOME
[2669207:0510/003951.675065:VERBOSE1:key_storage_linux.cc(118)] Selected backend for OSCrypt: GNOME_LIBSECRET
[2669207:0510/003954.651823:VERBOSE1:libsecret_util_linux.cc(169)] Dummy store to unlock the default keyring failed: No such interface “org.freedesktop.Secret.Collection” on object at path /org/freedesktop/secrets/collection/Default_5fkeyring
[2669207:0510/003954.651849:VERBOSE1:key_storage_linux.cc(165)] OSCrypt using Libsecret as backend.
[2669207:0510/003957.531557:VERBOSE1:key_storage_libsecret.cc(73)] Libsecret lookup failed: No such interface “org.freedesktop.Secret.Collection” on object at path /org/freedesktop/secrets/collection/Default_5fkeyring
[2669207:0510/003957.538272:INFO:CONSOLE(656)] "%cTRACE color: #888 [SecretStorageService] Encryption is not available, falling back to in-memory storage", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.545398:INFO:CONSOLE(656)] "%cTRACE color: #888 [NativeSecretStorageService] Notifying user that secrets are not being stored on disk.", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.548906:INFO:CONSOLE(796)] "You're running in a GNOME environment but the OS keyring is not available for encryption. Ensure you have gnome-keyring or another libsecret compatible implementation installed and running.", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (796)
[2669207:0510/003957.549118:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] encrypting secret for key: {"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.549318:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] storing encrypted secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.549540:INFO:CONSOLE(656)] "%cTRACE color: #888 [SecretStorageService] Notifying change in value for secret: {"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.552598:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] stored encrypted secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.558749:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Password set for: vscextn.extn1 key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.636472:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Getting password for vscextn.extn1 extension: key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.638363:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] getting secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.638899:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] decrypting gotten secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.641229:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] decrypted secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.641454:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Password found for: vscextn.extn1 key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.648246:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Getting password for vscextn.extn1 extension: key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.648418:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] getting secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.648558:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] decrypting gotten secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.649588:INFO:CONSOLE(656)] "%cTRACE color: #888 [secrets] decrypted secret for key: secret://{"extensionId":"vscextn.extn1","key":key1}", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)
[2669207:0510/003957.650248:INFO:CONSOLE(656)] "%cTRACE color: #888 [mainThreadSecretState] Password found for: vscextn.extn1 key1", source: vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (656)

[TylerLeonhardt]

Hi @cgrama, I have some context and a followup question:

(1) How can a vscode extension check if the extension secret is stored in-memory or in the vscode secret storage ?
(2) Can the extension secret store apis be enhanced to return an error or return value which informs the api caller that the secret is stored in-memory and not in vscode secret storage ?

As you've called out, we do notify the user that their keyring is in a bad state and they need to remediate. Including attaching our troubleshooting guide. It's intended to call out that they are in a bad state... Any solution would either be error prone (us fixing it for them) or a questionable security practice (storing the secrets in plaintext).

With that said, my question for you is... what would you like to do with this information that they are in a bad state?

(3) Also from the secret storage info shared at #748, is the default keyring used only for encryption OR is the default keyring used for storing the extension secrets also ?

The default keyring is only used for encryption. In other words, it holds a piece of the encryption key needed to encrypt/decrypt secrets which are then stored in the user's data directory with other state like UI state.

[TylerLeonhardt]

Here's some rough code on how to do this:

class MigrationAwareSecretStore {
	private migrationKey = 'migrated-secret-store';

	constructor(
		private secretStore: SecretStorage,
		private globalState: Memento,
		private customStore: CustomSecretStorage
	) { }

	/**
	 * NOTE: This needs to be called before any other method to ensure that the migration is complete
	 */
	async initialize() {
		const migrated = this.globalState.get<boolean>('migrated-secret-store');
		if (!migrated) {
			const newData = await this.secretStore.get('myData');
			const oldData = await this.customStore.get('myData');
			// If we have new data, or there is no data in either store, then we can delete the old data
			// and mark the migration as complete
			if (newData || (!newData && !oldData)) {
				this.customStore.delete('myData');
				await this.globalState.update('migrated-secret-store', true);
			} else {
				// There's no data, but there is custom data so we need to copy
				// the data from the old store to the new store
				if (oldData) {
					await this.secretStore.store('myData', oldData);
				}
			}
		}
	}

	get(key: string): Thenable<string | undefined> {
		return this.secretStore.get(key);
	}

	async update(key: string, value: string): Promise<void> {
		if (this.globalState.get(this.migrationKey)) {
			await this.secretStore.store(key, value);
		} else {
			await this.customStore.store(key, value);
			await this.customStore.store(key, value);
		}
	}
}

I haven't tested it, but it gives you a rough idea of what I'm talking about... and then in a few versions, you can remove this migration logic altogether and use only the SecretStorage

[cgrama]

Scenario 1 (vscode secret storage "store" api does not return any error/exception on failure (for example: bad or missingg keyring)):
In this scenario, every vscode extension migrating extension secrets from custom store to vscode secret storage, will have to do the following:
1.1. Migrate extension secrets from custom store to vscode secret storage
1.2. Implement the following (even if migration was successful and there are no keyring issues):
1.2.a. Implement the "migrationawaresecretstore" sample code
1.2.b. Force a reload/reopen of vscode to complete the migration.
1.2.c. Report migration failed if both customstore and secretstore don't have the same secret only after 1.2.b.

Scenario 2 (vscode secret storage "store" api returns some error/throws exception on failure (for example: bad or missingg keyring)):
In this scenario, every vscode extension migrating extension secrets from custom store to vscode secret storage, will have to do the following:
2.1. Migrate extension secrets from custom store to vscode secret storage
2.2. Report migration failed only if vscode secretstorage "store" api returns error/ throws an exception. Note that no reload/reopen of the vscode is needed.

Based on the above scenarios, vscode secretstorage "store" api returning error/ throwing an exception, results in a better outcome - less code for vscode extension and a better vscode usability experience.

[TylerLeonhardt]

I don't think this should be done:

1.2.b. Force a reload/reopen of vscode to complete the migration.

The user will eventually reload the window on their own time.

Additionally, the migration code is temporary. I would keep the migration in place for 6 months to 1 year... surely during that period, all users would have migrated and you can solely rely on the new Secret Store. Or use telemetry to see how many users still have data in the old store but not the new store.

Doing the above means no data loss for existing users and new users are told that they need to fix their keyring (which I think we, in VS Code, can be more aggressive about).

All of this with no pain to the user and minimal (temporary) code for your extension.

I'm pushing back because API in VS Code is forever. We cannot change it. You are the only extension author who has brought this up and additionally we are talking about a very small amount of users (only Linux without a working keyring). Having forever API for this amount of impact and with negatives (extensions spamming users with notifications) is why I don't think we should do this.

The code I wrote above should unblock your scenario while keeping VS Code's API lean.

[cgrama]

If the secret storage "store" api cannot be updated,
can vscode add a new secret storage "storewithNotify" api that can be used by extensions that want to get notified of failures when saving extension secrets to vscode secret storage ?
The proposed workaround to handle failures during migration of extension secrets to secretstorage is not an acceptable solution as it only delays notifying the vscode developer of the migration failure.

[TylerLeonhardt]

You can open an issue here asking for this:
https://github.com/microsoft/vscode/issues

Ultimately, adding API like this is something we only do with high impact and a lot of extension authors wanting it. Just as I said above, I don't think this falls into either camp.

You are the only extension author who has brought this up and additionally we are talking about a very small amount of users (only Linux without a working keyring).