[Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability #10133

Hnasar · 2024-08-01T23:49:48Z

Is there an existing issue for this bug?

I have searched the existing issues

Required Troubleshooting Steps

I have followed these troubleshooting steps
I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

I typically use vscode-remote to use vscode on a linux server over ssh. When I got the July 2024 release today, I started getting errors connecting to the server.

Looking at the output (included below) it seems like there were errors during server installation that resulted in crashes on startup due to missing javascript modules.

Workaround was to rm -rf ~/.vscode-server from my ssh server.

A handful of my colleagues also experienced similar issues. We seem to have these kinds of issues every time a new update gets released.

Expected Behavior

When updates roll out, it doesn't result in javascript errors that crash the app.

Steps To Reproduce

Wait for an automated upgrade to be rolled out

Remote-SSH Log

logs here (too big to attach) https://gist.github.com/Hnasar/4b295e51f2fc255ccbfc1dee3d4f9248

Anything else?

The home directory on the linux server is mounted with NFS, if that matters.

The text was updated successfully, but these errors were encountered:

kgarg8 · 2024-08-02T05:58:03Z

Facing the same issue since today morning. Downgrading to 1.91

kingpalethe · 2024-08-15T10:47:11Z

Terrible instability for me in the last 2 weeks for remote tunnels.

Hnasar · 2024-09-05T19:39:45Z

Once again, after another update, 1.93 and we're seeing reports of failed connections. One user reported deleting ~/.vscode-server on the remote machine fixed vscode being unable to connect over ssh.

(e.g. here's another bug filed #10229 )

Hnasar · 2024-09-05T19:54:39Z

One error I saw today in the macOS logs on the local side:

[15:11:02.850] Resolver error: Error: Unbound variable 'wgetTriesSegment' in script 'linux-exec-server-installer.sh'
	at /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:543500
	at String.replace (<anonymous>)
	at t.execServerInstallScript (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:543456)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:574465
	at async R (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:573569)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:549844
	at async T (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:547895)
	at async t.resolveWithLocalServer (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:549384)
	at async k (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:572515)
	at async t.resolve (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:576470)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:844816
[15:11:02.851] ------

Click to expand longer error log

[15:10:45.663] ------

[15:11:02.830] Log Level: 2
[15:11:02.834] SSH Resolver called for "ssh-remote+hostname-username", attempt 1
[15:11:02.835] "remote.SSH.useLocalServer": true
[15:11:02.835] "remote.SSH.useExecServer": true
[15:11:02.835] "remote.SSH.path": undefined
[15:11:02.835] "remote.SSH.configFile": undefined
[15:11:02.835] "remote.SSH.useFlock": true
[15:11:02.835] "remote.SSH.lockfilesInTmp": true
[15:11:02.835] "remote.SSH.localServerDownload": off
[15:11:02.835] "remote.SSH.remoteServerListenOnSocket": false
[15:11:02.835] "remote.SSH.showLoginTerminal": false
[15:11:02.835] "remote.SSH.defaultExtensions": []
[15:11:02.835] "remote.SSH.loglevel": 2
[15:11:02.835] "remote.SSH.enableDynamicForwarding": true
[15:11:02.835] "remote.SSH.enableRemoteCommand": false
[15:11:02.836] "remote.SSH.serverPickPortsFromRange": {}
[15:11:02.836] "remote.SSH.serverInstallPath": {}
[15:11:02.836] "remote.SSH.permitPtyAllocation": false
[15:11:02.836] "remote.SSH.preferredLocalPortRange: undefined
[15:11:02.836] "remote.SSH.useCurlAndWgetConfigurationFiles: false
[15:11:02.838] VS Code version: 1.93.0
[15:11:02.838] Remote-SSH version: remote-ssh@0.114.0
[15:11:02.839] darwin arm64
[15:11:02.844] SSH Resolver called for host: hostname-username
[15:11:02.844] Setting up SSH remote "hostname-username"
[15:11:02.846] Acquiring local install lock: /var/folders/cw/zvq4p9252sd4jfl5_mt5jndc0000gq/T/vscode-remote-ssh-e0e19ffb-install.lock
[15:11:02.846] Looking for existing server data file at /Users/username/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-e0e19ffb-4849ca9bdf9666755eb463db297b69e5385090e3-0.114.0-es/data.json
[15:11:02.846] No existing data file
[15:11:02.847] Using commit id "4849ca9bdf9666755eb463db297b69e5385090e3" and quality "stable" for server
[15:11:02.848] Script variables:
 {
  "InstallExitCode.24": "AlreadyInProgress",
  "InstallExitCode.25": "ServerDownloadFailed",
  "InstallExitCode.26": "NoDownloaderAvailable",
  "InstallExitCode.27": "UnsupportedArch",
  "InstallExitCode.28": "StatusCheckFailed",
  "InstallExitCode.29": "NeedInsidersArch",
  "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck",
  "InstallExitCode.31": "ServerTransferFailed",
  "InstallExitCode.32": "ServerFailedToStart",
  "InstallExitCode.33": "NeedInsidersWindows",
  "InstallExitCode.34": "CreateInstallDirFailed",
  "InstallExitCode.35": "UnsupportedPlatform",
  "InstallExitCode.36": "ServerTerminatedCVE20201416",
  "InstallExitCode.37": "UnpackFailed",
  "InstallExitCode.38": "ChangeDirFailed",
  "InstallExitCode.AlreadyInProgress": "24",
  "InstallExitCode.ServerDownloadFailed": "25",
  "InstallExitCode.NoDownloaderAvailable": "26",
  "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30",
  "InstallExitCode.UnsupportedArch": "27",
  "InstallExitCode.StatusCheckFailed": "28",
  "InstallExitCode.NeedInsidersArch": "29",
  "InstallExitCode.ServerTransferFailed": "31",
  "InstallExitCode.ServerFailedToStart": "32",
  "InstallExitCode.NeedInsidersWindows": "33",
  "InstallExitCode.CreateInstallDirFailed": "34",
  "InstallExitCode.UnsupportedPlatform": "35",
  "InstallExitCode.ServerTerminatedCVE20201416": "36",
  "InstallExitCode.UnpackFailed": "37",
  "InstallExitCode.ChangeDirFailed": "38",
  "InstallUnpackCode.Success": "success",
  "InstallUnpackCode.Error": "error",
  "InstallUnpackCode.MissingFiles": "missingFiles",
  "uuid": "4c0b283d861c",
  "startMarker": "4c0b283d861c: running",
  "commitId": "4849ca9bdf9666755eb463db297b69e5385090e3",
  "quality": "stable",
  "token": "1a1aaaaa-1111-11aa-11aa-aa111111a1a1",
  "vscodeAgentFolder": "$HOME/.vscode-server",
  "allowClientDownload": "0",
  "forceClientDownload": "0",
  "cliNameInArchive": "code",
  "ignoreWgetConfigFlag": " --no-config ",
  "ignoreCurlConfigFlag": " --disable ",
  "wgetTriesSegment": "",
  "listenArgs": "--on-host=127.0.0.1 --on-port",
  "getDownloadServerStartTrigger": "4c0b283d861c:trigger_server_download",
  "getDownloadServerEndTrigger": "4c0b283d861c:trigger_server_download_end",
  "getProgressDownloading": "4c0b283d861c%%1%%",
  "getProgressInstalling": "4c0b283d861c%%2%%"
}
[15:11:02.850] Resolver error: Error: Unbound variable 'wgetTriesSegment' in script 'linux-exec-server-installer.sh'
	at /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:543500
	at String.replace (<anonymous>)
	at t.execServerInstallScript (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:543456)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:574465
	at async R (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:573569)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:549844
	at async T (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:547895)
	at async t.resolveWithLocalServer (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:549384)
	at async k (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:572515)
	at async t.resolve (/Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:576470)
	at async /Users/username/.vscode/extensions/ms-vscode-remote.remote-ssh-0.114.0/out/extension.js:2:844816
[15:11:02.851] ------

Hnasar · 2024-09-12T22:53:49Z

@bamurtaugh I'm wondering if you're the right person to help with this severe issue. Every time there's a vscode upgrade, it breaks for the remote-ssh extension for a percentage of our team members. This is extremely disruptive.

I support our team's usage of vscode and our workflow relies on using remote linux servers for development. Team members report inability to connect, missing extensions, crashes, etc, and this requires they manually rm -rf ~/.vscode-server.

For example, today after one team member was unable to reconnect using remote-ssh after the upgrade rolled out, we were able to see these logs on their local macbook:

[17:44:01.598] [server] * Visual Studio Code Server
[17:44:01.702] [server] Server bound to /usr/tmp/code-681e9d68-4395-4d09-93c1-6cfe04cbc5ee
[17:44:01.702] [server] Extension host agent listening on /usr/tmp/code-681e9d68-4395-4d09-93c1-6cfe04cbc5ee
[17:44:01.703] [server] parsed location: "/usr/tmp/code-681e9d68-4395-4d09-93c1-6cfe04cbc5ee"
[17:44:01.703] [server] 
[17:44:01.897] [server] [17:44:01] 
[17:44:01.897] [server] Error: Cannot find module '@vscode/spdlog'
[17:44:01.897] [server] Require stack:
[17:44:01.897] [server] - /mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js
[17:44:01.897] [server]     at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
[17:44:01.897] [server]     at Module._load (node:internal/modules/cjs/loader:986:27)
[17:44:01.898] [server]     at Module.require (node:internal/modules/cjs/loader:1233:19)
[17:44:01.898] [server]     at require (node:internal/modules/helpers:179:18)
[17:44:01.898] [server]     at n (/mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js:3:98)
[17:44:01.898] [server]     at r.load (/mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js:2:1637)
<-- snip -->
[17:44:01.899] [server]     at be (/mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/vs/server/node/server.main.js:198:6500)
[17:44:01.899] [server]     at async /mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js:4:10852
[17:44:01.899] [server]     at async Server.<anonymous> (/mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js:13:100) {
[17:44:01.899] [server]   code: 'MODULE_NOT_FOUND',
[17:44:01.899] [server]   requireStack: [
[17:44:01.899] [server]     '/mnt/nfs_fs_1/vol3/user/home/username/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/server/out/server-main.js'
[17:44:01.899] [server]   ],
[17:44:01.899] [server]   phase: 'loading',
[17:44:01.899] [server]   moduleId: '@vscode/spdlog',
[17:44:01.899] [server]   neededBy: [ '===anonymous2===' ]
[17:44:01.899] [server] }

Full log in gist: https://gist.github.com/Hnasar/faac2d9f702a6c5d4e9deb1cb37a3331

They also added,

Using vscode reminds me of when I used to flash ROMs on my phone because I got pretty good at reinstalling all of the apps.

This issue has been plaguing us for months and is driving users to use pycharm. I'm wondering if we should firewall off the vscode upgrade domain due to this issue. I'm happy to schedule a debugging call or provide any other information needed. Thank you for your work — otherwise, I'm a big fan of this extension!

bamurtaugh · 2024-09-13T16:47:33Z

Thanks for looping me in, and sorry to hear about the trouble. Looping in @joshspicer for his insights too!

joshspicer · 2024-09-13T17:40:43Z

Thanks for the logs @Hnasar. Specifically for what you've shared above, does this connection issue reproduce on the previous release version (0.113.1)?

I see /mnt/nfs_fs_1/... in that stack trace. Is the remote machine mounting in the user's home folder from elsewhere? Could you please share a bit more about the configuration of the remote system?

The nfs is raising flags for me. One thing we can try is setting the serverInstallPath to a path on the remote that isn't mounted in from elsewhere:

We also have this setting that was introduced specifically for (seemingly) this kind of setup:

If you decide to give this a go, please use the latest pre-release build of the extension. Changing channels can be done easily in the extension UI:

joshspicer · 2024-09-13T17:42:29Z

As an aside

We're investing this iteration (and likely the next one or two after) on improvements on how we define a baseline reliability metric for this extension (to prevent these unnecessary regressions). Some details are published publicly on our iteration plan and other ideas are under discussion. Please let me know if you have any other feedback you think would be valuable for me to incorporate.

Hnasar · 2024-09-23T03:48:28Z

Thanks for the replies Josh and Brigit, and thanks for the focus on bug fixing!!

I'll try out the pre-release, 'lockfiles in tmp', and moving the .vscode-server install to a local filesystem. I can't easily reproduce the upgrade failures, but this has been happening to my teammates & I a few dozen times over the past few months. So I will wait and see unless there's a better way to repro?

Also, is a possibility that the 'lockfiles in tmp' could be automatically selected as needed (e.g. if the server install is NFS)?
Otherwise, is there a convenient way to toggle this option out for all my teammates, or this is something that people have to do manually?

As for information about our setup, we have home directories mounted via NFS v3 from a standard NetApp network filesystem. The mount options are:
rw,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmax=5,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountproto=tcp,local_lock=none

joshspicer · 2024-10-01T16:57:34Z

Also, is a possibility that the 'lockfiles in tmp' could be automatically selected as needed (e.g. if the server install is NFS)?

We don't currently have anyway to dynamically detect this. If you have any ideas on a portable, generic way to do such a thing let me know! (Also, doesn't seem like folks always need this, and when we change the default behavior for one person it inevitably breaks someone else's setup 😉 )

I'll try out the pre-release, 'lockfiles in tmp', and moving the .vscode-server install to a local filesystem. I can't easily reproduce the upgrade failures, but this has been happening to my teammates & I a few dozen times over the past few months. So I will wait and see unless there's a better way to repro?

Please let us know if you've been successful on the latest releases, and if not what snags you're currently running into!

vs-code-engineering · 2024-10-09T12:33:02Z

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

Hnasar added the ssh Issue in vscode-remote SSH label Aug 1, 2024

Hnasar assigned roblourens Aug 1, 2024

Hnasar changed the title ~~[Remote-SSH Bug]:~~ [Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability Aug 2, 2024

bamurtaugh assigned joshspicer Sep 13, 2024

joshspicer added the info-needed Issue requires more information from poster label Oct 1, 2024

vs-code-engineering bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 9, 2024

vs-code-engineering bot locked and limited conversation to collaborators Nov 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability #10133

[Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability #10133

Hnasar commented Aug 1, 2024 •

edited

Loading

kgarg8 commented Aug 2, 2024

kingpalethe commented Aug 15, 2024

Hnasar commented Sep 5, 2024

Hnasar commented Sep 5, 2024

Hnasar commented Sep 12, 2024

bamurtaugh commented Sep 13, 2024

joshspicer commented Sep 13, 2024

joshspicer commented Sep 13, 2024

Hnasar commented Sep 23, 2024

joshspicer commented Oct 1, 2024

vs-code-engineering bot commented Oct 9, 2024

[Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability #10133

[Remote-SSH Bug]: Failed upgrade causing remote-ssh errors, disconnects, instability #10133

Comments

Hnasar commented Aug 1, 2024 • edited Loading

Is there an existing issue for this bug?

Required Troubleshooting Steps

Connect Locally

->

Expected Behavior

Steps To Reproduce

Remote-SSH Log

Anything else?

kgarg8 commented Aug 2, 2024

kingpalethe commented Aug 15, 2024

Hnasar commented Sep 5, 2024

Hnasar commented Sep 5, 2024

Hnasar commented Sep 12, 2024

bamurtaugh commented Sep 13, 2024

joshspicer commented Sep 13, 2024

joshspicer commented Sep 13, 2024

Hnasar commented Sep 23, 2024

joshspicer commented Oct 1, 2024

vs-code-engineering bot commented Oct 9, 2024

Hnasar commented Aug 1, 2024 •

edited

Loading