Allow extensions to publish beta releases and users to opt-in to them

[DanTup]

Apologies if this has come up before; I'm unable to find any related issues.

It would be cool to be able to publish extensions in a beta form (similar to Code Insiders) so that they can be run "in production" with real users that have opted-in. This would reduce the disk in publishing an extension because you can push it to a smaller number of users for testing (I probably don't need to sell this to you, I guess you have Insiders for very similar reasons).

I know we can package and distribute extensions ourselves for now, but it's not as easy (eg. updating is not automatic).

I originally thought the "Preview" flag might've provided this functionality but unless I've overlooked something, it does not seem to.

[ramya-rao-a]

I would very much like this myself.

[joaomoreno]

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true? Once the feature is stable enough just remove the setting.

[DanTup]

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true?

This is an option, but I think it has some drawbacks

1. If you want to do a beta release every day, your "stable" users will get the update notifications too
2. It doesn't eliminate all the risk (if you're shipping bugs in the new features, maybe your feature detection is buggy, or how you're dealing with two code paths)
3. For things that aren't just new features, it's more work to maintain/switch between the branches
4. Users have to keep opting-in to each feature, rather than choosing to be a beta tester once; this could really limit the testing you get
5. Having the feature toggles show up in the config auto-complete isn't ideal (this might not be an issue, but I didn't get a response to Allow extensions to set configuration options. #1396 (comment) so it's unclear if "hidden options" are supposed to be used)

None of these issues are huge, but since the Insiders builds work so well and it seems like a similar scenario it seemed like a good request to raise for discussion :)

[davidanthoff]

I agree, this would be a really useful feature. Right now I'm distributing our beta releases via VSIX downloads, but that is cumbersome and I don't reach many folks that way.

Hiding things behind a feature flag wouldn't really help. For example, we are about to release a new version that changes a lot of code, the whole architecture of the extension. There is no way that we could have had that behind a feature flag.

[eamodio]

I would love to see this as well -- while feature flags work in certain cases, it doesn't cover them all. It also provides a sanity check and limits the "damage" of a bad release. And installing via vsix is just too user unfriendly to really be viable imo -- as it is completely manual.

[eamodio]

Just to kick this one again, because the issues surrounding the manual install and REQUIRED uninstall of vsix causes lots of pain for users and seemingly creates cases of corrupt installations -- I've definitely seen an up tick in GitLens issues caused by vsix/marketplace versions on top of one another.

[DanTup]

Yeah, I meant to post back here - when I did a beta release a few months ago I had reports of issues installing/uninstalling the vsix:

• Flutter beta testing Dart-Code/Dart-Code#318 (comment)
• Flutter beta testing Dart-Code/Dart-Code#318 (comment)
• Flutter beta testing Dart-Code/Dart-Code#318 (comment)

[DanTup]

I gave up on vsix testing because of the complications mentioned above. Yesterday I shipped a huge bug that only occurred if dart on your PATH was a symlink (I had to make a lot of changes for multi-root support, and putting it all behind flags was not feasible) :(

I'm starting to think about publishing a second version of my extension with Beta in the name to allow people to choose to test new releases earlier (seems like at least one other has done this).

[sandy081]

@DanTup @eamodio We did some improvements in this milestone (November) to support beta releases in VS Code when installed through VSIX. Refer to this issue #39024 for more details. Can you please check if that helps you in using VSIX?

[DanTup]

@sandy081 Sounds good! Am I right in thinking november milestone is 1.19 and not released to stable yet? I'll try and do some testing latest this/next week.

[sandy081]

Yes 1.19 will be released to stable this week. Let me know if you still want this feature to be opened after that.

[DanTup]

@sandy081 Haven't had chance to test it yet; but I think there's still value in this feature even if vsix works better than it did, for example:

• I think end users would have more confidence installing a beta vsix through the marketplace then from some random download url
• Allowing people to opt-in to a beta channel would allow them to get auto-updates of betas instead of having to do manual installations all the time
• Listing betas on the marketplace is much more discoverable - only a fraction of my users will follow the twitter account (etc.), so there are a lot of potential testers that simply won't know about beta vsix releases

[sandy081]

@DanTup All your requirements are valid. But I think this has to be supported by the Market place first.

@modigrv May I know if there are any plans to have this support in Market place?

[vscodebot]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[DanTup]

😔

[CsCherrYY]

@isidorn Thanks for the hard work. We're looking forward to a way to publish Type Hierarchy feature as a preview feature and pre-release extension is a great choice. However, the related APIs are in proposed state (see microsoft/language-server-protocol#1231 (comment) and https://github.com/microsoft/vscode-languageserver-node/blob/main/client/src/common/proposed.typeHierarchy.ts), enabled by enabledApiProposals and we can't publish it even to the pre-release channel.

The question is that we want to find a way to take advantage of the pre-release channel to make more users use our preview features with proposed APIs, is that possible? Or is there any consideration about this?

[isidorn]

@CsCherrYY there have been some discussions around this but nothing definite yet. We want to make sure that the pre-release extension support is adopted smoothly across our community before we decide to build any additional features on top.

[davidanthoff]

Is there some way to find out from code whether the currently running extension was installed as a pre-release version or not? We want to send telemetry to different destinations, depending on whether a user is running a pre-release version or not.

[TwitchBronBron]

Is there some way to find out from code whether the currently running extension was installed as a pre-release version or not? We want to send telemetry to different destinations, depending on whether a user is running a pre-release version or not.

A version cannot be both pre-release and not pre-release, so if you manage the extension, then you are in control of the process for launching pre-release versions and therefore know which versions are pre-release. If you follow the suggestions outlined in the documentation:

we recommend that extensions use major.EVEN_NUMBER.patch for release versions and major.ODD_NUMBER.patch for pre-release versions. For example: 0.2.* for release and 0.3.* for pre-release.

then you can easily check the current extension version and act accordingly based on whether is an even or odd minor release.

[isidorn]

@davidanthoff exactly as @TwitchBronBron explained: you can look at your extension version and act accordingly.

[rchiodo]

There's a lot to read through in this issue and I wasn't sure what the expected behavior is with regards to 'insiders'.

Should insiders users automatically be updated the prerelease of an extension?

This isn't currently happening for the Jupyter Extension.

I had to explicitly pick 'Switch to prerelease version'

[sean-mcmanus]

Uh oh...if VS Code Insiders users have to manually opt-in, we're likely to see our Insiders usage drop by a lot. @bobbrow Are you sure we should switch to the new mechanism if that's the case? And/or can we get VS Code to change the behavior first?

[isidorn]

We support automatically migrating separate extensions to the pre-release version of the main extension - extension authors just have to inform us they want this.
However since C++ and Jupyter have a special case and you do not have separate extensions we currently do not support automatic migration done by VS Code. However you can do the migration on your own as discussed here microsoft/vscode-cpptools#8507 (comment)

[gjsjohnmurray]

For me just now, when I used Insiders and searched for the GHPRI extension (which I know published pre-releases), the Install button defaulted to Pre-Release:

Doing the same with Stable defaults to a regular install:

[rchiodo]

We support automatically migrating separate extensions to the pre-release version of the main extension - extension authors just have to inform us they want this. However since C++ and Jupyter have a special case and you do not have separate extensions we currently do not support automatic migration done by VS Code. However you can do the migration on your own as discussed here microsoft/vscode-cpptools#8507 (comment)

@isidorn I'm not sure that's going to work? Won't it force all users to the latest? For me I didn't want the latest, but when I do, it should show me the latest is the prerelease version. That's the behavior we have right now. Users pick when they upgrade, but the recommended upgrade is the most recent insiders only build.

[isidorn]

@gjsjohnmurray correct.
@rchiodo yes, the insiders users will by default get the option to install Pre-Release. My previous answer was my misunderstanding of your question. For me this just works

[isidorn]

@rchiodo you would just do that code for your insider version of the extension. So that way you would transition only your insider users to the new pre-release version.

[rchiodo]

@isidorn that's the behavior with the jupyter extension not already installed. But if it was already installed, the update notification won't pick prerelease.

And I believe the command suggested would force an update? Maybe I'm wrong about that.

[rchiodo]

I guess it won't be terrible to force prerelease if the extension isn't already in insiders for a specific version (one time thing).

Maybe we can add code to ask the user on insiders.

[isidorn]

@rchiodo there is just one minor thing missing to have this flow super nice #139528
We plan to tackle this in the next two weeks.

And yes. So my full suggestion is the following:

1. For your current Insiders jupyter version do the following
2. Show a persistent notification to the user, or a modal dialog "bla bla we moved to pre-release - click here to be migrated"
3. Once user clicks -> execute the code. Once Opt in to pre-release versions when installing an extension with installPreReleaseVersion flag #139528 is done all should be good

Hope I was more clear now. Thank you

[eamodio]

Also another way to ensure users don't automatically get switched to a non-insiders release (if that is what you want) -- you can have your insiders versions always be greater than stable version. For example in GitLens I am planning to keep my YYYY.MM.DDHH version format for insiders.

[alexr00]

Also another way to ensure users don't automatically get switched to a non-insiders release (if that is what you want) -- you can have your insiders versions always be greater than stable version.

GitHub Pull Requests and issues does the same thing. Before releasing a new stable, we will always release a new insiders with the increased version.