Add windows gui / msi installer [difficult]

[brson]

To complete the Rust installation experience on Windows we want to be installing rustup via an msi.

Make a proof-of-concept rustup msi installer that embeds libmultirust. The behavior of this msi will be heavily customized - all it does is the standard rustup install, but presented in a windowsy way. To start with it can be really simple:

• On installation, present a screen that says more-or-less what the console installer says now. Have install/cancel buttons.
• After they press install, go to another window that displays whatever status updates we reasonably can as the install is progressing.
• After install show another screen that indicates success.
• Register the uninstaller with windows so it works from the add/remove programs screen.

This will require a lot of refactoring of the existing install code to get it embedded in this new context.

Our options for GUI's will be limited but we can't use something heavy. I'm thinking either something rust-centric like conrod, or just a very thin win32 wrapper.

[retep998]

Depending on OpenGL for Conrod might not work very well if they don't have the GPU manufacturer drivers that provide OpenGL, or if they're using Windows in a VM.

[Diggsey]

We could just use Inno Setup... It will allow us to produce a GUI installer which can call out to the command line rustup behind the scenes, and it's configured declaratively, so we won't even need any non-rust code.

[brson]

@Diggsey Inno Setup does not produce msi's (I believe), and msi's are the delivery format people most expect and prefer.

Rust itself uses Inno to produce .exe installers but we don't advertise them because the .msi's are nicer.

[retep998]

If we use the Windows Installer stuff then that can provide a user interface for us.

[pravic]

@brson

and msi's are the delivery format people most expect and prefer.

.msi is a really big pain due to windows installer cache bloat :(

[vadimcn]

If it's just a window with some text and [Install]/[Cancel] buttons, the built-in msi UI will do just fine. Anything more complicated/dynamic than, say, the Rust installer, would be quite annoying to do in it, and in that case I would suggest going with WinForms/WPF.

I also want to note that invoking an .exe to perform the actual system changes (like mucking with the registry to modify PATH) goes against the "best MSI practices", but I guess we'd be fine with it for cross-platform consistency's sake?

[Diggsey]

@vadimcn Modifying the PATH would have to be done in a custom action anyway, even if .msi is used, so there's no real benefit in that regard. Aside from that, rustup doesn't actually make any changes to your system (even rustup itself goes in your user folder)

[vadimcn]

Modifying the PATH would have to be done in a custom action anyway, even if .msi is used,

Why? MSI has direct support for modifying environment vars.
But even when CAs are used, the recommended way is not to perform system changes directly, but rather to schedule them for execution by the MSI engine in elevated part of the install process (by adding ephemeral records to File/Registry/etc tables).

[brson]

@vadimcn I don't want to invoke an exe.

What I want to do is have the msi system call functions in the multirust dll to perform the installation actions, while presenting the UI people expect from an msi installer. Can we do that?

[vadimcn]

What I want to do is have the msi system call functions in the multirust dll to perform the installation actions, while presenting the UI people expect from an msi installer. Can we do that?

Yes, that is possible.
(This still counts as performing changes directly, though)

[nodakai]

In any cases, please keep supporting non-Administrator users installing Rust to somewhere under C:\Users\username or even C:\Users\Public

[Boddlnagg]

You may also try using WiX (http://wixtoolset.org/), it's a declarative way of creating a Windows Installer (msi) that can be heavily customized. It also says that it supports custom actions written in C++, so it shouldn't be too hard to use custom actions written in Rust. On the other hand, however, this might be too big of a non-Rust build system/dependency.

[jminer]

As a user, I've always preferred Inno Setup or NSIS installers over msi. Most msi installers I've used are slower and don't have as nice of a UI as the open source installers. They've also been buggier, but that might not be the Windows Installer's fault. I really don't want a UI made with conrad or .NET. Conrad does not look native and either one would add overhead.

I've written an NSIS installer before with similar screens, and I know it can call C functions. I'd be willing to work on an NSIS installer if there was a chance it would be used.

[nxnfufunezn]

How about this https://github.com/andlabs/libui

[tiborgats]

How about using Qt Quick ? It is more mature than libui, has better documentation, the problem of high dpi (4K UHD) screens is solved too.

[nxnfufunezn]

@tiborgats that would add an unnecessary dependency for QtQuickControls etc for a simple installer.

[brson]

I'd like to keep using WiX for this. Seems to be the most 'modern' choice. It's what we're using today.

[brson]

The way to get started here is just prototyping: figure out how to make WiX, the rustup library and the Win32 GUI APIs work together to present something that looks plausibly like an installer.

[Boddlnagg]

I tried to get this working and have a prototype running that ...

• uses WixUIExtension (built-in UI that looks like standard windows installers, and can be customized e.g. with a page for rustup install settings), so there's no need to use Win32 GUI APIs.
• runs a custom action from a Rust library (cdylib), which currently only reads a property from WiX (so we can read settings that are changeable in the UI) and logs it to the MSI log.

The only issues I encountered were the following:

• When building the DLL with the custom action I had to link some libs from the WiX SDK and build with a 32-bit compiler.
• Windows Installer requires something to be installed by the installer itself (either a file or a registry key), so I chose a registry key under HKCU\Software\rustup. Maybe the %USERPROFILE%\.rustup directory can be used instead (but that probably means that the installer will fail when it exists).

I have uploaded my experiments here: https://gist.github.com/Boddlnagg/9d8f01e6d844cd78473651470282ebda

I might be able to work more on this over the next weeks, but I don't want to keep anyone else from doing so, when they are faster.

[Boddlnagg]

Further investigations led me to the MsiProcessMessage function (though it is better to use the WcaProcessMessage version from wcautil), which can be used to send progress and status updates from the custom action back to the UI. This requires the custom action to run as deferred (instead of immediate), which is the right thing to do anyway. I have updated my gist to use deferred.

The rustup installation routines will have to be refactored in such a way that status updates can be either reported to the console or via WcaProcessMessage.

[brson]

@Boddlnagg thanks for doing that research! Using a registry key for registration seems just fine.

When building the DLL with the custom action I had to link some libs from the WiX SDK and build with a 32-bit compiler.

This seems ok. We can use a 32-bit installer everywhere.

The rustup installation routines will have to be refactored in such a way that status updates can be either reported to the console or via WcaProcessMessage.

Makes sense.

In order to get the self-install to work from a library we're going to have to do some refactoring of the project structure. Right now there are two top-level artifacts: the rustup library and the rustup-cli binary. The rustup-cli bin is both rustup-init and rustup - it changes behavior based on how its invoked. The windows installer though is going to need access to that binary via a library. So we're going to have to create a new library that include!s the entire rustup binary. I'm not entirely sure how to make this happen but there are going to be at least a few steps.

• To start with I'd say we just create a new project rustup-win-installer in src/. This project will depend on the main project. To start with it can just expect that the rustup binary exists in a known location, since getting a dependency to output a binary is not possible.
• The self_update module in rustup-cli will require significant refactoring:
  • The code needs to move into the library rustup
  • The console specific bits need to be lifted out and injected from rustup-cli
  • The routine for finding the rustup binary likewise needs to be lifted out and provided by injection, since the win installer will be getting it from an included binary, not from the running executable.

With those refactorings we can whip up a basic installer that presents no options but does put the stuff in the right place.

[brson]

(I can probably help with the self_update refactoring since it could get ugly).

[Diggsey]

To avoid duplication it's technically possible to load an executable as though it were a DLL and call functions from it - http://www.codeproject.com/Articles/1045674/Load-EXE-as-DLL-Mission-Possible there's probably a cleaner way though.

[Boddlnagg]

I skimmed over https://github.com/rust-lang-nursery/rustup.rs/blob/master/src/rustup-cli/self_update.rs and was thinking that maybe we could remove the complex Windows-only logic to remove the running exe, by always using the MSI for uninstall/update (installed MSIs are cached, so it will exist on the system). That requires that rustup has been installed using the MSI, but if that will be the only method on Windows (i.e., rustup-init will no longer exist), it should be fine. I don't know about a possible upgrade path for already existing installations that have never used the MSI, though.

Also some other routines, that have Windows-specific codepaths (such as updating the PATH) could be handled by MSI directly. WiX provides ways of doing this easily, since it's something that installers often do.

[retep998]

@Boddlnagg Just tell the user to delete their rustup installation and download and install using the MSI instead?

[Boddlnagg]

@retep998 That would be possible, of course, but will it be convenient enough?

@brson I don't quite understand these sentences:

To start with it can just expect that the rustup binary exists in a known location, since getting a dependency to output a binary is not possible. [...] The routine for finding the rustup binary likewise needs to be lifted out and provided by injection, since the win installer will be getting it from an included binary, not from the running executable.

Why are you talking about the rustup binary? I was expecting that the MSI just includes the custom action DLL (cdylib), which links everything in statically, so no binary would be included.
Update: Oh well, I think I got it now ... the rustup binary is the thing that will be installed, after all, so it must be included. But then it should be extracted by the MSI itself. Is the place where it should be put known in advance or does some Rust code need to run to determine that?

[brson]

I skimmed over https://github.com/rust-lang-nursery/rustup.rs/blob/master/src/rustup-cli/self_update.rs and was thinking that maybe we could remove the complex Windows-only logic to remove the running exe, by always using the MSI for uninstall/update (installed MSIs are cached, so it will exist on the system).

I think this is a reasonable goal to aim for, though perhaps we can get there incrementally. If it ends up just being a lot easier to implement then maybe we can jump straight to that model. The transition story can be worked out later - there's a lot of problems to solve just to get to the point where we have a working GUI installer.

Update: Oh well, I think I got it now ... the rustup binary is the thing that will be installed, after all, so it must be included. But then it should be extracted by the MSI itself. Is the place where it should be put known in advance or does some Rust code need to run to determine that?

It's not obvious to me that it should be extracted by the msi itself since there is other logic to installation than just extracting the binary, but if we can make it work that way then that's probably best. The logic for deciding the installation is in self_update::install_bins and utils::cargo_home but it's pretty simple - if CARGO_HOME is set put it there; if not put it in a pre-determined location. This logic will likely expand somewhat in the future.

The bulk of the GUI customization work we'll need to do is for configuring the global installation options and for installing/updating/uninstalling toolchains. If the basic work of putting the rustup bin in the right place can be done by the MSI system itself that seems good.

[brson]

@Boddlnagg if you find yourself needing to make changes to rustup self update to support the new uninstall then can I suggest doing it under a feature flag for now so we can continue producing rustup's with the current model while we iterate on the new installer?

[Boddlnagg]

@brson Sure, I will use a feature flag. But I won't be able to do much now until the end of next week, just as a heads-up. (Maybe someone else wants to chime in and already start with the refactorings? 😉 )

[brson]

I'll put it on my todo list to start refactoring the installer code, but suspect I won't get to it this week.

[Boddlnagg]

I just went through self_update.rs and tried to identify those parts that need to be shared by the MSI installer. To be able to reuse them, those parts will need to be moved from the executable into the library (and potentially refactored and adapted).

• do_pre_install_sanity_checks() (maybe offer to uninstall MSI-installed Rust automatically)
• cleanup_legacy()
• Creation of hardlinks, currently part of install_bins()
• maybe_install_rust()
• uninstall(), but it should not modify PATH, should not remove rustup.exe, but should remove the hardlinks (MSI will modify PATH, and delete rustup.exe and the containing folder)

[brson]

Thanks for doing the analysis!

[Boddlnagg]

I opened an initial WIP PR at #635.

[brson]

Thanks @Boddlnagg. Great progress.

[brson]

We've merged the first iteration, which puts the basic structure into place. Thanks @Boddlnagg!

It's even set up to build on appveyor, but there's a bit left to do. What I know of:

• The msi-ified rustup needs to build with the MSVC toolchain, and thus needs to be statically linked to the CRT so that the custom msi actions don't depend on an unavailable CRT dll. Figure out how to get static linkage for MSVCRT #660
• It looks to me like the CI still needs to be modified to generate a new rustup-msi.exe.sha256 file for upgrades
• appveyor.yml artifacts need to be adjusted for the msi case
• need a strategy for having old non-msi rustup upgrade to the msi rustup. Before we're ready to deploy we can probably put the code in place and have it activated through an environment variable, so that we can test without disrupting existing users.

@Boddlnagg can you give a braindump of other remaining work?

[Boddlnagg]

I had already prepared a table of steps that need to be executed in the final installer (this should probably also be placed somewhere in the code as documentation). I marked those that are already implemented (unfortunately checkboxes don't work within tables). Some steps also need a bit of discussion (e.g. uninstall previous MSI-installed Rust automatically? Is cleanup_legacy() still required?).

The biggest chunk, where I could really use some help, is "install default toolchain, show status/progress in UI". This requires a refactoring of the status reporting during toolchain installation.

Condition	Responsibility	Description
Upgrade	rustup.exe	rustup self upgrade checks if update is available and runs msiexec /i https://path/to/rustup.msi
Uninstall	rustup.exe	[DONE] rustup self uninstall fetches installed product code from registry and runs msiexec /x {<product-code>}
Install	MSI/WiX	Don't allow installation if using Windows < 7 (?)
Install	MSI/WiX	Check if Rust MSI is installed and uninstall automatically (show warning at least)?
Any (?)	MSI/WiX	[DONE] Installer runs RustupPrepare CA:
Install	RustupPrepare (CA) or MSI/WiX	Maybe run do_pre_install_sanity_checks?
Any (?)	RustupPrepare (CA)	Set EXISTS := rustup installed? (check if .multirust and/or .cargo\bin\rustup.exe exist)
Any (?)	RustupPrepare (CA)	Get installation path using utils::cargo_home() (currently using %USERPROFILE%\.rustup-test instead)
Install	MSI/WiX	Show UI for selecting custom installation options (Add to PATH? Default toolchain?)
Install	SetInstallOptions (CA/WiX)	[DONE] Pass Installation options to deferred CA
Upgrade	MSI/WiX	[DONE] Automatically run RemoveExistingProducts during upgrade
Uninstall/Upgrade	MSI/WiX	[DONE] Remove bin directory from PATH (automatically scheduled before RemoveFiles)
Uninstall/Upgrade	MSI/WiX	[DONE] RemoveFiles removes rustup.exe
Uninstall	MSI/WiX	[DONE] Run RustupUninstall CA (after RemoveFiles), only for true Uninstall (not for Upgrade)
Uninstall	RustupUninstall (CA)	Remove .cargo and .multirust, using utils::remove_dir (currently deleting .rustup-test instead)
Install/Upgrade	MSI/WiX	[DONE] InstallFiles extracts and installs rustup.exe
Install/Upgrade	MSI/WiX	[DONE] Run RustupInstall CA:
Install, if EXISTS	RustupInstall (CA)	Remove from PATH (old method, if pre-MSI version is installed)
Install, if EXISTS	RustupInstall (CA)	Maybe run cleanup_legacy()
Install/Upgrade	RustupInstall (CA)	[DONE] Create hardlinks for other bins
Install, if NOT EXISTS	RustupInstall (CA)	(If .multirust does not exist – does it matter if it does?) install default toolchain, show status/progress in UI
Install/Upgrade	MSI/WiX	[DONE] Add bin directory to PATH

CA is short for "Custom Action". "RustupPrepare" is currently called "RustupSetInstallLocation" but should be renamed.

[Boddlnagg]

Also, most of the rustup self update and rustup self uninstall tests won't work anymore, because rustup only runs msiexec and exits. The tests can only check the results once msiexec has finished. So maybe new tests need to be written that explicitly target the MSI (if it is even possible to test MSIs).

[cyplo]

Great progress, thank you !

[Boddlnagg]

I tried to improve the error handling inside the Custom Actions, but found that Custom Actions always should have a corresponding Rollback Custom Action that rolls back the changes done by the normal CA whenever one of the installation steps fails. There is an option to disable rollback completely, but this doesn't seem to work reliably (in my tests, a rollback was performed regardless). All the possible scenarios (successfull/failed installation/upgrade/uninstall) need to be tested thoroughly, so to do it right, everyone recommends to use Custom Actions as sparingly as possible, and instead rely on proven built-in actions. We won't be able to get rid of all CAs (e.g. to create hardlinks and do the toolchain install), but we should wherever we can, even though this means less shared code between platforms. I don't know how important it is that rollback works correctly, but MSI users usually expect it to work (on a failed installation the installer even explicitly says "Your system has not been modified").

[brson]

@Boddlnagg It seems reasonable to me that as much of the self_update module as possible is re-written as MSI-isms. Toolchain installation is probably not worth the duplication, and toolchain installation does have rollback-on-fail behavior.

[Boddlnagg]

@brson We could get rid of custom actions completely by creating copies of rustup.exe instead of hardlinks (see DuplicateFiles Action), using RemoveFolderEx for the cleanup and delaying the default toolchain installation until the first start of rustup.exe. The user-selected default would be written to a file or the registry and a message could be shown at the end of the installation process like "When you run rustup for the first time, the default toolchain will be downloaded and installed". That would also eliminate the need of refactoring rustup output and of figuring out the MSVC static linking. Don't know if that would be a satisfying alternative.

[retep998]

I would be okay with the rustup installer not actually installing any toolchains and just installing rustup itself.

[brson]

@Boddlnagg That sounds like a good starting point, though I think ultimately we do want to be in a place where the GUI can be used for all installation tasks - that's what windows users expect. Losing the hardlinks is kind of a drag though. Is it desirable to not have custom actions?

[Boddlnagg]

@brson It is desirable to not have custom actions, because of rollback: When Windows Installer installs a file and would have to overwrite an existing file, it first creates a backup of that file at some hidden location, then installs the new file, then deletes the backup when the installation has completed. We would have to implement this logic ourselves for the "Create Hardlink" custom action in order to support rollback correctly, and I don't know if that's worth it, if we can use machinery that's already there. Do you expect that the number of hardlinks will grow in the future?

A GUI for all installation tasks (i.e. for rustup itself, not just for the initial setup) should not be the goal for the MSI. MSI is just not built for downloading additional data and running arbitrary application logic, it is meant for installing files and changing system settings. Even WiX uses custom bootstraping applications for things that go beyond that. One could build on top of the Windows Installer infrastructure to install toolchains, but that would require to repackage every installable package as Windows Installer modules, and it would still require an additional GUI for managing everything (because there seems to be no way to dynamically update the features/modules that are available in an installer) and the Windows version would be basically totally different from all other platforms, which constitutes a heavy maintenance burden.

One could write such a GUI, but not using WiX/MSI ... however, I think a graphical user interface for rustup should rather be part of an IDE. So users who don't want to use these graphical tools can still use the command line as on any other platform, and IDEs can provide toolchain management by wrapping rustup and showing whatever GUI fits their needs. Users who don't use an IDE probably are fine without a GUI, and that usecase should be consistent with other platforms. It would be nice if rustup could make it easy for other applications to wrap it (especially the download progress updates are probably a bit hard to deal with if there's no alternate "machine readable" output format).

[dataf3l]

NSIS Anyone?

[brson]

@Boddlnagg ok, thanks for the explanation. If it's not appropriate to have a component within the MSI then we shouldn't.

As a starting point just getting a GUI replacement for rustup-init is great. Though rustup-init does allow component selection. Would you see us not having any kind of of screen in the MSI that allows for selecting components?

I still feel like for the best experience there should be a GUI that allows one to do component selection at install / update time. Take the VS installer for example. That's the kind of experience I think windows users expect.

I do understand the notion that IDES should handle rustup installation, and agree. I hope we get to the point in Rust where we can say "Install VS code, install the Rust plugin", and the plugin deals with rustup however is most appropriate for VS code. But there are a lot of IDEs, and they're not all going to reach that level of integration.

[Boddlnagg]

It is definitely possible to have a screen to select components, but I think (at least I didn't find a way to do it – though I would not consider myself an expert with MSI) they need to be hardcoded into the MSI, so can't be updated on the fly when the installer is run. This is what I meant when I said that dynamic updates to features are not possible. This makes it unsuitable to e.g. have every single Rust nightly release as selectable MSI component.

[Boddlnagg]

@brson What exactly is the definition of a "component" in this context? How dynamic is the set of selectable components?

[workingjubilee]

@rustbot label: +O-windows

[Keavon]

Rust is very developer-friendly with its tooling and most areas of the experience. But having to use a console-based installer is really not a great first impression. It also doesn't make the MSVC requirement very clear and it's easy to skip by accident. A GUI could help clarify that. This would bring Rust's installation process more on par with other languages and dev tooling I'm used to. I can't think of any other programming languages that require a command line installation process.