Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated the HA Upgrade Instructions #20206

Merged
merged 7 commits into from
Apr 28, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 3 additions & 84 deletions website/content/docs/upgrading/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,95 +73,14 @@ upgrade notes.

## HA Installations

!> **Important:** Note that these instructions are not relevant if you're on a
version of Vault greater than or equal to 1.11 and you have Autopilot enabled.
If so, you should let Autopilot do the upgrade for you, as that's easier and
The recommended upgrade procedure depends on the version of Vault you're currently on and the storage backend of Vault. If you're currently running on Vault 1.11 or later with Integrated Storage and you have Autopilot enabled, you should let Autopilot do the upgrade for you, as that's easier and
less prone to human error. Please refer to our [automated
upgrades](/vault/docs/concepts/integrated-storage/autopilot#automated-upgrades)
documentation for information on this feature and our
upgrades](/vault/docs/concepts/integrated-storage/autopilot#automated-upgrades) documentation for information on this feature and our
[Automate Upgrades with Vault
Enterprise](/vault/tutorials/raft/raft-upgrade-automation)
tutorial for more details.

This is our recommended upgrade procedure if you're on a version of Vault before
1.11, or you've chosen not to use Autopilot. However, you should consider how to
apply these steps to your particular setup since HA setups can differ on whether
a load balancer is in use, what addresses clients are being given to connect to
Vault (standby + leader, leader-only, or discovered via service discovery), etc.
If you're currently on a version of Vault before 1.11, or you've chosen to opt-out the Autopilot automated upgrade features when running Vault after 1.11 with Integrated Storage, or if you are running Vault with other storage backend such as Consul. Please refer to our [Vault HA upgrades Pre 1.11/Without Autopilot Upgrade Automation](/vault/docs/upgrading/vault-ha-upgrade) documentation for more details. Please note that this upgrade procedure also applies if you are upgrading Vault from pre 1.11 to post 1.11.

Whatever method you use, you should ensure that you never fail over from a
newer version of Vault to an older version. Our suggested procedure is designed
to prevent this.

Please note that Vault does not support true zero-downtime upgrades, but with
proper upgrade procedure the downtime should be very short (a few hundred
milliseconds to a second depending on how the speed of access to the storage
backend).

Perform these steps on each standby:

1. Properly shut down Vault on the standby node via `SIGINT` or `SIGTERM`
2. Replace the Vault binary with the new version; ensure that `mlock()` capability is added to the new binary with [setcap](/vault/docs/configuration#disable_mlock)
3. Start the standby node
4. Unseal the standby node
5. Verify `vault status` shows correct Version and HA Mode is `standby`
6. Review the node's logs to ensure successful startup and unseal

At this point all standby nodes will be upgraded and ready to take over. The
upgrade will not be complete until one of the upgraded standby nodes takes over
active duty. To do this:

1. Properly shut down the remaining (active) node

~> **Note:** It is important that you shut the node down properly.
This will perform a step-down and release the HA lock, allowing a standby
node to take over with a very short delay.
If you kill Vault without letting it release the lock, a standby node will
not be able to take over until the lock's timeout period has expired. This
is backend-specific but could be ten seconds or more.

2. Replace the Vault binary with the new version; ensure that `mlock()` capability is added to the new binary with [setcap](/vault/docs/configuration#disable_mlock)
3. Start the node
4. Unseal the node
5. Verify `vault status` shows correct Version and HA Mode is `standby`
6. Review the node's logs to ensure successful startup and unseal

Internal upgrade tasks will happen after one of the upgraded standby nodes
takes over active duty.

Be sure to also read and follow any instructions in the version-specific
upgrade notes.

## Replication Installations

-> **Note:** Prior to any upgrade, be sure to also read and follow any instructions in the version-specific
upgrade notes which are found in the navigation menu for this documentation.

Upgrading installations of Vault which participate in [Enterprise Replication](/vault/docs/enterprise/replication) requires the following basic order of operations:

- **Upgrade the replication secondary instances first** using appropriate
guidance from the previous sections depending on whether each secondary
instance is Non-HA or HA
- Verify functionality of each secondary instance after upgrading
- When satisfied with functionality of upgraded secondary instances, upgrade
the primary instance

~> **Note:** It is not safe to replicate from a newer version of Vault to an
older version. When upgrading replicated clusters, ensure that upstream clusters
are always on older versions of Vault than downstream clusters.

Here is an example of upgrading four Vault replicated Vault clusters:

![Upgrading multiple replicated clusters](/img/vault-replication-upgrade.png)

In the above scenario, the ideal upgrade procedure would be as follows,
verifying functionality after each cluster upgrade.

1. Upgrade Clusters B and D, using the HA upgrade process above. These clusters
have no downstream clusters, so they should be upgraded first, but the
ordering of B vs D does not matter.
2. Upgrade Cluster C, which now has an upgraded downstream cluster (Cluster D).
Because Cluster C is a cluster, it should also use the HA upgrade process.
3. Finally, upgrade Cluster A. All clusters downstream of A will already be
upgraded. It should be upgraded last, as it is a Performance Primary and a DR
Primary.
141 changes: 141 additions & 0 deletions website/content/docs/upgrading/vault-ha-upgrade.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
---
layout: docs
page_title: Vault HA upgrades without Autopilot Upgrade Automation (Pre 1.11)
description: |-
Upgrade instructions for Vault HA Pre 1.11 or Vault without autopilot upgrade automation being enabled. Be sure to read the Upgrading-Vault Guides as well.
---

# Vault HA upgrades without Autopilot Upgrade Automation (Pre 1.11)

This is our recommended upgrade procedure if **one** of the following applies:

- Running Vault version earlier than 1.11
- Opt-out the [Autopilot automated upgrade]((/vault/docs/concepts/integrated-storage/autopilot#automated-upgrades)) features with Vault 1.11 or later
- Running Vault with external storage backend such as Consul

You should consider how to apply the steps described in this document to your
particular setup since HA setups can differ on whether a load balancer is in
use, what addresses clients are being given to connect to Vault (standby +
leader, leader-only, or discovered via service discovery), etc.

If you are running on Vault 1.11+ with Integrated Storage and wish to enable the
Autopilot upgrade automation features, read to the [automated
upgrades](/vault/docs/concepts/integrated-storage/autopilot#automated-upgrades)
documentation for details and the [Automate Upgrades with Vault
Enterprise](/vault/tutorials/raft/raft-upgrade-automation) tutorial for
additional guidance.


## HA Installations

Regardless of the method you use, do not fail over from a newer version of Vault
to an older version. Our suggested procedure is designed to prevent this.

Please note that Vault does not support true zero-downtime upgrades, but with
proper upgrade procedure the downtime should be very short (a few hundred
milliseconds to a second depending on how the speed of access to the storage
backend).

<Warning title="Important">

If you are currently running on Vault 1.11+ with Integrated Storage and have
chosen to opt-out the Autopilot automated upgrade features, please disable the
default automated upgrade migrations feature of the Vault. To disable this
feature, follow the [Automate Upgrades with Vault Enterprise Autopilot
configuration](/vault/tutorials/raft/raft-upgrade-automation#autopilot-configuration)
tutorial for more details. Without disabling this feature, you may run into Lost
Quorum issue as described in the [Quorum lost while upgrading the vault from
1.11.0 to later version of
it](https://support.hashicorp.com/hc/en-us/articles/7122445204755-Quorum-lost-while-upgrading-the-vault-from-1-11-0-to-later-version-of-it)
article.

</Warning>

Perform these steps on each standby:

1. Properly shut down Vault on the standby node via `SIGINT` or `SIGTERM`
2. Replace the Vault binary with the new version; ensure that `mlock()`
capability is added to the new binary with
[setcap](/vault/docs/configuration#disable_mlock)
3. Start the standby node
4. Unseal the standby node
5. Verify `vault status` shows correct Version and HA Mode is `standby`
6. Review the node's logs to ensure successful startup and unseal

At this point all standby nodes are upgraded and ready to take over. The
upgrade will not complete until one of the upgraded standby nodes takes over
active duty.

To complete the cluster upgrade:

1. Properly shut down the remaining (active) node

<Note>

It is important that you shut the node down properly.
This will perform a step-down and release the HA lock, allowing a standby
node to take over with a very short delay.
If you kill Vault without letting it release the lock, a standby node will
not be able to take over until the lock's timeout period has expired. This
is backend-specific but could be ten seconds or more.

</Note>

2. Replace the Vault binary with the new version; ensure that `mlock()`
capability is added to the new binary with
[setcap](/vault/docs/configuration#disable_mlock)
3. Start the node
4. Unseal the node
5. Verify `vault status` shows correct Version and HA Mode is `standby`
6. Review the node's logs to ensure successful startup and unseal

Internal upgrade tasks will happen after one of the upgraded standby nodes
takes over active duty.

Be sure to also read and follow any instructions in the version-specific
upgrade notes.

## Enterprise Replication Installations

<Note>

Prior to any upgrade, be sure to also read and follow any instructions in the
version-specific upgrade notes which are found in the navigation menu for this
documentation.

</Note>

Upgrading Vault Enterprise clusters which participate in [Enterprise
Replication](/vault/docs/enterprise/replication) requires the following basic
order of operations:

- **Upgrade the replication secondary instances first** using appropriate
guidance from the previous sections depending on whether each secondary
instance is non-HA or HA
- Verify functionality of each secondary instance after upgrading
- When satisfied with functionality of upgraded secondary instances, upgrade
the primary instance

<Note>

It is not safe to replicate from a newer version of Vault to an older version.
When upgrading replicated clusters, ensure that upstream clusters are always on
older versions of Vault than downstream clusters.

</Note>

Here is an example of upgrading four Vault replicated Vault clusters:

![Upgrading multiple replicated clusters](/img/vault-replication-upgrade.png)

In the above scenario, the ideal upgrade procedure would be as follows,
verifying functionality after each cluster upgrade.

1. Upgrade Clusters B and D, using the HA upgrade process above. These clusters
have no downstream clusters, so they should be upgraded first, but the
ordering of B vs D does not matter.
2. Upgrade Cluster C, which now has an upgraded downstream cluster (Cluster D).
Because Cluster C is a cluster, it should also use the HA upgrade process.
3. Finally, upgrade Cluster A. All clusters downstream of A will already be
upgraded. It should be upgraded last, as it is a Performance Primary and a DR
Primary.
4 changes: 4 additions & 0 deletions website/data/docs-nav-data.json
Original file line number Diff line number Diff line change
Expand Up @@ -1810,6 +1810,10 @@
"title": "Overview",
"path": "upgrading"
},
{
"title": "Vault HA upgrades",
"path": "upgrading/vault-ha-upgrade"
},
{
"title": "Upgrade Plugins",
"path": "upgrading/plugins"
Expand Down