-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rpm-ostree/deploy: downgrade logic mistakenly blocks some upgrades #481
Comments
I'm investigating this. I think it's an issue in libostree's handling of downgrade protection in combination with how rpm-ostree handles So this might have broken automatic updates. An easy manual workaround which still respects barriers is something like:
|
We also need to expand upgrade testing to include older revisions than just the previous release. That would've caught this. |
For reference, the downgrade-blocking logic is active since Zincati 0.0.7: https://github.com/coreos/zincati/releases/tag/0.0.7. We introduced a barrier on |
This should fix itself, though, right? Once a node is permitted to update to 31.20200505.2.1, won't it do so successfully? |
Yeah, I should be more precise. I think this breaks automatic updates for nodes sitting before an update barrier and there are releases past the barrier. Since as Luca mentioned, we only had one barrier so far, and downgrade protection wasn't on back then, this is OK for now. But we need to resolve this before creating any new barriers (which we're contemplating doing for the f32 update: #480). |
In some cases it may not fix itself. As per Zincati state-machine, once an update is detected the client keeps trying staging+finalizing it. The machine in the original report is stuck in such state, even if now the newest rollout is available:
|
IIUC the way a client would get into a "stuck forever" state is by attempting an upgrade to
Any auto updating nodes should have already been on |
Hmm, I wonder if it should go back to |
clarification/update:
|
Both libostree and rpm-ostree support downgrade protection. But what that means is different between the two. For libostree, downgrade protection means not fetching any commit which is older than what the current ref is pointing at. For rpm-ostree, it means not *deploying* any commit which is older than what the current *deployment* is on. These two are mostly the same most of the time, but can differ. For example, on a remote ref which has commits A -> B -> C, where the client is sitting on a deployment from A, downgrade protection should not prevent the client from upgrading to B even if there is a newer commit C. Since there is no hard relation enforced between what the state of the OSTree ref is locally and deployments (e.g. we fully support users manually doing `ostree pull`), it doesn't make sense to compare against the tip of the ref. Instead, use the new `timestamp-check-from-rev` to tell libostree to compare against our base revision, which is what we care about. Closes: coreos/fedora-coreos-tracker#481
The way `timestamp-check` works might be too restrictive in some situations. Essentially, we need to support the case where users want to pull an older commit than the current tip, but while still guaranteeing that it is newer than some even older commit. This will be used in Fedora CoreOS. For more information see: coreos/rpm-ostree#2094 coreos/fedora-coreos-tracker#481
This is fixed with ostreedev/ostree#2099 and coreos/rpm-ostree#2094! |
A tentative enhancement on Zincati side is at coreos/zincati#286. |
The way `timestamp-check` works might be too restrictive in some situations. Essentially, we need to support the case where users want to pull an older commit than the current tip, but while still guaranteeing that it is newer than some even older commit. This will be used in Fedora CoreOS. For more information see: coreos/rpm-ostree#2094 coreos/fedora-coreos-tracker#481
Both libostree and rpm-ostree support downgrade protection. But what that means is different between the two. For libostree, downgrade protection means not fetching any commit which is older than what the current ref is pointing at. For rpm-ostree, it means not *deploying* any commit which is older than what the current *deployment* is on. These two are mostly the same most of the time, but can differ. For example, on a remote ref which has commits A -> B -> C, where the client is sitting on a deployment from A, downgrade protection should not prevent the client from upgrading to B even if there is a newer commit C. Since there is no hard relation enforced between what the state of the OSTree ref is locally and deployments (e.g. we fully support users manually doing `ostree pull`), it doesn't make sense to compare against the tip of the ref. Instead, use the new `timestamp-check-from-rev` to tell libostree to compare against our base revision, which is what we care about. Closes: coreos/fedora-coreos-tracker#481
Both libostree and rpm-ostree support downgrade protection. But what that means is different between the two. For libostree, downgrade protection means not fetching any commit which is older than what the current ref is pointing at. For rpm-ostree, it means not *deploying* any commit which is older than what the current *deployment* is on. These two are mostly the same most of the time, but can differ. For example, on a remote ref which has commits A -> B -> C, where the client is sitting on a deployment from A, downgrade protection should not prevent the client from upgrading to B even if there is a newer commit C. Since there is no hard relation enforced between what the state of the OSTree ref is locally and deployments (e.g. we fully support users manually doing `ostree pull`), it doesn't make sense to compare against the tip of the ref. Instead, use the new `timestamp-check-from-rev` to tell libostree to compare against our base revision, which is what we care about. Closes: coreos/fedora-coreos-tracker#481
Both libostree and rpm-ostree support downgrade protection. But what that means is different between the two. For libostree, downgrade protection means not fetching any commit which is older than what the current ref is pointing at. For rpm-ostree, it means not *deploying* any commit which is older than what the current *deployment* is on. These two are mostly the same most of the time, but can differ. For example, on a remote ref which has commits A -> B -> C, where the client is sitting on a deployment from A, downgrade protection should not prevent the client from upgrading to B even if there is a newer commit C. Since there is no hard relation enforced between what the state of the OSTree ref is locally and deployments (e.g. we fully support users manually doing `ostree pull`), it doesn't make sense to compare against the tip of the ref. Instead, use the new `timestamp-check-from-rev` to tell libostree to compare against our base revision, which is what we care about. Closes: coreos/fedora-coreos-tracker#481
Thanks @jlebon and @lucab for jumping on this problem and doing a root cause so quickly this morning (EDT) from the limited information I had available. Now that we have a bead on fixing this problem for future deployments I'd like to brainstorm if there is any possible way we can fix it without human intervention for existing systems who are currently "stuck" in the "attempting to upgrade" state. One idea I have is to:
This would mean for a period of time we would no longer fail the "downgrade protection" check. It looks like zincati retries every 5 minutes so 30 minutes should be plenty I would think. In order to not have a long period where we aren't serving update edges via cincinnati we could spread out this across several days or weeks (i.e. just do a few per day or something). |
We discussed this on IRC, but essentially: this won't work around the downgrade protection check because those nodes would've already fetched the latest ref and that's what downgrade protection checks against. So looking at the last 4
So there have been two rollouts so far where Zincati 0.0.9 (which has downgrade protection) was used. But since nodes need to be at N-2 or older to hit this, I think the only time window during which a node could enter the Zincati deadend state are those started at 31.20200323.3.2 which witnessed the 31.20200420.3.0 rollout and were given an edge to 31.20200407.3.0. That rollout lasted 48h, during which it became progressively less likely that you'd be handed out that edge. Looking at It'd be good to have some way to measure how many nodes are stuck there. The good news is that for That said, we shouldn't add any more rollouts without ostreedev/ostree#2099 and coreos/rpm-ostree#2094 (and/or coreos/zincati#286) in so we don't increase further the number of starting nodes that are susceptible to this. |
As a mild mitigation for coreos/fedora-coreos-tracker#481, let's remove the previous rollout from the graph so that no more nodes get into that state.
This is a short-term hack until we can depend on the new `timestamp-check-from-rev` from ostree: ostreedev/ostree#2099 That way, we still get downgrade protection, but wrt the checked out deployment, not the local ref. For more information, see #2094 coreos/fedora-coreos-tracker#481
New rpm-ostree with fix: coreos/fedora-coreos-config#401 |
@jlebon correct me if I'm wrong here but during our video meeting on Friday I think we looked at the code and it turns out we do think #481 (comment) would work because when fetching the ref we don't consider downgrade protection. I don't know if it's something we'll do but it's an option. |
@lucab - @jlebon and I chatted more towards the end of the day on Friday. We had the option of backporting just the fix for rpm-ostree into a testing release to do today (Monday) and then later in the week doing a stable release based on that testing release. We hit a complication because the rpm-ostree status in dist-git had moved past the rpm-ostree version that was already in FCOS stable/testing; we did a snapshot build recently for COSA. This means we'd have to make a backport branch in order to do the backport of just the fix for rpm-ostree, which we could easily do, but... Rather than doing all of that, since it was late on Friday, we decided to just move ahead with the newly released version of rpm-ostree (including the fix and new features) into testing-devel and then forge ahead with the round of releases for this week as planned. This means this week's testing release will contain the new release of rpm-ostree, but the stable release will not because we want to soak test the new release of rpm-ostree like we normally do. We think this is OK because we know how to mitigate this problem server side by removing the "to |
This landed upstream in rpm-ostree and a new rpm-ostree release was done. The fix for this went into testing stream release |
The fix for this went into stable stream release |
For anybody that has a machine which is blocked in this state, it is possible to manually allow downgrades for the current boot in order to get back into the normal auto-updates flow:
After the first successful upgrade, the configuration fragment will be automatically cleaned (with the rest of /run). |
Older FCOS will complain about an upgrade target being 'chronologically older than current' This is documented in coreos/fedora-coreos-tracker#481 We'll workaround the problem here in a config dropin.
Older FCOS will complain about an upgrade target being 'chronologically older than current' This is documented in coreos/fedora-coreos-tracker#481 We'll workaround the problem here in a config dropin.
Older FCOS will complain about an upgrade target being 'chronologically older than current' This is documented in coreos/fedora-coreos-tracker#481 We'll workaround the problem here in a config dropin.
Older FCOS will complain about an upgrade target being 'chronologically older than current' This is documented in coreos/fedora-coreos-tracker#481 We'll workaround the problem here in a config dropin.
Older FCOS will complain about an upgrade target being 'chronologically older than current' This is documented in coreos/fedora-coreos-tracker#481 We'll workaround the problem here in a config dropin.
Currently, on the
testing
channel there are two rollouts:31.20200505.2.0
at 100%31.20200505.2.1
at 0% (which will start in a few minutes)Right now, creating a new
testing
machine using an older OS image (31.20200420.2.0
in the example above), results in the following error in Zincati journal:Cross-referencing releases with their revisions, it seems that:
rpm-ostree status
andZincati
agree that31.20200420.2.0
is the booted deployment31.20200420.2.0
->31.20200505.2.0
rpm-ostreed
is detecting and blocking a downgrade31.20200505.2.1
->31.20200505.2.0
The text was updated successfully, but these errors were encountered: