Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have a dedicated lockfile updater for testing-devel and next-devel #293

Closed
jlebon opened this issue Oct 8, 2019 · 6 comments
Closed

Have a dedicated lockfile updater for testing-devel and next-devel #293

jlebon opened this issue Oct 8, 2019 · 6 comments
Labels
jira for syncing to jira

Comments

@jlebon
Copy link
Member

jlebon commented Oct 8, 2019

In #290, we discussed that the current model of inheriting the lockfile from bodhi-updates is limiting because it means that any regression causes testing-devel to stop consuming all updates. Instead, we should have the ability to freeze troublesome packages and keep consuming everything else.

One mentioned solution was to instead have a dedicated lockfile bumper. At which point, bodhi-updates really just becomes a way to track FCOS - overrides.

And of course, we really want the lockfile bump workflow to be CI gated. But having daily/multiple PRs per day would probably get noisy and tiring fast. So one middle ground here is something like this:

  • check out the latest testing-devel
  • run cosa fetch --update-lockfile
  • push updated lockfile to a staging branch, e.g. testing-devel-staging
  • wait for CI to pass
  • if it passes, then push into testing-devel (obviously being careful to handle races)
  • if it fails, open a PR

@dustymabe mentioned also we could make it always open a PR under certain conditions, such as if some packages are added or dropped.
@jlebon
Copy link
Member Author

jlebon commented Oct 8, 2019

Also need to watch for coreos-koji-tagger integration: coreos/fedora-coreos-releng-automation#32

@dustymabe
Copy link
Member

  • push updated lockfile to a staging branch, e.g. testing-devel-staging

  • wait for CI to pass

A few questions:

  • IIUC this implies that we have CI set up on the testing-devel-staging branch so we'd need a separate buildconfig set up for it in the fedora-coreos-pipeline repo?
  • could we possibly just use a staging repo so we don't end up with branches in our config repo that may confuse people?

@jlebon
Copy link
Member Author

jlebon commented Oct 8, 2019

IIUC this implies that we have CI set up on the testing-devel-staging branch so we'd need a separate buildconfig set up for it in the fedora-coreos-pipeline repo?

This would be the Jenkins CI which already runs on PRs right now, not fedora-coreos-pipeline. They're related though, because we'll want to share much more between those to make gating more meaningful.

could we possibly just use a staging repo so we don't end up with branches in our config repo that may confuse people?

Hmm, maybe. Though it'd only be one branch per "locked" stream, so e.g. testing-devel-staging and next-devel-staging. Seems OK? Definitely wouldn't be hard to e.g. make the bot use its fork though. A minor advantage of having it in the same repo means that it's easier to fetch and build locally when debugging issues.

jlebon added a commit to jlebon/fedora-coreos-releng-automation that referenced this issue Oct 8, 2019
@jlebon
WIP: config-bot: add bump-lockfiles functionality
d5bc826 
This is meant to address a smarter way to bump lockfiles.
See: coreos/fedora-coreos-tracker#293

Only supports `git push` for now. Working on making that smarter.
jlebon added a commit to jlebon/fedora-coreos-releng-automation that referenced this issue Oct 8, 2019
@jlebon
WIP: config-bot: add bump-lockfiles functionality
addc95c 
This is meant to address a smarter way to bump lockfiles.
See: coreos/fedora-coreos-tracker#293

Only supports `git push` for now. Working on making that smarter.
@jlebon jlebon mentioned this issue Oct 8, 2019
Closed
jlebon added a commit to jlebon/fedora-coreos-releng-automation that referenced this issue Oct 8, 2019
@jlebon
WIP: config-bot: add bump-lockfiles functionality
7abcd65 
This is meant to address a smarter way to bump lockfiles.
See: coreos/fedora-coreos-tracker#293

Only supports `git push` for now. Working on making that smarter.
@dustymabe
Copy link
Member

Definitely wouldn't be hard to e.g. make the bot use its fork though.

+1 - I think I'd much prefer using a fork!

jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Oct 9, 2019
@jlebon
Pin podman due to systemctl preset-all regression
33a1e05 
coreos/fedora-coreos-tracker#290

The fix for this was merged, but it'll be some time before it makes it
to the stable repos. This is a temporary hack to get updates going again
into FCOS. Working on a cleaner approach to this in:

coreos/fedora-coreos-tracker#293
@jlebon jlebon mentioned this issue Oct 9, 2019
Merged
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Oct 9, 2019
@jlebon
Pin podman due to systemctl preset-all regression
de779f6 
coreos/fedora-coreos-tracker#290

The fix for this was merged, but it'll be some time before it makes it
to the stable repos. This is a temporary hack to get updates going again
into FCOS. Working on a cleaner approach to this in:

coreos/fedora-coreos-tracker#293
@jlebon jlebon added the jira for syncing to jira label Oct 11, 2019
@jlebon jlebon mentioned this issue Oct 22, 2019
Merged
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Nov 8, 2019
@jlebon
manifest: temporarily add fedora-coreos-pool repo
e2671ad 
As discussed in coreos#104,
we don't actually want the bodhi-updates stream to pull in newer
packages from the pool.

That said, bodhi-updates also current acts as our lockfile updater in
testing-devel, so it's crucial that it keeps working. We're working to
decouple that, see:

coreos/fedora-coreos-tracker#293
coreos/fedora-coreos-releng-automation#48

But for now, the updates must flow...
@jlebon jlebon mentioned this issue Nov 8, 2019
Merged
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Nov 8, 2019
@jlebon
manifest: temporarily add fedora-coreos-pool repo
0d577b0 
As discussed in #104,
we don't actually want the bodhi-updates stream to pull in newer
packages from the pool.

That said, bodhi-updates also current acts as our lockfile updater in
testing-devel, so it's crucial that it keeps working. We're working to
decouple that, see:

coreos/fedora-coreos-tracker#293
coreos/fedora-coreos-releng-automation#48

But for now, the updates must flow...
@jlebon jlebon mentioned this issue Mar 12, 2020
Merged
@jlebon jlebon mentioned this issue Mar 20, 2020
Closed
5 tasks
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Apr 7, 2020
@jlebon
manifest: drop fedora-coreos-pool repo
a97049c 
So... one thing I just realized is that now that we have a bunch of f32
packages in the pool, we really can't have `bodhi-updates` picking up
packages from there unlocked.

So this is sort of forcing our hand to get
coreos/fedora-coreos-tracker#293 fixed very
soon so that we can unblock package bumps again.

As a first step and to prevent `testing-devel` ending up with a bunch of
f32 packages, let's remove the pool from the list of repos here.
@jlebon jlebon mentioned this issue Apr 7, 2020
Merged
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Apr 7, 2020
@jlebon
manifest: drop fedora-coreos-pool repo
777c3f1 
So... one thing I just realized is that now that we have a bunch of f32
packages in the pool, we really can't have `bodhi-updates` picking up
packages from there unlocked.

So this is sort of forcing our hand to get
coreos/fedora-coreos-tracker#293 fixed very
soon so that we can unblock package bumps again.

As a first step and to prevent `testing-devel` ending up with a bunch of
f32 packages, let's remove the pool from the list of repos here.
@jlebon
Copy link
Member Author

jlebon commented Apr 17, 2020

OK the relevant patches here are:

coreos/rpm-ostree#2058
coreos/fedora-coreos-config#355

With those, we'll be able to make the updater code in coreos/fedora-coreos-releng-automation#48 simply just do cosa fetch --update-lockfile the same way a developer would.

@jlebon jlebon mentioned this issue Apr 17, 2020
Merged
@jlebon jlebon changed the title Have a dedicated lockfile updater for testing-devel Have a dedicated lockfile updater for testing-devel and next-devel Apr 20, 2020
jlebon added a commit to jlebon/coreos-assembler that referenced this issue Apr 22, 2020
@jlebon
cmd-fetch: include overrides when updating lockfile
9647387 
I'm working on a lockfile updater[1], and I'd like it to just be able to
do `cosa fetch --update-lockfile` the same way a human would. This
normally would run `rpm-ostree compose tree` without passing any
lockfiles and blocking out the pool so that all packages come from the
base repo.

Sadly, that doesn't currently work for FCOS right now we have an
override for crypto-policies from f32 so that we don't pull in Python,
and also have an `exclude-packages` for Python. Which means that without
the f32 crypto-policies, we can't get a depsolve from just the base f31
repos. (This is the same issue that killed bodhi-updates[2].)

As a short-term hack, just include the overrides for now. This dillutes
the meaning of a "base" lockfile of course, because it will now will
include the same packages as the overrides. We can drop this hack once
we move to f32, though I have some ideas too on how to solve this more
correctly.

[1] coreos/fedora-coreos-tracker#293
[2] coreos/fedora-coreos-config#335 (comment)
jlebon added a commit to jlebon/coreos-assembler that referenced this issue Apr 22, 2020
@jlebon
cmd-fetch: include overrides when updating lockfile
1ace3c0 
I'm working on a lockfile updater[1], and I'd like it to just be able to
do `cosa fetch --update-lockfile` the same way a human would. This
normally would run `rpm-ostree compose tree` without passing any
lockfiles and blocking out the pool so that all packages come from the
base repo.

Sadly, that doesn't currently work for FCOS right now because we have an
override for crypto-policies from f32 so that we don't pull in Python.
And because we also have an `exclude-packages` for Python, without the
f32 crypto-policies, we can't get a depsolve from just the base f31
repos. (This is the same issue that killed bodhi-updates[2].)

As a short-term hack, just include the overrides for now. This dillutes
the meaning of a "base" lockfile of course, because it will now will
include the same packages as the overrides. We can drop this hack once
we move to f32, though I have some ideas too on how to solve this more
correctly.

[1] coreos/fedora-coreos-tracker#293
[2] coreos/fedora-coreos-config#335 (comment)
jlebon added a commit to jlebon/coreos-assembler that referenced this issue Apr 22, 2020
@jlebon
cmd-fetch: include overrides when updating lockfile
ffe7d21 
I'm working on a lockfile updater[1], and I'd like it to just be able to
do `cosa fetch --update-lockfile` the same way a human would. This
normally would run `rpm-ostree compose tree` without passing any
lockfiles and blocking out the pool so that all packages come from the
base repo.

Sadly, that doesn't currently work for FCOS right now because we have an
override for crypto-policies from f32 so that we don't pull in Python.
And because we also have an `exclude-packages` for Python, without the
f32 crypto-policies, we can't get a depsolve from just the base f31
repos. (This is the same issue that killed bodhi-updates[2].)

As a short-term hack, just include the overrides for now. This dillutes
the meaning of a "base" lockfile of course, because it will now include
the packages from the overrides. I don't think this really matters for
now though (it does make the checking for whether overrides are still
needed harder, but I'd like to automate dropping overrides eventually
too).

Anyway, we can drop this hack once we move to f32, though I have some
ideas too on how to solve this more correctly.

[1] coreos/fedora-coreos-tracker#293
[2] coreos/fedora-coreos-config#335 (comment)
@jlebon jlebon mentioned this issue Apr 22, 2020
Merged
openshift-merge-robot pushed a commit to coreos/coreos-assembler that referenced this issue Apr 23, 2020
@jlebon @openshift-merge-robot
cmd-fetch: include overrides when updating lockfile
3a853e1 
I'm working on a lockfile updater[1], and I'd like it to just be able to
do `cosa fetch --update-lockfile` the same way a human would. This
normally would run `rpm-ostree compose tree` without passing any
lockfiles and blocking out the pool so that all packages come from the
base repo.

Sadly, that doesn't currently work for FCOS right now because we have an
override for crypto-policies from f32 so that we don't pull in Python.
And because we also have an `exclude-packages` for Python, without the
f32 crypto-policies, we can't get a depsolve from just the base f31
repos. (This is the same issue that killed bodhi-updates[2].)

As a short-term hack, just include the overrides for now. This dillutes
the meaning of a "base" lockfile of course, because it will now include
the packages from the overrides. I don't think this really matters for
now though (it does make the checking for whether overrides are still
needed harder, but I'd like to automate dropping overrides eventually
too).

Anyway, we can drop this hack once we move to f32, though I have some
ideas too on how to solve this more correctly.

[1] coreos/fedora-coreos-tracker#293
[2] coreos/fedora-coreos-config#335 (comment)
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
70af1ed 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
@jlebon jlebon mentioned this issue May 1, 2020
Merged
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
152e4fd 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
1d35ea1 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
b6884fd 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
667d371 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 1, 2020
@jlebon
jobs: add bump-lockfile
455039b 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 4, 2020
@jlebon
jobs: add bump-lockfile
a6c585d 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 4, 2020
@jlebon
jobs: add bump-lockfile
1fac390 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 4, 2020
@jlebon
jobs: add bump-lockfile
b6cb4e7 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 4, 2020
@jlebon
jobs: add bump-lockfile
3eb4d1b 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
jlebon added a commit to coreos/fedora-coreos-pipeline that referenced this issue May 5, 2020
@jlebon
jobs: add bump-lockfile
8873a9c 
This job will implement lockfile bumping for testing-devel and
next-devel: coreos/fedora-coreos-tracker#293.

The original plan for this functionality was to have it in config-bot:
coreos/fedora-coreos-releng-automation#48

But in the end, I think it's more natural to have it as a Jenkins job
given that it does a lot of the same things as the pipeline/upstream CI
jobs. So that way it looks and feels just like another job that runs
cosa, and we get kola artifacts, we can re-use the shared library,
it's easily inspectable, we can hook it to Slack, etc...
@jlebon
Copy link
Member Author

jlebon commented May 6, 2020

This was fixed by coreos/fedora-coreos-pipeline#229.

@jlebon jlebon closed this as completed May 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
jira for syncing to jira
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dustymabe @jlebon