-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
app: add 'makecache' command #1035
Conversation
src/daemon/rpmostreed-daemon.c
Outdated
@@ -451,7 +451,7 @@ update_status (RpmostreedDaemon *self) | |||
guint64 readytime = g_source_get_ready_time (self->idle_exit_source); | |||
guint64 curtime = g_source_get_time (self->idle_exit_source); | |||
guint64 timeout_micros = readytime - curtime; | |||
if (timeout_micros < 0) | |||
if (readytime < curtime) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd probably vote to split this out into a separate commit.
/* point the core to the passwd & group of the merge deployment */ | ||
rpmostree_context_set_passwd_dir (self->ctx, passwddir); | ||
/* make sure yum repos and passwd used are from our cfg merge */ | ||
rpmostree_context_configure_from_deployment (self->ctx, self->sysroot, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a nice change!
g_autofree char *origin_deployment_dirpath = | ||
ostree_sysroot_get_deployment_dirpath (sysroot, origin_merge_deployment); | ||
g_autofree char *origin_deployment_root = | ||
g_build_filename (sysroot_path, origin_deployment_dirpath, NULL); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit ugly too, but we can fix up the core API later.
MakeCacheTransaction *self = (MakeCacheTransaction *) transaction; | ||
OstreeSysroot *sysroot = rpmostreed_transaction_get_sysroot (transaction); | ||
g_autoptr(OstreeRepo) repo = NULL; | ||
if (!ostree_sysroot_get_repo (sysroot, &repo, cancellable, error)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In new code this can just be:
OstreeRepo *repo = ostree_sysroot_repo (sysroot);
.
Now with support for |
Rebased and added tests! ⬆️ Lifting WIP. |
It's no longer being built and is now older than the latest CentOS AH release. This should help us no longer see messages like: (rpm-ostree pkg-add:5662): GLib-CRITICAL **: g_variant_dict_lookup: assertion 'is_valid_dict (dict)' failed which happen because in coreos#1034, we started using `G_VARIANT_DICT_INIT`, whose special magic values only make sense in glib2 >= 2.50. (The alpha image stopped at 2.46). Saw this while debugging coreos#1035.
It's no longer being built and is now older than the latest CentOS AH release. This should help us no longer see messages like: (rpm-ostree pkg-add:5662): GLib-CRITICAL **: g_variant_dict_lookup: assertion 'is_valid_dict (dict)' failed which happen because in #1034, we started using `G_VARIANT_DICT_INIT`, whose special magic values only make sense in glib2 >= 2.50. (The alpha image stopped at 2.46). Saw this while debugging #1035. Closes: #1040 Approved by: cgwalters
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
High level comments:
I personally never liked the makecache
name...there's too many things that can be a cache, and the make
bit seems redundant.
refresh-mdcache
mdcache-refresh
md-refresh
rpmmd-refresh
Or something?
Anyways I'd also be OK if we just went with it.
The only other high level comment here is I think we should consider how this also affects the ostree side.
@@ -30,6 +30,17 @@ | |||
</defaults> | |||
</action> | |||
|
|||
<action id="org.projectatomic.rpmostree1.makecache"> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We already have org.projectatomic.rpmostree1.repo-refresh
...one thing we need to keep in mind is that it's not necessary for polkit actions to be 1-1 with dbus calls. The more polkit actions we have, the more anyone locking down a system needs to think about them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I'll merge it with repo-refresh
.
@@ -46,6 +47,7 @@ static GOptionEntry option_entries[] = { | |||
{ "check-diff", 0, G_OPTION_FLAG_HIDDEN, G_OPTION_ARG_NONE, &opt_preview, "Check for upgrades and print package diff only", NULL }, | |||
{ "preview", 0, 0, G_OPTION_ARG_NONE, &opt_preview, "Just preview package differences", NULL }, | |||
{ "check", 0, 0, G_OPTION_ARG_NONE, &opt_check, "Just check if an upgrade is available", NULL }, | |||
{ "cache-only", 'C', 0, G_OPTION_ARG_NONE, &opt_cache_only, "Do not update repo metadata cache", NULL }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah...but what about the ostree side? The option makes sense there too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. I was mostly going off of the --cache-only
naming from the compose side. I suppose we'll have to have a new switch there too for unified core re. ostree caching? Would be nice to keep it consistent across those commands.
tests/vmcheck/test-basic.sh
Outdated
cat > vmcheck-httpd.service << EOF | ||
[Service] | ||
# use py2 for CentOS | ||
ExecStart=/usr/bin/python -m SimpleHTTPServer 8888 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's also just systemd-run
FWIW.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ouhhh, that's much better!
Move the logging of yum repo information from `prepare` to `download_metadata`, since the latter could be called without necessarily calling the former, as is the case with `makecache`.
Addressed the comments! I had to rebase to change the commit messages. Hmm, actually, I could have kept the fixups still. Sorry about that! |
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree. To complete the picture, this goes hand in hand with the `-C` equivalent, which is added in the next patch.
This is the equivalent version of `yum/dnf -C`. It goes together with the new `makecache` command to allow completely asynchronous cache update and usage.
I've been lazy about actually using using rsync instead of scp when copying new RPMs over to the VM. We do this here. Also make `vm_send_test_repo` take a mode argument that allows callers to completely skip the sending of the repo file itself. This will be needed for the `makecache` test, in which we *don't* want the repo to be local. It looks cleaner anyway for the gpgcheck use case as well.
Fixups restored! ⬆️ |
I'm cool with So then the other issue is the I can't think of a case offhand where I wouldn't want it to do both. The main wrinkle here is around things like COPR repos...but we already have the |
Alrighty, renamed everything to - vm_raw_rsync --delete ${test_tmpdir}/yumrepo $VM:/tmp/vmcheck
+ # note we use -c here because we might be called twice within a second
+ vm_raw_rsync -c --delete ${test_tmpdir}/yumrepo $VM:/tmp/vmcheck which was causing the sporadic errors seen in the testers. Basically, if we didn't pass another second within the next repo regen, to
That's fine with me as well. I'm not sure what it even means in the ostree context. I suppose you mean e.g. fail if we don't have the RPM already in our pkgcache? |
For |
Ahh OK, that makes more sense. Let's do that as a follow-up? With e.g. an associated |
Move the logging of yum repo information from `prepare` to `download_metadata`, since the latter could be called without necessarily calling the former, as is the case with `makecache`. Closes: #1035 Approved by: cgwalters
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree. To complete the picture, this goes hand in hand with the `-C` equivalent, which is added in the next patch. Closes: #1035 Approved by: cgwalters
This is the equivalent version of `yum/dnf -C`. It goes together with the new `makecache` command to allow completely asynchronous cache update and usage. Closes: #1035 Approved by: cgwalters
I've been lazy about actually using using rsync instead of scp when copying new RPMs over to the VM. We do this here. Also make `vm_send_test_repo` take a mode argument that allows callers to completely skip the sending of the repo file itself. This will be needed for the `makecache` test, in which we *don't* want the repo to be local. It looks cleaner anyway for the gpgcheck use case as well. Closes: #1035 Approved by: cgwalters
Closes: #1035 Approved by: cgwalters
💔 Test failed - status-atomicjenkins |
⚡ Test exempted: merge already tested. |
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree. To complete the picture, this goes hand in hand with the `-C` equivalent, which is added in the next patch. Closes: #1035 Approved by: cgwalters
This is the equivalent version of `yum/dnf -C`. It goes together with the new `makecache` command to allow completely asynchronous cache update and usage. Closes: #1035 Approved by: cgwalters
I've been lazy about actually using using rsync instead of scp when copying new RPMs over to the VM. We do this here. Also make `vm_send_test_repo` take a mode argument that allows callers to completely skip the sending of the repo file itself. This will be needed for the `makecache` test, in which we *don't* want the repo to be local. It looks cleaner anyway for the gpgcheck use case as well. Closes: #1035 Approved by: cgwalters
Closes: #1035 Approved by: cgwalters
Just taking what I learned from coreos#1035 and applying it here. What's nice about this is that there's no cleanup needed. Once the process is killed (or worst case, we reboot the VM), there's no traces left at all. Also added a few extra "ok" outputs.
As @cgwalters mentioned in coreos#1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642
As Colin mentioned in coreos#1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642
As Colin mentioned in coreos#1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642
As Colin mentioned in coreos#1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642 Closes: coreos#687
As Colin mentioned in coreos#1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642 Closes: coreos#687
As Colin mentioned in #1035, the new `--cache-only` implemented only the rpmmd half of the story. Here we complete that story by also ensuring that when in cache-only mode, we don't download new ostree data nor new packages. We try to complete the requested operation with what we have. To do this, we add support for the same `SYNTHETIC` pull that was added in ostree[1] so that we don't actually pull, but still perform timestamp checking. On the pkgcache side, we disable all remote repos and instead insert all our cached RPMs into the `DnfSack`. Care is taken to still perform SHA256 verification for local pkg installs/replacements. [1] ostreedev/ostree#642 Closes: #687 Closes: #1049 Approved by: cgwalters
This is essentially the
dnf
/yum
equivalent for rpm-ostree. Tocomplete the picture, we're missing the
-C
equivalent as well for allthe commands that make use of this metadata. Though
makecache
on itsown should be particularly helpful in test environment provisioning
where one may have cached metadata available to inject and just wants
rpm-ostree to sanity check it.
Marking as WIP for now. It works, but it does... unsavory things that I'd like to clean up first.