-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
core: handle shared files and multilib #1227
Conversation
Marking as WIP for now. There's some things to split out and minor issues to iron out on CentOS. And tests. |
@@ -2560,20 +2736,8 @@ delete_package_from_root (RpmOstreeContext *self, | |||
if (!g_str_has_prefix (fn, "usr/")) | |||
continue; | |||
|
|||
/* avoiding the stat syscall is worth a bit of userspace computation */ | |||
if (rpmostree_str_has_prefix_in_ptrarray (fn, deleted_dirs)) | |||
continue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dropped this optimization here since we now do things like librpm does: we traverse the file list backwards, which means directories are emptied out before we delete them.
This is analogous to the filtering support for the commit API: we allow library users to skip over checking out specific files. This is useful in some tricky situations where we *know* that the files to be checked out will conflict with existing files in subtle ways. One such example is in rpm-ostree support for multilib. There, we want to allow checking out a package onto an existing tree, but skipping over files that are not coloured to our preferred value (e.g. not overwriting an i686 version of `ldconfig` if we already have the `x86_64` version). See coreos/rpm-ostree#1227 for details.
This is analogous to the filtering support for the commit API: we allow library users to skip over checking out specific files. This is useful in some tricky situations where we *know* that the files to be checked out will conflict with existing files in subtle ways. One such example is in rpm-ostree support for multilib. There, we want to allow checking out a package onto an existing tree, but skipping over files that are not coloured to our preferred value (e.g. not overwriting an i686 version of `ldconfig` if we already have the `x86_64` version). See coreos/rpm-ostree#1227 for details.
This really ramps up quite significantly the complexity around doing checkouts. I don't have any clever ideas for solving the file coloring problem without that, but it's unfortunate. One random example, I'd been thinking about multi-threading checkouts and we'd have to evaluate doing that a lot more carefully with app callbacks involved. OK here's a random idea: what if we we ordered the package set by "primary arch", then "other" and did checkouts of "other" with |
Hmm, not sure. I don't have a good grasp on what kinds of third-party packages are out there. :) Otherwise, I think that might work... Actually, this won't work if we're overlaying a "primary" arch when the "other" arch is the one baked in, right? More generally though, this means we're turning off conflict detection for the whole package if only a single file is of a different colour. (Or just fall back to librpm's detection in that case, I guess?)
Yeah, that makes sense; it makes things trickier. OTOH, filtering during checkout would allow us to do more sophisticated things down the line. One example that immediately comes to mind is the RPM caching problem in unified core mode. We wouldn't have to invalidate the whole cache if we changed e.g. |
This is analogous to the filtering support for the commit API: we allow library users to skip over checking out specific files. This is useful in some tricky situations where we *know* that the files to be checked out will conflict with existing files in subtle ways. One such example is in rpm-ostree support for multilib. There, we want to allow checking out a package onto an existing tree, but skipping over files that are not coloured to our preferred value (e.g. not overwriting an i686 version of `ldconfig` if we already have the `x86_64` version). See coreos/rpm-ostree#1227 for details. Closes: #1441 Approved by: cgwalters
This is analogous to the filtering support for the commit API: we allow library users to skip over checking out specific files. This is useful in some tricky situations where we *know* that the files to be checked out will conflict with existing files in subtle ways. One such example is in rpm-ostree support for multilib. There, we want to allow checking out a package onto an existing tree, but skipping over files that are not coloured to our preferred value (e.g. not overwriting an i686 version of `ldconfig` if we already have the `x86_64` version). See coreos/rpm-ostree#1227 for details. Closes: #1441 Approved by: cgwalters
This is analogous to the filtering support for the commit API: we allow library users to skip over checking out specific files. This is useful in some tricky situations where we *know* that the files to be checked out will conflict with existing files in subtle ways. One such example is in rpm-ostree support for multilib. There, we want to allow checking out a package onto an existing tree, but skipping over files that are not coloured to our preferred value (e.g. not overwriting an i686 version of `ldconfig` if we already have the `x86_64` version). See coreos/rpm-ostree#1227 for details. Closes: #1441 Approved by: cgwalters
Split out to reduce noise in upcoming patch.
This is useful to have in a more global location since we deal with RPM and OSTree paths in various places.
Not all files from an RPM are necessarily removed during pkg erasure. For example, files which are shared between pkgs shouldn't be deleted. Similarly, not all files in an RPM are necessarily copied during pkg installs. This is the case for multilib handling, which is a mess in its own right. But such is the cost of trying to replace major parts of a long-standing foundational project like RPM. This patch adds some smarts to the way we do overlays and overrides to handle these cases by calculating beforehand which files we *should't* checkout/delete. Closes: coreos#1217 Closes: coreos#1145
Now working on CentOS and with tests! ⬆️ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall...looks pretty good. I certainly believe you've sucessfully
reverse-engineered all the file color bits. The comments are really helpful.
Tests look sane.
One thing that had me really pausing about this before was I thought
we'd need to do a lot of computation in the filter, but in fact it's
just a hash table lookup. (So in the future it could be safely multithreaded)
I also like that we don't need to have hash table entries for all files,
just the exceptions.
Still though, I had to scan through handle_file_dispositions()
a few times to
really feel like I understood it. It's complex, but it's solving a complex
problem.
I wonder if somehow we could only involve this if we detect multi-colored packages in the transaction?
src/libpriv/rpmostree-core.c
Outdated
const char *fn = rpmfiFN (fi); | ||
g_assert (fn != NULL); | ||
|
||
if (g_str_has_suffix (fn, "ldconfig")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's remember to delete this?
src/libpriv/rpmostree-util.c
Outdated
@@ -234,6 +234,23 @@ rpmostree_pkg_get_local_path (DnfPackage *pkg) | |||
} | |||
} | |||
|
|||
char* | |||
rpmostree_translate_path_for_ostree (const char *path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this one deserves at least a quick comment at the top like:
/* Convert a "traditional" path (normally from e.g. an RPM) into its final location in ostree */
src/libpriv/rpmostree-core.c
Outdated
|
||
/* let's make the safe assumption that the color mess is only an issue for /usr */ | ||
const char *fn_rel = fn; | ||
fn_rel += strspn (fn, "/"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not const char *fn_rel = fn + strspn (fn, "/")
?
src/libpriv/rpmostree-core.c
Outdated
/* this is a bit awkward; we relativize for the translation, but then make it absolute | ||
* again to match libostree */ | ||
const char *path_rel = path; | ||
path_rel += strspn (path, "/"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could just be path += strspn (path, "/")
right?
src/libpriv/rpmostree-core.c
Outdated
@@ -2378,6 +2378,37 @@ rpmostree_context_consume_package (RpmOstreeContext *self, | |||
return TRUE; | |||
} | |||
|
|||
static void |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/* Build a mapping from filename → rpm "color" */
if (color && other_color && (color != other_color)) | ||
{ | ||
/* do we already have the preferred color installed? */ | ||
if (color & ts_prefcolor) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just for my own understanding...could this be ==
instead of &
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you're correct AFAIU. They're both bitfields, but ts_color
contains all the colors we're compatible with, while ts_prefcolor
contains only a single color from that set. And files can only one specific color, so ==
would work equally well here.
src/libpriv/rpmostree-core.c
Outdated
GHashTable *files_skip = user_data; | ||
if (g_hash_table_contains (files_skip, path)) | ||
return OSTREE_REPO_CHECKOUT_FILTER_SKIP; | ||
/* hack for nsswitch.conf, which we modify during treecompose; a better heuristic here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand why this is necessary now; what would break without this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I expanded on the comment a bit to clarify.
Fixups! ⬆️ |
⚡ Test exempted: pull fully rebased and already tested. |
This is useful to have in a more global location since we deal with RPM and OSTree paths in various places. Closes: #1227 Approved by: cgwalters
Not all files from an RPM are necessarily removed during pkg erasure. For example, files which are shared between pkgs shouldn't be deleted. Similarly, not all files in an RPM are necessarily copied during pkg installs. This is the case for multilib handling, which is a mess in its own right. But such is the cost of trying to replace major parts of a long-standing foundational project like RPM. This patch adds some smarts to the way we do overlays and overrides to handle these cases by calculating beforehand which files we *should't* checkout/delete. Closes: #1217 Closes: #1145 Closes: #1227 Approved by: cgwalters
We basically do a light-weight version of what librpm does here to determine which files to really remove/install during a transaction.
Closes: #1217
Closes: #1145
Requires: ostreedev/ostree#1441