Use retain_package_versions to filter out older pkgs at sync time #2547

dralley · 2022-05-26T20:36:18Z

dralley · 2022-05-31T04:06:51Z

Successful proof of concept, tests using RHEL 7 repo:

immediate, no package limit

Sync time: I ran out of disk space and the whole thing failed (at least I think that was what happened, the DB started refusing to execute heartbeat writes), but it took about an hour to get ~80% finished

Package data downloaded: 60 Gb (measured separately)

immediate, package limit = 1

Sync time: 570 seconds (9m30s)

Package data downloaded: 4.9 Gb

on_demand, package limit = 1

Sync time reduced from 460 -> 203 seconds (7m40s -> 3m3s)

One thing I (re) discovered is that the EVRs need to be grouped by arch, which makes my calculation in the initial post wrong. Instead of ~4200 packages vs. 32,000 it's ~5500 packages vs 32,000. Didn't change the situation too much though.

dralley · 2022-06-03T17:22:35Z

pulp_rpm/app/rpm_version.py

@@ -0,0 +1,243 @@
+# Sourced from https://github.com/nexB/univers


Vendor, or pull the whole huge package? It supports version comparisons for a dozen different package types (arch, debian, python, ruby, go, etc.)

My take would be to just take the code in-line, rather than dragging a huge surface into pulp_rpm most of which we do not need.

dralley · 2022-06-03T17:26:49Z

@ipanova @ggainey This is ready for a first-pass lookthrough, I left some open questions inline.

dralley · 2022-06-04T01:04:37Z

So the new test I wrote actually passes despite failing on CI, but the unfortunate reason is that the packages used in our fixtures are the same packages, so if the non-modular fixture gets synced first Pulp sets is_modular: false on the content unit and if the modular fixture gets synced first then Pulp sets is_modular: true. I think this is going to potentially cause problems for a lot of tests and it might be a general problem as well.

dralley · 2022-06-04T01:07:05Z

The first commit and second commit use two different strategies if you want to compare. The second is simpler and probably safer.

dralley · 2022-06-24T19:26:36Z

pulp_rpm/app/tasks/synchronizing.py


+        # TODO: should this reflect the actual # of packages or just the # synced?
+        # Do we need a new progress bar?


Need feedback on this

I think just synced otherwise it can bring questions "why the counter is at 95/100 packages"

dralley · 2022-06-26T03:43:27Z

I deleted the previous coment about TODOs, because one was addressed and the other I realized that if it is a problem, it's a problem we would already have, not a new one. I will file it as a new task to investigate that.

dralley · 2022-06-26T21:08:25Z

@ipanova This should be ready for review now. Tests were failing because of #2617

ipanova

You'll probably want to rebase once you merge the #2611
I gave it a look and changes seem to be fine.

closes pulp#2479

ggainey

Def a significant improvement. Between the experimentation you did initially, the test modifications, and going thru the code-logic in my head a few times, it looks like it gives us what we (and our users) are looking for. lg2m!

patchback · 2022-07-18T13:58:49Z

Backport to 3.17: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 788687f on top of patchback/backports/3.17/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547

Backporting merged PR #2547 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/pulp/pulp_rpm.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.17/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547 upstream/3.17

Now, cherry-pick PR Use retain_package_versions to filter out older pkgs at sync time #2547 contents into that branch:
```
$ git cherry-pick -x 788687f1fe1f9d443f3edd855070fde5c9caaceb
```
If it'll yell at you with something like fatal: Commit 788687f1fe1f9d443f3edd855070fde5c9caaceb is a merge but no -m option was given., add -m 1 as follows intead:
```
$ git cherry-pick -m1 -x 788687f1fe1f9d443f3edd855070fde5c9caaceb
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Use retain_package_versions to filter out older pkgs at sync time #2547 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.17/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2022-07-26T21:34:44Z

Backport to 3.14: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 788687f on top of patchback/backports/3.14/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547

Backporting merged PR #2547 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/pulp/pulp_rpm.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.14/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547 upstream/3.14

Now, cherry-pick PR Use retain_package_versions to filter out older pkgs at sync time #2547 contents into that branch:
```
$ git cherry-pick -x 788687f1fe1f9d443f3edd855070fde5c9caaceb
```
If it'll yell at you with something like fatal: Commit 788687f1fe1f9d443f3edd855070fde5c9caaceb is a merge but no -m option was given., add -m 1 as follows intead:
```
$ git cherry-pick -m1 -x 788687f1fe1f9d443f3edd855070fde5c9caaceb
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Use retain_package_versions to filter out older pkgs at sync time #2547 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.14/788687f1fe1f9d443f3edd855070fde5c9caaceb/pr-2547

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

dralley force-pushed the filter-old-pkgs branch 10 times, most recently from 0679f60 to 0bbf62f Compare May 31, 2022 03:51

dralley force-pushed the filter-old-pkgs branch 6 times, most recently from bd5843f to 7ae000a Compare June 3, 2022 17:17

dralley requested review from ggainey and ipanova June 3, 2022 17:17

dralley marked this pull request as ready for review June 3, 2022 17:18

dralley commented Jun 3, 2022

View reviewed changes

dralley force-pushed the filter-old-pkgs branch 2 times, most recently from 17b2bab to 47fa6c2 Compare June 3, 2022 20:42

dralley force-pushed the filter-old-pkgs branch from aaf699c to 657880a Compare June 4, 2022 01:04

dralley force-pushed the filter-old-pkgs branch from 657880a to a3a8d4f Compare June 4, 2022 01:09

dralley added the backport-3.17 label Jun 4, 2022

dralley mentioned this pull request Jun 16, 2022

Remove location_href and location_base from DB metadata, add filename #2586

Closed

dralley force-pushed the filter-old-pkgs branch 3 times, most recently from 066c25b to a65e277 Compare June 16, 2022 14:28

dralley force-pushed the filter-old-pkgs branch 2 times, most recently from 1e73e66 to eeb49ae Compare June 24, 2022 15:59

dralley commented Jun 24, 2022

View reviewed changes

dralley force-pushed the filter-old-pkgs branch 2 times, most recently from 503a915 to c3e6bf8 Compare June 26, 2022 03:28

dralley force-pushed the filter-old-pkgs branch from c3e6bf8 to 61fb289 Compare June 26, 2022 21:04

dralley force-pushed the filter-old-pkgs branch from 61fb289 to 4dd667e Compare June 26, 2022 22:16

ipanova reviewed Jul 1, 2022

View reviewed changes

Use retain_package_versions to filter out older pkgs at sync time

601dfb5

closes pulp#2479

dralley force-pushed the filter-old-pkgs branch from 4dd667e to 601dfb5 Compare July 3, 2022 04:10

ggainey approved these changes Jul 18, 2022

View reviewed changes

dralley merged commit 788687f into pulp:main Jul 18, 2022

dralley deleted the filter-old-pkgs branch July 18, 2022 13:58

dralley added the backport-3.14 label Jul 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use retain_package_versions to filter out older pkgs at sync time #2547

Use retain_package_versions to filter out older pkgs at sync time #2547

dralley commented May 26, 2022

dralley commented May 31, 2022 •

edited

Loading

dralley Jun 3, 2022

ggainey Jun 9, 2022

dralley commented Jun 3, 2022

dralley commented Jun 4, 2022 •

edited

Loading

dralley commented Jun 4, 2022

dralley Jun 24, 2022

ipanova Jul 1, 2022

dralley commented Jun 26, 2022 •

edited

Loading

dralley commented Jun 26, 2022

ipanova left a comment

ggainey left a comment

patchback bot commented Jul 18, 2022 •

edited

Loading

patchback bot commented Jul 26, 2022 •

edited

Loading

		@@ -0,0 +1,243 @@
		# Sourced from https://github.com/nexB/univers


		# TODO: should this reflect the actual # of packages or just the # synced?
		# Do we need a new progress bar?

Use retain_package_versions to filter out older pkgs at sync time #2547

Use retain_package_versions to filter out older pkgs at sync time #2547

Conversation

dralley commented May 26, 2022

dralley commented May 31, 2022 • edited Loading

immediate, no package limit

immediate, package limit = 1

on_demand, package limit = 1

dralley Jun 3, 2022

Choose a reason for hiding this comment

ggainey Jun 9, 2022

Choose a reason for hiding this comment

dralley commented Jun 3, 2022

dralley commented Jun 4, 2022 • edited Loading

dralley commented Jun 4, 2022

dralley Jun 24, 2022

Choose a reason for hiding this comment

ipanova Jul 1, 2022

Choose a reason for hiding this comment

dralley commented Jun 26, 2022 • edited Loading

dralley commented Jun 26, 2022

ipanova left a comment

Choose a reason for hiding this comment

ggainey left a comment

Choose a reason for hiding this comment

patchback bot commented Jul 18, 2022 • edited Loading

Backport to 3.17: 💔 cherry-picking failed — conflicts found

Backporting merged PR #2547 into main

patchback bot commented Jul 26, 2022 • edited Loading

Backport to 3.14: 💔 cherry-picking failed — conflicts found

Backporting merged PR #2547 into main

dralley commented May 31, 2022 •

edited

Loading

dralley commented Jun 4, 2022 •

edited

Loading

dralley commented Jun 26, 2022 •

edited

Loading

patchback bot commented Jul 18, 2022 •

edited

Loading

patchback bot commented Jul 26, 2022 •

edited

Loading