Skip to content

Commit

Permalink
Fixed concurrent-overlapping-subrepo syncs.
Browse files Browse the repository at this point in the history
Syncing the 'same' kickstart repo into multiple repositories at
the same time would cause fatal errors. Taught pulp_rpm to notice
the collisions and assume "whoever got there first" is The Winner.

Issue pulp#2278 has reliable reproducer script for the failure-case.

fixes pulp#2278.
[nocoverage]
  • Loading branch information
ggainey committed Oct 17, 2022
1 parent 55226c7 commit 8cfaecb
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 9 deletions.
1 change: 1 addition & 0 deletions CHANGES/2278.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Addressed concurrent-sync subrepo collisions.
34 changes: 25 additions & 9 deletions pulp_rpm/app/tasks/synchronizing.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from django.core.files import File
from django.db import transaction
from django.db import transaction, IntegrityError
from django.db.models import Q


Expand Down Expand Up @@ -555,14 +555,30 @@ def is_subrepo(directory):
namespace=directory,
)

dv = RpmDeclarativeVersion(first_stage=stage, repository=repo, mirror=mirror)
repo_version = dv.create() or repo.latest_version()

repo_config["sync_details"]["most_recent_version"] = repo_version.number
repo.last_sync_details = repo_config["sync_details"]
repo.save()

repo_sync_results[directory] = repo_version
# Concurrent syncs colliding while attempting to create the same version in the same
# repo is Bad. It can happen if multiple repositories have the same sub-repositories
# (e.g., multiple repos syncing from the same remote kickstart-repo with sub-repos,
# such as RHEL6-kickstart). The collision is due to the kickstart-sub-repos naming
# scheme of `name = f"{repodata}-{treeinfo['hash']}"` from 100 lines or so up.
# This is not unique across repos, if multiple repos are syncing identical .treeinfo
# files.
# When a collision happens, instead of a fatal error, assume "whoever got there first
# won the new-version derby" and that the is nothing for 'this' thread to do here.
try:
dv = RpmDeclarativeVersion(first_stage=stage, repository=repo, mirror=mirror)
repo_version = dv.create() or repo.latest_version()

repo_config["sync_details"]["most_recent_version"] = repo_version.number
repo.last_sync_details = repo_config["sync_details"]
repo.save()

repo_sync_results[directory] = repo_version
except IntegrityError:
log.info(
_("Collision creating a new version for repository {} - skipping.").format(
repo.name
)
)

if skipped_syncs:
with ProgressReport(
Expand Down

0 comments on commit 8cfaecb

Please sign in to comment.