500 errors in Packit #3372
Comments
|
Last 6 hours nothing suspicious. One of the events mentioned above created this traceback: |
|
Trying to open https://download.copr.fedorainfracloud.org/results/mcrha or https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/ leads to: Is this anyhow related to this issue or I may file a new one, please? |
|
@mcrha thank you for reporting that! Wes we had copr-backend issues yesterday, sorry for the inconvenience (should be working OK now). The problem discussed here is in copr-frontend.rpm (different VM). |
|
Aha, I see, different thing then. I'm sorry for the noise. You are right, it cured on itself an hour or so after I wrote a note here. |
|
@FrostyX , @praiskup I was quickly checking the last occurrences of this exception on the Packit side and I saw that this happened last time on August the 22nd around 10AM. |
|
If it can be of any help I checked the Packit logs again, here the latest exception we collected: 2024-09-16T06:00:25.410799561+00:00 rpm-software-management/mock#1452 |
This is TOCTOU issue. The other checks for duplications (on so many places) seem kinda redundant because nothing but try/except for commit() may catch these concurrency problems. Fixes: fedora-copr#3372
This is TOCTOU issue. The other checks for duplications (on so many places) seem kinda redundant because nothing but try/except for commit() may catch these concurrency problems. Fixes: fedora-copr#3372
This is TOCTOU issue. The other checks for duplications (on so many places) seem kinda redundant because nothing but try/except for commit() may catch these concurrency problems. Fixes: fedora-copr#3372
|
Hello, has the fix been deployed to copr? This issue did occur 8 hours ago: containers/ramalama#185 (comment) |
|
Not yet, the ETA plan for the release is next Thursday (if everything goes OK). Is that OK, or do you want us to hot-fix this in production? |
|
@praiskup this morning Packit logged a new exception (for the ramalama project): I thought we should handle it silently. |
@praiskup just to double check, Would be great if you could hotfix this. Else I'll just ask people to wait some more. |
|
Oh, yes - I meant "next week Thursday" rather than "this week Thursday". But I can try to hotfix tomorrow, seems pretty easy to rollback if problems appear at least. |
commit 4b1576f Merge: 57a303e 8b0977a Author: jait <jai8.jacob@gmail.com> Date: Wed Sep 25 22:59:20 2024 +0530 Merge branch 'fedora-copr:main' into Webhook-History-UI commit 8b0977a Author: Pavel Raiskup <praiskup@redhat.com> Date: Tue Sep 24 21:09:24 2024 +0200 rpmbuild: unblock testsuite commit 5d77d36 Author: Jiri Kyjovsky <j1.kyjovsky@gmail.com> Date: Tue Sep 17 14:35:28 2024 +0200 rpmbuild: specify snippets to mock config via copr-rpmbuild config file This allows us to specify tpm fs size to rpmbuild in order to be able to automatically generate its size for performance builders. See fedora-copr#3268 commit 57a303e Author: Jait Jacob <jai8.jacob@gmail.com> Date: Wed Sep 25 13:26:34 2024 +0530 remove unused import & revert localized_time filter method commit 658230d Author: Pavel Raiskup <praiskup@redhat.com> Date: Mon Sep 23 15:20:09 2024 +0200 backend: unknown resalloc tickets helper cleanup If no tickets are taken (which often happens in the staging environment), this script encountered corner case issues. commit d6a3472 Author: Miroslav Suchý <msuchy@redhat.com> Date: Mon Sep 23 22:12:43 2024 +0200 rpmbuild: do not require rpkg,pyp2rpm,pyp2spec,gem2rpm and fedora-review on rhel Resolves: RHBZ#2313878 commit 6ae7c6c Author: Miroslav Suchý <msuchy@redhat.com> Date: Mon Sep 23 22:04:11 2024 +0200 rpmbuild: do not require qemu-user-static on rhel Resolves: RHBZ#2313879 commit 1559b85 Author: Aurélien Bompard <aurelien@bompard.org> Date: Mon Jul 29 10:54:02 2024 +0200 Use `super()` without argument to make pylint happy Signed-off-by: Aurélien Bompard <aurelien@bompard.org> commit f3b1643 Author: Aurélien Bompard <aurelien@bompard.org> Date: Mon Jul 29 10:40:13 2024 +0200 Message schemas: set chroot message severity to DEBUG Signed-off-by: Aurélien Bompard <aurelien@bompard.org> commit c192726 Author: Aurélien Bompard <aurelien@bompard.org> Date: Mon Jul 29 10:38:34 2024 +0200 Message schemas: one-line descriptions should be the summary Signed-off-by: Aurélien Bompard <aurelien@bompard.org> commit 609d369 Author: Pavel Raiskup <praiskup@redhat.com> Date: Fri Sep 20 14:14:31 2024 +0200 frontend: fix the 500 for racy creation attempts This is TOCTOU issue. The other checks for duplications (on so many places) seem kinda redundant because nothing but try/except for commit() may catch these concurrency problems. Fixes: fedora-copr#3372 commit 446dcb3 Author: Jiri Kyjovsky <j1.kyjovsky@gmail.com> Date: Wed Sep 18 16:49:52 2024 +0200 beaker: use podman for testing inside container if installed commit 030740a Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 22 23:17:17 2024 +0530 remove unaccessed import commit cf1cb81 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Fri Sep 20 02:36:09 2024 +0530 webhook_history stores UNIX timestamps instead of DateTime commit 6aeb686 Author: Jiri Kyjovsky <j1.kyjovsky@gmail.com> Date: Mon Sep 16 10:15:37 2024 +0200 docker: set hard ulimits for docker container Because of bug in python3-daemon [1] we need to set ulimits inside docker container, otherwise backend and dist-git ooms. [1] - https://bugzilla.redhat.com/show_bug.cgi?id=2307635 commit 6aa1d5e Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 15 19:55:11 2024 +0530 resolve pylint warnings commit a2adf81 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 15 19:44:07 2024 +0530 remove unused code commit 14faa50 Author: Jakub Kadlcik <frostyx@email.cz> Date: Thu Sep 5 11:10:15 2024 +0200 frontend, python, cli: allow admins to set storage for new projects See fedora-copr#2533 This will be useful for beaker tests where we can now add basic tests for every supported storage. commit ad36b8b Author: Jakub Kadlcik <frostyx@email.cz> Date: Thu Sep 5 10:36:46 2024 +0200 frontend: make the default storage for new projects configurable See fedora-copr#2533 commit 286a913 Author: Jakub Kadlcik <frostyx@email.cz> Date: Tue Sep 10 08:36:23 2024 +0200 backend: add a timeout for waiting until a Pulp task finishes commit 23a2fa3 Author: Jakub Kadlcik <frostyx@email.cz> Date: Mon Jul 22 11:23:53 2024 +0200 backend: actions don't call uses_devel_repo function anymore commit c5166a1 Author: Jakub Kadlcik <frostyx@email.cz> Date: Thu Sep 5 09:43:09 2024 +0200 backend, frontend: implement project and build deletion in Pulp Fix fedora-copr#3318 Fix fedora-copr#3319 commit 19eff0c Author: Jakub Kadlcik <frostyx@email.cz> Date: Sun Sep 8 12:03:15 2024 +0200 copr: wait until Pulp publication is finished commit ff5288d Author: Miroslav Suchý <msuchy@redhat.com> Date: Wed Sep 11 08:19:41 2024 +0200 common: cleanup - remove six dependency commit 6c71993 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Wed Sep 11 16:21:34 2024 +0530 fix pylint complaint commit 649fe51 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Wed Sep 11 16:15:14 2024 +0530 optimize db calls, remove client side js, resolve review comments commit 31880b4 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 8 22:32:14 2024 +0530 cleanup commit 593a13f Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 8 22:17:58 2024 +0530 add pagination commit 99e142e Merge: 11d7fb1 2bdec45 Author: jait <jai8.jacob@gmail.com> Date: Sun Sep 8 15:03:36 2024 +0530 Merge branch 'fedora-copr:main' into Webhook-History-UI commit 11d7fb1 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sun Sep 8 15:03:29 2024 +0530 return webhook history in descending order & without any duplicates commit b7f2b34 Author: Jait Jacob <jai8.jacob@gmail.com> Date: Sat Sep 7 00:22:06 2024 +0530 frontend: show webhook history table under Setting->Integration
great. Thanks @praiskup |
|
I applied the patch now, I am sorry it took several days... got quite busy elsewhere. |
Thanks @praiskup . I'll watch out for further occurrences if any. |
|
@praiskup still seeing it unfortunately containers/buildah#5765 (comment) |
|
Thank you for the report. I'm locked in a meeting room, but this seems like a different issue, not sure if related: #3443. |
looks like they started running without anyone from the team restarting them. So maybe it works but some tmp failure messages need to be silenced? |
ack thanks @praiskup . I'll watch out for further occurrences. |
Reported by @majamassarini in #3329 (comment)
The text was updated successfully, but these errors were encountered: