frontend: align delete_after_days validators

[majamassarini]

Should fix packit/packit-service#2446

[praiskup]

Nit: a note in the commit message that we are synchronizing defaults of two forms would be nice (or in-line comment). But LGTM

[praiskup]

LGTM

[lsm5]

@praiskup @FrostyX can we please deploy a hotfix with this patch asap? It'll help silence a lot of annoying packit error messages on podman and related projects. Thanks!

[FrostyX]

I hotfixed this in production. Can you please try if it works as expected?

[lsm5]

I hotfixed this in production. Can you please try if it works as expected?

Thanks!!. We'll watch out if the issue reoccurs.

[lsm5]

@FrostyX @majamassarini still happening containers/podman#23601 (comment) :(

[nforro]

Would it be possible to check the logs to see what exactly causes the error on Copr side? Unfortunately on Packit side we do not log the exact request we send, but we can enable that if needed.

[FrostyX]

I am checking the Copr logs and I cannot see any error there either :-/
(in regards to the packit/containers-podman-23601 project)

[nforro]

If it helps, the request happened between 2024-08-13T13:42:00.635998798+00:00 and 2024-08-13T13:42:00.762452094+00:00 🙂

[FrostyX]

Thank you @nforro, I hoped it'll help but probably nothing.

[root@copr-fe ~][PROD]# cat /var/log/copr-frontend/frontend.log |grep "2024-08-13 13:42" |grep packit |grep ERROR
2024-08-13 13:42:07,770 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-dev.
2024-08-13 13:42:08,347 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:08,573 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:21,730 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-dev.
2024-08-13 13:42:22,285 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:22,465 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:32,338 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-dev.
2024-08-13 13:42:33,282 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:33,866 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.
2024-08-13 13:42:53,364 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 b'Project packit-stg/packit-hello-world-1636 does not exist.'
2024-08-13 13:42:57,361 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-dev.
2024-08-13 13:42:58,306 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][ANON] Response error: 404 Chroot centos-9-x86_64 does not exist in packit/packit-stable.

Also this is empty:

[root@copr-fe ~][PROD]# cat /var/log/copr-frontend/frontend.log | grep "packit/containers-podman-23601" |grep ERROR

[majamassarini]

@FrostyX thanks for looking at the logs.
I was thinking that probably we should open an issue for this. I hoped it was somehow related to the above fix but sadly it isn't.
I looked in our logs and what catches me is that we got 500 errors from COPR in the last 7 days just in a bunch of projects and the exceptions are scattered all along the period of time; so I would say it does not depend on an high volume of requests or high load in the COPR service.

2024-08-13T13:42:03 containers/podman#23601
2024-08-10T09:39:55 containers/podman#23569
2024-08-12T13:29:25 containers/podman#23581
2024-08-12T15:38:41 containers/podman#23587
2024-08-07T14:38:59 containers/podman#23537

2024-08-13T16:26:09 containers/common#2124
2024-08-09T23:14:02 containers/common#2119

2024-08-08T00:58:49 containers/crun#1513
2024-08-12T17:50:45 containers/crun#1519
2024-08-12T21:16:54 containers/crun#1520

2024-08-10T00:08:19 containers/buildah#5680
2024-08-12T19:43:26 containers/buildah#5681
2024-08-12T20:15:48 containers/buildah#5682

2024-08-11T14:02:53 containers/netavark#1052

2024-08-13T10:50:34 rpm-software-management/dnf5#1625

2024-08-08T11:29:19 cockpit-project/cockpit-machines#1760
2024-08-11T19:30:25 cockpit-project/cockpit-machines#1761
2024-08-12T03:47:41 cockpit-project/cockpit-machines#1762

The containers projects and the cockpit-machines project both use the packages key. With the packages key I would expect more requests from Packit to COPR in a short period of time in comparison to other Packit projects. I could be wrong, but to me it looks like a race condition on the COPR side. Also because this does not happen always on the same PR, thus, probably, it is not the data we submit to COPR.

The dnf5 project, instead, has the most simple packit config we could find and nevertheless has been hit by this problem. I can explain it again just with some kind of race condition...

I can't spot anything else interesting in our logs but let us know if we can help you in some way debugging it.

[nforro]

FTR, I enabled logging of the response content we get from Copr, perhaps we could find something useful in there (or perhaps not).

I could also try to enable logging of the raw requests and responses, it could be possible by exploiting urllib3 logger and filtering messages of interest, if you think that could be useful.

[FrostyX]

I created #3372 let's move the discussion there