Failed update from 17.3.1~ynh1 to 17.6.1~ynh1

[colmoneill]

Failed update from 17.3.1~ynh1 to 17.6.1~ynh1

My gitlab instance is acting a little weird, so I tried updating it. It seems to fail because it is unable to complete the backup, but I am not certain.

WARNING - /usr/bin/env: ‘/opt/gitlab/embedded/bin/bundle’: Permission denied

Full logs: https://paste.yunohost.org/raw/udogucunur

Context

• Hardware: Old computer
• YunoHost version: 12.0.7
• I have access to my server: Through SSH, through the webadmin and direct access via keyboard / screen
• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: not particularly, but I did just complete the update to ynh12, which failed half way (I recently had to replace my boot drive, I reinstalled grub fine, but during the system-update the grub-pc package had traces of the old hard-drive. Reconfigured it, and was able to complete the update)
• If upgrading, current package version: 17.3.1~ynh1

Steps to reproduce

Tried upgrading via CLI and via webadmin

First I:

sudo yunohost app upgrade gitlab

which failed so I

sudo yunohost app upgrade --force gitlab

because it was suggested here: https://doc.yunohost.org/en/bookworm_migration_issues_faq

logs for the CLI attempt: https://paste.yunohost.org/raw/udogucunur

then I tried the webadmin, I was able to update other apps (I just completed the migration to yhn12), but the results are the same.

Logs for that webadmin attempt: https://paste.yunohost.org/raw/xukosuvudi

Expected behavior

• yhn has made me expect a backup as part of the upgrade of an app, this did not happen, nor did the app update.

Logs

Most recent logs:
https://paste.yunohost.org/raw/xukosuvudi

Thanks

Thanks for any help you could provide! I really appreciate it! All the best

[colmoneill]

Some progress: I found out about the gitlab-ctl CLI tool that seems to be bundled with ynh_gitlab, and running the status operation indicates that many of gitlab's services were completely down. I attempted a gitlab-ctl upgrade which indicated that postgres was missing.

I noted that this post on the forum had the same issue, so I apt install postgresql-15-postgis-3 postgresql-15-postgis-3-scripts and reran the upgrade which seemed to run successfully... Or so I thought.

Gitlab's status is still mostly down, and I also am still running into the same permission issue described above.

admin@server:~ $ sudo gitlab-ctl status
down: alertmanager: 1s, normally up, want up; run: log: (pid 2106) 157590s
down: gitaly: 1s, normally up, want up; run: log: (pid 2100) 157590s
down: gitlab-exporter: 1s, normally up, want up; run: log: (pid 2115) 157590s
down: gitlab-kas: 1s, normally up, want up; run: log: (pid 2112) 157590s
down: gitlab-workhorse: 1s, normally up, want up; run: log: (pid 2102) 157590s
run: logrotate: (pid 3409175) 152s; run: log: (pid 2113) 157590s
run: nginx: (pid 3409181) 152s; run: log: (pid 2103) 157590s
down: node-exporter: 1s, normally up, want up; run: log: (pid 2116) 157590s
down: postgres-exporter: 1s, normally up, want up; run: log: (pid 2101) 157590s
down: postgresql: 1s, normally up, want up; run: log: (pid 2135) 157590s
down: prometheus: 1s, normally up, want up; run: log: (pid 2104) 157590s
down: puma: 1s, normally up, want up; run: log: (pid 2109) 157590s
down: redis: 1s, normally up, want up; run: log: (pid 2118) 157590s
down: redis-exporter: 1s, normally up, want up; run: log: (pid 2114) 157590s
down: sidekiq: 1s, normally up, want up; run: log: (pid 2117) 157590s

[colmoneill]

All these services are down and I can't figure out why.

I'm tempted to try to reinstall gitlab, but I'd also like to recover what I had on there. In particular, I have a project that uses this instance of gitlab for a Static CMS, and I have my CI configs set in this instance. I don't know if I should venture out of the ynh docs to fix this Gitlab?

sudo gitlab-ctl restart
timeout: down: alertmanager: 1s, normally up, want up
timeout: down: gitaly: 0s, normally up, want up
timeout: down: gitlab-exporter: 0s, normally up, want up
timeout: down: gitlab-kas: 0s, normally up, want up
timeout: down: gitlab-workhorse: 0s, normally up, want up
ok: run: logrotate: (pid 34162) 0s
ok: run: nginx: (pid 34185) 1s
timeout: down: node-exporter: 0s, normally up, want up
timeout: down: postgres-exporter: 1s, normally up, want up
timeout: down: postgresql: 1s, normally up, want up
timeout: down: prometheus: 1s, normally up, want up
timeout: down: puma: 1s, normally up, want up
ok: run: redis: (pid 38050) 0s
timeout: down: redis-exporter: 1s, normally up, want up
timeout: down: sidekiq: 0s, normally up, want up

[kay0u]

Hi, it seems like all your permissions are fucked up in /opt/gitlab?

Do you have logs in /var/log/gitlab/*/ corresponding to gitlab-ctl services with error message? We could find some help here

[colmoneill]

The files listed in /opt/gitlab all seem to be owned by the root user, which I guess is bad? I don't know how that would have happened?

sudo ls -alh /opt/gitlab/
total 33M
drwxr-x--- 11 gitlab gitlab 4.0K Aug 28 18:29 .
drwxr-xr-x  9 root   root   4.0K Oct 22 18:32 ..
drwxr-xr-x  2 root   root   4.0K Aug 28 18:29 bin
-rw-r--r--  1 root   root    19M Aug 20 21:55 dependency_licenses.json
drwxr-xr-x 15 root   root   4.0K Dec  6 17:16 embedded
drwxr-xr-x 12 root   root   4.0K Dec  6 17:16 etc
drwxr-xr-x  2 root   root   4.0K Apr  3  2024 init
-rw-r--r--  1 root   root    14M Aug 20 21:55 LICENSE
drwxr-xr-x  2 root   root   4.0K Aug 28 18:29 licenses
drwxr-xr-x  2 root   root    20K Aug 28 18:29 LICENSES
drwxr-xr-x  2 root   root   4.0K Apr  3  2024 service
drwxr-xr-x 17 root   root   4.0K Apr  3  2024 sv
drwxr-xr-x  3 root   root   4.0K Apr  3  2024 var
-rw-r--r--  1 root   root    30K Aug 20 21:56 version-manifest.json
-rw-r--r--  1 root   root    16K Aug 20 21:56 version-manifest.txt

Regarding the content of the /var/log/gitlab/* / folder, I see folders relating to all the services that gitlab-ctl seems to use, but no folder specific to gitlab-ctl so I listed the content of the folder here: https://paste.yunohost.org/anuxuxogoy.lua I hope that helps.

Thanks for the support @kay0u, it looks like I have an individual issue, maybe not something that applies to the entire gitlab_ynh package?

All the best!

[kay0u]

The files listed in /opt/gitlab all seem to be owned by the root user, which I guess is bad? I don't know how that would have happened?

Nope, I do have the same on my server, let's continue the investigation then.

Can you run sudo ls -alh /opt/gitlab/embedded/bin too? I'd like to compare with my server.

Regarding the content of the /var/log/gitlab/* / folder, I see folders relating to all the services that gitlab-ctl seems to use, but no folder specific to gitlab-ctl so I listed the content of the folder here: paste.yunohost.org/anuxuxogoy.lua I hope that helps.

Can you provide the output of sudo gitlab-ctl tail ?

[colmoneill]

I ran both those commands and the output is below

https://paste.yunohost.org/becasaqico.yaml

Thanks again for the invetigation

[kay0u]

That's so weird, I can find some similar issues, for example: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3721
Maybe a postgresql migration gone wrong, or a postgresql version change due to the migration YNH11->12 (postgresql should be integrated into the gitlab package if I understand correctly, so this shouldn't happen)?

Can you try sudo gitlab-ctl reconfigure?

[colmoneill]

Yes, I had tried this too, when I found out about the gitlab-ctl tool but I don't think it is actually managing to reconfigure.

I re-ran a gitlab-ctl status and a gitlab-ctl restart command after the reconfigure; output is in the hastebin below;

https://paste.yunohost.org/lokoculubu.sql