feat: add ceph-mon service to osds hosts firewall

[jackhodgkiss]

Hosts that carry just OSDs appear to require ceph-mon service firewalls to ensure communication for Ceph MDS connectivity.

[Alex-Welsh]

nit: Should probably have a release note

[jackhodgkiss]

/cherry-pick stackhpc/2023.1

[priteau]

/cherry-pick stackhpc/2023.1

Shouldn't you merge first?

[Alex-Welsh]

/cherry-pick stackhpc/2023.1

Shouldn't you merge first?

From the bot docs:

If you add/edit the comment on an unmerged pull request, it will not do anything immediately. When the pull request is merged, cherry-pick-bot will scan all the comments on the pull request, collect valid target branches to cherry-pick to, and then attempt to cherry-pick each one.

[jackhodgkiss]

/cherry-pick stackhpc/2023.1

Shouldn't you merge first?

The cherry pick will only happen after this has merged. https://github.com/googleapis/repo-automation-bots/tree/main/packages/cherry-pick-bot#on-unmerged-pull-request

This way I can line up the cherry-pick now and not after it has merged.

[priteau]

This is odd behaviour, can you confirm that you see OSDs listening on ports 3300 or 6789?

[priteau]

It is also worth noting that the highest port number used by Ceph was increased in [1] but the firewalld service definition was not updated right away [2].

[1] ceph/ceph#42210
[2] firewalld/firewalld#1329

[priteau]

Use double backticks.

[jackhodgkiss]

It is also worth noting that the highest port number used by Ceph was increased in [1] but the firewalld service definition was not updated right away [2].

[1] ceph/ceph#42210 [2] firewalld/firewalld#1329

I think maybe solution is a bit clouded and maybe the firewall change here is a redherring. I don't have any appetite to revert the change in the environment where the problem was observed.

It was certainly missing ceph service from the host and enabling the firewall without ceph-mon caused sssd within the slurm cluster to fail. Though things didn't properly return until after a reboot of NFS hosts.

I think this could be tested within virtualised environment to be sure if this is required at all. Will mark as draft for now.

[jackhodgkiss]

This is odd behaviour, can you confirm that you see OSDs listening on ports 3300 or 6789?

Checked impacted hosts those ports are not in use.