Unable to start workspaces due to Syntax error in che-plugin-registry

[davidwindell]

Describe the bug

Unable to start workspaces as the che-plugin-registry is failing to start.

=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
AH00526: Syntax error on line 39 of /opt/rh/httpd24/root/etc/httpd/conf.d/mod_security.conf:
ModSecurity: Failed to open debug log file: /var/log/httpd24/modsec_debug.log

Che version

• nightly

Steps to reproduce

Try to start any new or existing workspace.

Runtime

• kubernetes

Installation method

• helm

Environment

• Cloud
  • other (Rancher/AWS)

[tolusha]

I successfully deployed che on minkube using the following command
chectl server:start --platform=minikube --installer=helm --k8spodreadytimeout=500000

@davidwindell
Have you tried with chectl?

[davidwindell]

@tolusha no, I'm using Rancher/Vanilla K8s so no chectl.

I've just forced the helm chart to use 7.2.0 instead of nightly and it works. From what I can tell therefore is that there is definitely a bug in quay.io/eclipse/che-plugin-registry:nightly

[amisevsk]

I haven't tested on k8s but quay.io/eclipse/che-plugin-registry:nightly starts without issue on OpenShift.

[davidwindell]

@amisevsk any ideas on how I could provide helpful debug for this? Where might the syntax error be coming from?

[amisevsk]

@davidwindell I'm kind of at a loss -- we don't do any modification to the Apache config from the upstream registry.centos.org/centos/httpd-24-centos7 image. Are you able to start a plain registry.centos.org/centos/httpd-24-centos7 container on your cluster (i.e. update the image in the plugin registry deploy to the the httpd container we're basing off of)?.

In my working container, the referenced line is

$ sed '39!d' /opt/rh/httpd24/root/etc/httpd/conf.d/mod_security.conf
    SecDebugLog /var/log/httpd24/modsec_debug.log

and the /var/log/httpd24/ directory has permissions

$ ls -al /var/log/httpd24
total 0
drwxrwx---. 1 default    root 54 Oct  9 13:09 .
drwxr-xr-x. 1 root       root 21 Sep 11 18:24 ..
-rw-r-----. 1 1000450000 root  0 Oct  9 13:09 modsec_audit.log
-rw-r-----. 1 1000450000 root  0 Oct  9 13:09 modsec_debug.log

Could you check your deploy is similar? Perhaps it's a permissions issue related to UID stuff.

[davidwindell]

I've just pulled today's nightly image and it randomly seems to be resolved. I'm going to close for now, thank you for looking into it.

[amisevsk]

@davidwindell Let us know if it crops up again -- I don't like when issues mysteriously go away.

[davidwindell]

Neither do I, will do 👍

[davidwindell]

This came back (perhaps it wasn't running nightly when I thought it was before). I've confirmed and I can't run the upstream image...

[davidwindell]

There something about my node(s) that doesn't want to run it, but I've no idea why. It runs fine on my local docker and anything non-k8s

[davidwindell]

Ah, I've just tried to force the container to run as UID 0 (root) and that worked:

docker run -u 0 registry.centos.org/centos/httpd-24-centos7

tldr; setting the below on the plugin registry in our helm chart fixes this for me

securityContext:
          runAsUser: 0

I'd really like to understand why a simple docker run on our K8's cluster throws the error but locally it doesn't!

[amisevsk]

Thanks for the analysis @davidwindell -- looks like it's still a bug somewhere, since running as root should not be necessary.

[davidwindell]

Just FYI this is also now happening on the latest devfile-registry images (not just plugin-registry). Something must have changed upstream that broke running as non-root.

[davidwindell]

This is still an issue with 7.5.1

[davidwindell]

Also reproduced in 7.6.0. I think it could be something to do with the permissions of the /var/log/httpd24 folder which is current drwxrwx--- 2 default root 4096 Jan 2 11:09 httpd24.

The helm chart doesn't by default specify a runasuser

[amisevsk]

@davidwindell From my experience, the user that runs in k8s/OpenShift container is a member of the root group (and so should have write permissions to httpd24). Can you confirm this is the case in your cluster (i.e. open a terminal into the container and check group number)?

[davidwindell]

@amisevsk here's the output when I force the image to run with a custom entrypoint:

[davidwindell]

It looks like this is now fixed somehow, closing.