Agent file permission issue when upgrading from ECK 2.3 to 2.4

[rigl1]

Bug Report

What did you do?
Tried to Upgrade ECk operator from 2.3 to 2.4
What did you expect to see?
Rolling Update of cluster ressources (Fleet server, elastic-agent)
What did you see instead? Under which circumstances?
CrashLoopBackOff of containers

Environment
Openstack

• ECK version: 2.3

  insert version or git commit id here

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:
  Openstack privat cloud

  for each of them please give us the version you are using

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11", GitCommit:"38d3c1f3d5306401bcf39a71bad3b5a5106033d7", GitTreeState:"clean", BuildDate:"2022-03-16T14:02:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
v1.21.11

• Resource definition:

if relevant insert the resource definition

• Logs:

Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Error: preparing STATE_PATH(/usr/share/elastic-agent/state) failed: mkdir /usr/share/elastic-agent/state/data: permission denied
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
insert operator logs or any relevant message to the issue here

│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.831Z","log.logger":"agent-es","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"55", │
│ "namespace":"elastic-cluster","agent_name":"fleet-server","took":0.000279918} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.831Z","log.logger":"agent-fleetserver","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","itera │
│ tion":"74","namespace":"elastic-cluster","agent_name":"fleet-server"} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.831Z","log.logger":"agent-fleetserver","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iterati │
│ on":"74","namespace":"elastic-cluster","agent_name":"fleet-server","took":0.000072987} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.871Z","log.logger":"agent-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteratio │
│ n":"154","namespace":"elastic-cluster","agent_name":"fleet-server","took":0.157195099} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.871Z","log.logger":"agent-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iterat │
│ ion":"155","namespace":"elastic-cluster","agent_name":"fleet-server"} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.889Z","log.logger":"agent-kibana","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":" │
│ 57","namespace":"elastic-cluster","agent_name":"fleet-server","took":0.073012416} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:05.934Z","log.logger":"agent-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteratio │
│ n":"155","namespace":"elastic-cluster","agent_name":"fleet-server","took":0.062874746} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:17.173Z","log.logger":"agent-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iterat │
│ ion":"156","namespace":"elastic-cluster","agent_name":"fleet-server"} │
│ {"log.level":"info","@timestamp":"2022-12-03T12:03:17.238Z","log.logger":"agent-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteratio │
│ n":"156","namespace":"elastic-cluster","agent_name":"fleet-server","took":0.065490427}

[BenB196]

You're probably hitting this: #5993

[thbkrkr]

Hello,

Yes, it is very likely related to the breaking change made in 2.4:

• Configure Elastic Agent host path volume to point to correct path #5890

As BenB196 pointed out in #5993, we forgot to document the need to run the container as root when using the hostpath. An alternative to not running as root is to use an emptyDir volume.

The documentation has now been updated to reflect this:

• Update agent docs to mention root user requirement #6147

Apologies for the inconvenience.

Feel free to reopen if there is anything else.