Kibana logs and Kibana pid configuration settings seem to be ignored in 7.11

[willemdh]

Kibana version: 7.11.0

Elasticsearch version: 7.11.0

Server OS version: CentOS 7

On our DV Elastic, we tested 7.11 and noticed the Kibana logs were no longer generated in the expected location. After some debuging, it seems the logs are written to the default path /var/log/kibana/kibana.log instead of to the configured logging.dest /logdisk/kibana/kibana.json. We also noticed that there no kibana.pid file in the configured location /etc/kibana/pid/kibana.pid.

Relevant kibana.yml configuration:

logging.dest: /logdisk/kibana/kibana.json
logging.json: true
logging.timezone: Europe/Brussels
pid.file: /etc/kibana/pid/kibana.pid

We did find a kibana.pid file in the following location: /var/run/kibana/kibana.pid.

it's important that we can write the Kibana logs to a separate disk to prevent the os disk from filling up.

https://discuss.elastic.co/t/kibana-7-11-0-kibana-pid-en-kibana-json-not-created-in-configured-locations/264694

[willemdh]

It seems like only /var/log/kibana/kibana.log is being logged to. Can anyone confirm this issue is not only in my installation?

Updated to 7.11.1 in the meantime, but same issue..

[willemdh]

Anyone?

[boutetnico]

@willemdh I'm seeing the same issue since updating to Kibana 7.11.0 and 7.11.1. Haven't tried 7.11.2 yet.

[willemdh]

We are on 7.11.2 and still have this issue.

[boutetnico]

7.12.0 has this issue too.

[n0othing]

As of 7.11, the RPM's systemd unit file passes in --pid.file="/run/kibana/kibana.pid" (see [1]) and that appears to take priority over pid.file in the kibana.yml.

Until this is fixed, a reasonable workaround would be to add in an override (e.g systemctl edit kibana.service).

[1] https://github.com/elastic/kibana/pull/82209/files

[willemdh]

Imho is the main problem here not the pid, but the fact that the Kibana logfile logs to the default location? This can possibly result in a full os disk.. Thats why we log Kibana logs to a separate disk.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

cc @elastic/kibana-operations

See https://github.com/elastic/kibana/pull/82209/files, issue is caused by the way we package the distribution, not by core's configuration system.

[jbudz]

Confirming, we'll need to introduce a fix by moving this unit file args to kibana.yml

[hjmwijnen]

It looks like after log file rotation all our user sessions are getting lost. Could this be related? https://discuss.elastic.co/t/session-lost-after-log-file-rotation/273531/2

This problem did't appear in 7.10, and it does appear in 7.12.1. I'm not able to verify if it also started to appear in 7.11, but that would be very well possible.

My current logging settings are:

logging.dest: "/var/log/kibana/kibana.log"
logging.rotate.enabled: true
logging.quiet: true
logging.verbose: false

My session settings are:

xpack.security.session.idleTimeout: "7d"
xpack.security.session.lifespan: "1Y"

The last log lines in the kibana.log (before rotation) look like this:

{"type":"log","@timestamp":"2021-05-18T12:47:20+02:00","tags":["info","cli","config"],"pid":440,"message":"Reloading logging configuration due to SIGHUP."}
{"type":"log","@timestamp":"2021-05-18T12:47:20+02:00","tags":["info","config"],"pid":440,"message":"New logging configuration:\n{\n "ops": {\n "interval": 5000\n },\n "logging": {\n "dest": "/var/log/kibana/kibana.log",\n "rotate": {\n "enabled": true,\n "everyBytes": 10485760,\n "keepFiles": 7,\n "pollingInterval": 10000,\n "usePolling": false\n },\n "verbose": false,\n "silent": false,\n "quiet": false,\n "events": {},\n "filter": {},\n "json": true\n }\n}"}