Support dynamic configuration without restarts

[joshdover]

The Kibana Platform supports watching config files for changes, and then propagating those config changes to Core and Plugins via an observable. The basic plumbing is already in place.

Currently, we only formally support changing the logging configuration at runtime. This is issue is to discuss other configurations we may want to also support changing at runtime without a restart.

While some configurations will be as simple as listening to the config observable and updating accordingly, other configurations (such as enabling and disabling plugins without restarts) will require more work in Core to support.

Please add config keys that need to be able to dynamically change without restarts of the Kibana server here and the Platform team will make sure these get delegated to the proper teams:

• xpack.cloud.apm.url / xpack.cloud.apm.secret_token: they would allow us introduce changes on Cloud and cut quite a bit the time it takes to fully initialize a deployment with APM.
• xpack.security.audit.*: Allow audit logging to be configured/updated at runtime. ES already supports this, so we should too!

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[kobelb]

/cc @ruflin, as we were talking about whether or not it's possible for Kibana to dynamically reload configuration values from the kibana.yml the other day.

[joshdover]

To my surprise, I learned that the existing implementation of config reloading used for logging configuration actually works for all configuration values at the Core API level today.

Any config changes in the kibana.yml (or other config file specified with -c or --config) will be updated by Core and exposed to plugins via the PluginInitializer.config.create() API whenever a SIGHUP signal is received by the main Kibana process. In order to actually support this in the product, plugins need to be sure to subscribe to changes to their config and update any necessary state accordingly.

We could however improve the ergonomics of this feature by actually watching for file changes on the configuration file(s) so that it's not required for orchestrators to send the SIGHUP signal at all. Further research would need to be done on whether or not this would be desirable, is typical of other server-side software, or should be enabled by default.

Any plugins that take advantage of this feature should ideally be testing that it works end-to-end via a functional test before documenting this as supported feature in our public documentation. If/when plugins start adopting this feature, we should also standardize on a way to indicate that a config supports hot-reloads in our documentation.

I would like to note that the log messages from Core (well, actually CLI) are misleading since they only mention "logging configuration":

  log   [19:14:01.239] [info][config][cli] Reloading logging configuration due to SIGHUP.
  log   [19:14:01.246] [info][config][cli] Reloaded logging configuration due to SIGHUP.

Since the primary ask of this issue is already supported from Core, I think we should opt to fix the log message and then close this issue. Further improvements can be considered as separate issues; and only if they come up as a request from users or our Cloud team.