[dev/cli] switch to static basePath, send sockets to forks rather than running HTTP proxy

[spalger]

I'm currently investigating the way that dev cli works and how we can start to improve the DX. Part of that process would be decoupling the dev cli from the core so that we can work on the code more easily and have more control over what is happening in the parent process. The primary reason for the coupling between the dev cli and core is the BasePathProxy, which previously was independent of the Kibana Server but that lead to a number of issues with request handling being different between the Kibana Server in production/development, but now the BasePathProxy is developed as an extension of the Kibana Server in core.

I'd like to suggest that we choose another method of preventing the BasePathProxy from diverging from the Kibana Server: the BasePathProxy shouldn't process requests.

The secondary purpose of the BasePathProxy is to delay requests while the server is restarting or the @kbn/optimizer is running, so that when users make changes to the code and refresh their browser they always get the most recent version of the code they have saved. This is an important feature, but one that could be implemented at the TCP level and doesn't need to process or parse requests in order to work.

Additionally, as described in #79748, we would like to run multiple Kibana instances in a high-availability configuration. An option for this is to run a proxy which sends http requests to the Kibana Server. Another option is to fork the servers from a parent process and send them handles to individual TCP connections via IPC. In the later model the child process would consume the HTTP request directly from the connection and process the request as if it was received directly, which I think is the approach we should take rather than parsing requests in the parent process and making secondary requests to backend instances of the Kibana Server. In this model I don't expect we'll be able to consume any of the information from the HTTP request in the BasePathProxy unless we do some really funky junk.

To accommodate these requests and requirements I think we should transition away from using random basePaths (most people hate them anyway) and instead use a static basePath which the dev-cli will make sure is synchronized between the Kibana Servers. This will break the redirects we do now between /abc/app/kibana and /xyz/app/kibana, but if the basePath is static we don't need those redirects anyway. Additionally, we will be able to implement simple SSL configuration handling in the dev-cli, maybe powered by @kbn/config or maybe just handled by parsing the kibana.dev.yml and kibana.yml files ourselves similar to how we have a simpler, less feature complete, plugin discover implementation for dev tooling.

Thoughts?

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[kobelb]

Given that the Kibana Core team is planning to investigate #68626 during 7.x, I think we can ignore the HA requirements for now. If we're using Node.js clustering in development, we don't really need the dev-cli to handle this complexity.

I'm honestly not familiar with the current architecture of the base path proxy to have an intelligent opinion on the other benefits we could get from this new architecture.

[tylersmalley]

I am +1 on removing the parsing of requests and using a static path if it would prevent people from disabling it entirely. Could the basePath still be random for functional tests since we don't need the redirect?

[spalger]

@tylersmalley we don't use the basePath for functional tests because we want compatibility with cloud and haven't ever plumbed some sort of automatic basePath change functionality in the tests.

[spalger]

Alright, after digging in more into how cluster works, and doing a POC of this functionality (which totally works) I think going this path will require a lot more work from more people and so I'm instead pursuing other options. Thanks for taking a look @kobelb and @tylersmalley

[spalger]

For historical purposes:

While investigating this option I realized that #79748 is describing a production feature that we might implement in core. This feature would conflict with the approach described for sending sockets to the child process unless we chose to manage socket distribution ourselves rather than relying on the cluster module to automatically manage the connections. I think there are two best options that will lead to the dev CLI being something we can work on in a package:

1. dev CLI parses the kibana.yml file and extracts the necessary config to setup a potentially secure socket server and then sends the connections to that server to the Kibana server. The server would use a subclass of HttpServer as a custom listener, passed to Hapi, which emits those connections so that Hapi will consume the received socket and process the request. To implement HA, core will do something similar to what the dev CLI does, launching processes with child_process.fork() and then routing connections to those processes where they will emit the request on their own listener. This is similar to how the cluster module works on Linux platforms.

   Pros: This will provide the most consistent experience in dev and prod without the dev CLI needing to reproduce the Kibana server in the parent process (except for some settings used to setup the socket server).

   Cons: core will need to configure Hapi with a custom listener, which isn't a well documented feature and might have issues we aren't aware of or cause issues with untested functionality.

2. consume @kbn/config and extract some core components into an @kbn/http-tools package, then run a proxy like we do today in the dev CLI, assigning an alternate port to the Kibana server and mimicking timeouts, ssl, etc in the dev CLI to match the HTTP(s) interface of the Kibana server. Core will then be able to implement Development Server - HA deployment of Kibana #79748 however it wants.

   Pros: Core can use the cluster module directly and possibly write a little less code, though in my experience the cluster module doesn't expose a turn-key solution and a decent amount of code will be needed to manage the cluster regardless.

   Cons: we'll need to extract more parts of the core to be consumable by the dev cli, which I've attempted but am struggling with.

---

I'm no longer planning to extract the dev cli to a package and instead just extract the basePathProxy from core so that we don't need to load more code than necessary in the parent process of the dev CLI.