[POC] multi-process Kibana with Node clustering

[lukeelmers]

This is a POC of #68626, based on the final RFC #94057. It was forked from the original POC (#93380).

Main differences from the main POC include:

• Updating configs based on the approved RFC
• Changing naming per RFC clustering -> node
• Adding experimental logging integration, where worker processes send their log messages to the coordinator for appending.

In particular the logging implementation needs more thought. At the moment the logic is hacked into BaseLogger, with a conversion process that makes each LogRecord serializable. The conversion process has potential to negatively impact performance as it happens for every record; we may want to consider broader changes to the LogRecord interface itself to avoid the need for it.

Rather than hacking BaseLogger, it may be worth considering an internal "IPC appender" that handles making the record serializable and sending the IPC message for the coordinator.

Also more thought needs to be put into how the coordinator handles augmenting each record with worker-specific information. This is where a logging MDC could potentially come in handy.

[kibanamachine]

⏳ Build in-progress, with failures

• continuous-integration/kibana-ci/pull-request
• Commit: 5368bf7
• Storybooks Preview
• This comment will update when the build is complete

Failed CI Steps

• Execute xpack-firefox
• Execute kibana-ciGroup8
• Jest Unit Tests
• Execute xpack-savedObjectsFieldMetrics
• Execute xpack-savedObjectsFieldMetrics
• Execute kibana-ciGroup10
• API Integration Tests
• Execute xpack-kibana-ciGroup10
• Execute xpack-kibana-ciGroup9
• Execute xpack-kibana-ciGroup2
• Execute kibana-ciGroup1
• Execute kibana-ciGroup6
• Execute xpack-kibana-ciGroup5
• Execute xpack-kibana-ciGroup1
• Execute kibana-ciGroup9
• Check Doc API Changes
• serverIntegration
• Execute kibana-ciGroup4
• Execute xpack-kibana-ciGroup8
• Execute xpack-accessibility
• Execute kibana-ciGroup2
• Check Types and Public API Docs
• Execute xpack-kibana-ciGroup7
• Execute oss-firefox
• Execute kibana-ciGroup5
• Execute oss-pluginFunctional
• Execute kibana-ciGroup7
• Execute xpack-kibana-ciGroup11
• Execute xpack-kibana-ciGroup3
• Execute kibana-ciGroup12
• Execute xpack-kibana-ciGroupDocker
• Report failed tests, if necessary
• Execute oss-accessibility
• Execute kibana-ciGroup3
• Execute xpack-kibana-ciGroup4
• Execute xpack-kibana-ciGroup6
• Execute xpack-kibana-ciGroup13
• Execute xpack-kibana-ciGroup12

---

Test Failures

Jest Tests.src/core/server/logging.uses default memory buffer logger until config is provided

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

TypeError: Cannot read property 'isEnabled' of undefined
    at BaseLogger.log (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:63:23)
    at BaseLogger.trace (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:35:10)
    at LoggerAdapter.trace (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger_adapter.ts:28:17)
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logging_system.test.ts:48:10)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:106:37)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:45:12
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:28:19)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:75:41
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

---

Jest Tests.src/core/server/logging.flushes memory buffer logger and switches to real logger once config is provided

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

TypeError: Cannot read property 'isEnabled' of undefined
    at BaseLogger.log (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:63:23)
    at BaseLogger.trace (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:35:10)
    at LoggerAdapter.trace (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger_adapter.ts:28:17)
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logging_system.test.ts:65:10)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:106:37)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:45:12
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:28:19)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:75:41
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

---

Jest Tests.src/core/server/logging.appends records via multiple appenders.

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

TypeError: Cannot read property 'isEnabled' of undefined
    at BaseLogger.log (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:63:23)
    at BaseLogger.info (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger.ts:43:10)
    at LoggerAdapter.info (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logger_adapter.ts:36:17)
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/logging/logging_system.test.ts:97:23)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:106:37)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:45:12
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:28:19)
    at /var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:75:41
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

---

and 20 more failures, only showing the first 3.

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream