SpacesSavedObjectsClient.openPointInTimeForType throws in usage collectors

[rudolf]

Starting with 7.16.x many clusters have stack traces like the following:

Error: Bad Request
    at Function.createBadRequestError (/usr/share/kibana/src/core/server/saved_objects/service/lib/errors.js:80:36)
    at SpacesSavedObjectsClient.openPointInTimeForType (/usr/share/kibana/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.js:257:48)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at PointInTimeFinder.open (/usr/share/kibana/src/core/server/saved_objects/service/lib/point_in_time_finder.js:144:11)
    at PointInTimeFinder.find (/usr/share/kibana/src/core/server/saved_objects/service/lib/point_in_time_finder.js:99:5)
    at getStats (/usr/share/kibana/src/plugins/vis_types/vega/server/usage_collector/get_usage_collector.js:94:20)
    at UsageCollector.fetch (/usr/share/kibana/src/plugins/vis_types/vega/server/usage_collector/register_vega_collector.js:34:11)
    at /usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:104:21
    at async Promise.all (index 31)
    at Object.bulkFetch (/usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:91:25)
    at getKibana (/usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_kibana.js:68:17)
    at async Promise.all (index 3)
    at /usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_local_stats.js:79:76
    at async Promise.all (index 0)
    at getLocalStats (/usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_local_stats.js:78:10)
    at async Promise.all (index 0)
    at Object.getStatsWithXpack [as statsGetter] (/usr/share/kibana/x-pack/plugins/telemetry_collection_xpack/server/telemetry_collection/get_stats_with_xpack.js:27:48)
    at TelemetryCollectionManagerPlugin.getUsageForCollection (/usr/share/kibana/src/plugins/telemetry_collection_manager/server/plugin.js:295:19)
    at TelemetryCollectionManagerPlugin.getStats (/usr/share/kibana/src/plugins/telemetry_collection_manager/server/plugin.js:229:29)
    at /usr/share/kibana/src/plugins/telemetry/server/routes/telemetry_usage_stats.js:42:21
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9) {
  isBoom: true,
  isServer: false,
  data: null,
  output: {
    statusCode: 400,
    payload: { statusCode: 400, error: 'Bad Request', message: 'Bad Request' },
    headers: {}
  },
  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/badRequest'
}

It appears as if usage collection fails in SpacesSavedObjectsClient.openPointInTimeForType. The root cause might be that we've switched many usage collectors from using a custome ES client (which wasn't space aware) to using the saved objects client which is space aware #99023

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

And the telemetry SO client adds namespace: * to the find requests. We added support to it in 7.16.0 #99044 (could there be a bug in it?)

[rudolf]

It's also explicitly added in the usage collector

https://github.com/elastic/kibana/blob/main/src/plugins/vis_types/vega/server/usage_collector/get_usage_collector.ts#L73-L77

What I don't understand is how this triggers a Bad Request. Either we're hitting https://github.com/elastic/kibana/blob/main/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.ts#L331 but I can't see where the 403 error would come from?

Or we're hitting https://github.com/elastic/kibana/blob/main/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.ts#L337 but when specifying namespaces: [*] we couldn't have 0 namespaces matching, there always has to be a default space.

[afharo]

May be a duplicate of #120422

[rudolf]

Yeah closing as a duplicate of #120422