[KPT] Kibana Persistence Toolkit design

[pgayvallet]

This issue is a retranscription of the Kibana Persistence ToolKit (aka KPT) proposal that was initially presented during a company event.

The main problems KPT is trying to solve are:

• Access to ES feature via the SO APIs is too limited
• The SO document format does not play nice with data

Some background

Why does Saved Objects need to restrict access to Elasticsearch features

• Model abstraction layer
  • Required for backward compatibility
  • Hides the underlying persistence implementation (ES)
  • Emulates multiple-types in a single index
• Audit logging
  • Needs to be able to identify which objects are targeted for each operation
• Multiple spaces support
  • Performs preflight checks on write operations
  • Complex to implement with script/query based updates
• Security
  • script (and others) is scary

The consequences of having restricted Elasticsearch features

• SO client only exposes a subset of ES APIs
  • e.g. no update_by_query
• Exposed APIs are restrained
  • SOC.find is a very limited /_search
• Queries must be re-written
  • Makes advanced filtering (KQL) & aggregations hard to implement

{
 aggs: {
   evil: {
     terms: {
       field: 'alert.id',
       min_doc_count: 0,
     }
   }
 }
}

• ES index management features are unavailable
  • ILM, Data Streams
• Index mappings are not ECS-compliant

Features that could be used on other Kibana app-specific indices

• Clear need to be able to leverage some features on other indices
  • RBAC
  • Spaces
• Other features could potentially be used
  • Audit Logging
  • Additive schema migrations
• Some features are too tightly coupled with the SO API or structure
  • Migrations (non-additive)
  • Import & Export
  • HTTP APIs

Impacts of the limitations

• Alerts & Cases
  • RAC group has to workaround these limitations in order to provide visualizations on alert & case data
  • Alerting had to implement their own RBAC
  • Cases will need to implement custom RBAC for inherited permissions between related objects
• Notification Center
  • ILM cannot be leveraged for pruning old notifications or moving them into cold storage
• Non exhaustive, need to find impacts on other solutions

The Kibana Persistence Toolkit

The goal of KPT is to provide a low level persistence API, that would support RBAC, Spaces and possibly Audit Logging, that would be leveraged by the SO implementation, and could also be used directly with app-specific indices.

Some characteristics of the KPT layer:

• Thin layer over a raw elasticsearch client
  • Built-in support for Spaces, RBAC and Audit Logging
  • Supports customizable RBAC filtering
• Accepts raw ES requests and returns raw ES responses
  • Mirrors the @elastic/elasticsearch client signatures
  • Transforms queries under-the-hood to add RBAC and Spaces capabilities
• Can be used with app-specific indices or with Saved Objects
  • For Saved Objects, the underlying document model isn't planned to change
• Only exposed server-side
  • Requires type owners to write their own domain-based HTTP APIs (see [discuss] Plugin / SavedObjects architecture #72028)
• As the APIs get implemented, the SO service will leverage them
  • Plan to develop a _search API first, that SOR.find will use

Architecture changes

Saved Objects

App-specific indices

What KPT is not

• A full replacement for Saved Objects
  • Saved Objects APIs will gradually leverage it
  • Both APIs can be used interchangeably
• A solution to have all SO features on app-specific indices
  • Only provides RBAC, Spaces, and Audit Logging
  • No migrations
  • No import & export
• A way to have custom data formats on SO types
  • SO document format remains unchanged
  • However, when used against SO indices, consumers will have to use the underlying document model
    • e.g. alert.group.id instead of alert.attributes.group.id

Usage examples (non-final API)

Creating a scoped KPT client

// for any arbitrary app-specific index
const scopedClient = core.persistence
 .getClientFactory({
   namespaceField: 'namespace',
   typeField: 'type',
 })
 .forType('alert')
 .asScoped(request);

// shorter version for savedObjects
const scopedClient2 = core.savedObjects.getPersistenceClient('alert').asScoped(request);

const { body } = scopedClient.search({
 index: '.my-alert-index',
 body: {
   // security filter will be added under the hood
   query: { term: { 'alert.group.id': 42 } },
 },
});

const serializer = core.savedObjects.createSerializer();
const myAlerts = body.hits.hits.map(serializer.rawToSavedObject);

Creating a custom security filter

During the transition period, an API will be added to allow consumers to create ES filters to apply security and spaces restriction. The API could look like

Note: The example is simplified here, the real API needs to handle a little more than that, such as multi-space support

type SecurityFilterFactory = (options: {
 namespaceField: string;
 typeField: string;
}) => SecurityFilterBuilder;

interface SecurityFilterBuilder {
 create(options: { namespace: string; type: string }): estypes.QueryContainer;
}

const filterFactory = core.persistence.getFilterFactory({
 namespaceField: 'namespace',
 typeField: 'type',
});
const securityFilter = filterFactory.create({
 namespace: 'default',
 type: 'alert',
});

esClient.search({
 body: {
   query: { bool: { filter: securityFilter } },
 },
});

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

cc @elastic/kibana-security @kobelb

[jportner]

We had some discussion on Slack today about potential "soft deletion" of saved objects. This would allow us to implement a sort of Trash feature where an object gets permanently removed after a period of time (30 days?)

A quick search turned up one such existing ER: #89564
Perhaps with the KPT redesign it would be easier to implement such a feature. Thoughts?

[pgayvallet]

Perhaps with the KPT redesign it would be easier to implement such a feature. Thoughts?

Given that KPT is supposed to be a lower level wrapper than the SO APIs, I'd say the opposite to be honest 😅 . If I can see such soft deletion being a feature of a high-level API such as SOs, I personally don't think a low level wrapper such as KPT could and/or should hold such responsibilities.

@rudolf wdyt?

[lukeelmers]

I tend to agree with @pgayvallet here: While I'm on board with the idea of soft deletes, and it seems like it would be a useful feature to implement, it does feel like a higher-level concern than the KPT and would probably make more sense as (yet another) addition to the existing SO APIs. But let's see what everyone else thinks 🙂

[rudolf]

I think the important thing is that nothing in KPT might prevent us from implementing soft-deletes.

Architecturally I agree it would be better in a layer higher than KPT. If KPT exposed scripted updates and scripted update by query then we would probably have the required building blocks to create fairly performant soft-deletes.