[Ingest Manager] Manually build Fleet kuery with Node arguments

[nchaulet]

Summary

This PR #75693 introduced a way to manually build filter instead of parsing the KQL string to improve perfomance. While improving a lot the performance there is still room for improvment if we use buildNodeWithArgumentNodes.

Benchmarking the potential improvment

  const suite = new Benchmark.Suite();

  suite
    .add('parse KQL', function () {
      esKuery.fromKueryExpression(
        'not fleet-agent-actions.attributes.sent_at: * and fleet-agent-actions.attributes.agent_id:1234567'
      );
    })
    .add('buildNode', function () {
      esKuery.nodeTypes.function.buildNode('and', [
        esKuery.nodeTypes.function.buildNode(
          'not',
          esKuery.nodeTypes.function.buildNode('is', 'fleet-agent-actions.attributes.sent_at', '*')
        ),
        esKuery.nodeTypes.function.buildNode(
          'is',
          'fleet-agent-actions.attributes.agent_id',
          '1234567'
        ),
      ]);
    })
    .add('buildNodeWithArgumentNodes', function () {
      nodeTypes.function.buildNodeWithArgumentNodes('and', [
        nodeTypes.function.buildNode(
          'not',
          nodeTypes.function.buildNodeWithArgumentNodes('is', [
            nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at`),
            nodeTypes.wildcard.buildNode(nodeTypes.wildcard.wildcardSymbol),
            nodeTypes.literal.buildNode(false),
          ])
        ),
        nodeTypes.function.buildNodeWithArgumentNodes('is', [
          nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.agent_id`),
          nodeTypes.literal.buildNode('123456'),
          nodeTypes.literal.buildNode(false),
        ]),
      ]);
    })
    // add listeners
    .on('cycle', function (event) {
      console.log(String(event.target));
    })
    .on('complete', function () {
      console.log('Fastest is ' + this.filter('fastest').map('name'));
    })
    // run async
    .run({ async: true, minSamples: 200 });

Give the following result

parse KQL x 1,005 ops/sec ±4.58% (81 runs sampled)
buildNode x 6,143 ops/sec ±4.75% (84 runs sampled)
buildNodeWithArgumentNodes x 905,728 ops/sec ±5.77% (84 runs sampled)
Fastest is buildNodeWithArgumentNodes

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[kobelb]

Very nice! I did not know this was an option. @lukasolson, you're the KQL expert, does this make sense to you?

[lukasolson]

Hmm, if this performance improvement is really necessary, this is fine, but I'd generally like to avoid using buildNodeWithArgumentNodes outside of the grammar.peg file, since it leads to worse readability.

[kobelb]

@lukasolson perhaps we should address this by providing a better interface for building these KQL queries which hide the intricacies? I believe you mentioned considering implementing this before, is there an existing GitHub issue for that?

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@lukasolson Do you think we can move forward with this PR?

[lukasolson]

Yes, it's fine. I've opened #77085 to track building a better programmatic API.

[nchaulet]

@lukasolson thanks for the issue, a simple API like the one you put in the issue will be amazing :)

There is one change to the WildcardType in this PR if you have time for a review

[nchaulet]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7067bba

Build metrics

✅ unchanged

History

• 💚 Build #73206 succeeded ee2dd65
• 💚 Build #72120 succeeded 7ed80d5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream