Elasticsearch sink with api_version set to "auto" does not recognize the API version of ES6 as V6

[syedriko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The integration test below fails when running against ES 6, but passes when running against ES 7 or ES 8:

---- sinks::elasticsearch::integration_tests::structures_events_correctly stdout ----
2023-04-26T18:08:38.811967Z ERROR vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;\"},\"status\":400}" internal_log_rate_limit=true
2023-04-26T18:08:38.812176Z ERROR vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-04-26T18:08:38.812340Z ERROR vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being rate limited.
thread 'sinks::elasticsearch::integration_tests::structures_events_correctly' panicked at 'Failed to assert compliance, errors:
  - Missing event `BytesSent`
  - Missing metric `component_sent_bytes_total{endpoint,protocol}`
', src/test_util/components.rs:418:16

Hardcoding the API version to V6 'fixes' the test:

diff --git a/src/sinks/elasticsearch/integration_tests.rs b/src/sinks/elasticsearch/integration_tests.rs
index 2adc6eec7..3ad06a9a6 100644
--- a/src/sinks/elasticsearch/integration_tests.rs
+++ b/src/sinks/elasticsearch/integration_tests.rs
@@ -139,6 +139,7 @@ async fn structures_events_correctly() {
         id_key: Some("my_id".into()),
         compression: Compression::None,
         batch: batch_settings(),
+        api_version: ElasticsearchApiVersion::V6,
         ..Default::default()
     };
     let common = ElasticsearchCommon::parse_single(&config)

This suggests that Elasticsearch sink with api_version set to "auto" does not recognize the API version of ES6 as V6.

Configuration

No response

Version

vector 0.30.0-custom-752d4245c (x86_64-unknown-linux-gnu debug=full)

Debug Output

No response

Example Data

No response

Additional Context

At a closer look, the elasticsearch sink code looks for the version of the ES server it is running against at the following path: _cluster/state/version, hoping to find the ES version in the version field. Here's what is actually found at that resource:

ES6

> curl -k http://localhost:9200/_cluster/state/version
{"cluster_name":"docker-cluster","cluster_uuid":"ZfBjZ9ZdTJSoHwpFn1JpFg","version":25,"state_uuid":"ITOVSGJeRuq0Oq7HrjBevA"}

ES7

>curl -k http://localhost:9200/_cluster/state/version
{"cluster_name":"docker-cluster","cluster_uuid":"iXRj75GjSeWKYX04oQq3Ww","version":45,"state_uuid":"sbQ5DvBoRMqEOk-WtcXP7Q"}

ES8

>curl -k http://localhost:9200/_cluster/state/version
{"cluster_name":"docker-cluster","cluster_uuid":"00Bw-HrAQ_GHtEnb6_A8tA","version":25,"state_uuid":"Mj1Lw_DlRaSSOMM_I5n5kg"}

According to the ES API docs, the 'version' here is 'cluster state version' and does not seem to reflect the version of the ES server.

GETting the root path does get the version of the ES server:

ES6

>curl -k http://localhost:9200
{
  "name" : "f794W4s",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "pUfqzdGhRKaQdICVzUz45A",
  "version" : {
    "number" : "6.8.23", <<<<<<<<<<<<<<<<<<
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "4f67856",
    "build_date" : "2022-01-06T21:30:50.087716Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

ES7

>curl -k http://localhost:9200
{
  "name" : "1b177aa6bfd4",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "BHluRHdFS0OPq6XpXUMf4A",
  "version" : {
    "number" : "7.17.9", <<<<<<<<<<<<<<<<<<
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

ES8

>curl -k http://localhost:9200
{
  "name" : "5e1bf94b0586",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "00Bw-HrAQ_GHtEnb6_A8tA",
  "version" : {
    "number" : "8.7.0", <<<<<<<<<<<<<<<<<<
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "09520b59b6bc1057340b55750186466ea715e30e",
    "build_date" : "2023-03-27T16:31:09.816451435Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

References

No response

[syedriko]

There is another issue with Elastic API version auto-detection that this issue and its PR does not address.
At the moment, version auto-detection happens at config parsing time and if the Elastic server is not available, we're going to hit

vector/src/sinks/elasticsearch/common.rs

Line 151 in 1e43208

Err(error) => {

, which is going to assume a version, which can end up incorrect come time to talk to Elastic.
It appears that version auto-detection needs to happen not at config parsing time but right after we've established a connection with Elastic.

[spencergilbert]

It appears that version auto-detection needs to happen not at config parsing time but right after we've established a connection with Elastic.

@syedriko would you mind opening this as another issue. This may be a larger departure to how Vector has build components so I'd like to keep discussion on that focused 👍

[syedriko]

It appears that version auto-detection needs to happen not at config parsing time but right after we've established a connection with Elastic.

@syedriko would you mind opening this as another issue. This may be a larger departure to how Vector has build components so I'd like to keep discussion on that focused +1

@spencergilbert Sure: #17279

[spencergilbert]

Should be resolved in #17227