[BUG] 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"

[HugoKuo]

The filebeat shows this error after upgrading the OpenSearch to 2.0.0

2022-05-31T23:38:55.864Z ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

To Reproduce
Upgrade the OpenSearch from 1.3 to 2.0

• OS: CentOS 7.9
• Version : OpenSearch 2.0.0
• Filebeat 7.10.2-1 and 7.12.1-1

Are these related?

#2979 (comment)
#3131

[Wadlazky]

Same here.
OpenSearch from 1.3.2 to 2.0.0
Filebeat 7.12.1 and 7.17.3

[dreamer-89]

Thanks @HugoKuo and @Wadlazky for sharing this issue.

Looks like the Filebeat uses docType in bulk request metadata. As types are removed from OpenSearch 2.0, the request fails during Bulk request parsing, which is expected.

I suggest upgrading to latest OpenSearch clients for compatibility as legacy (ES) clients are not supported with OpenSearch 2.0+

[dreamer-89]

On closer look, _type field is defined with omitEmpty tag in BulkMeta struct, which means if _type is nil then _type should be absent in converted json bulk request.

@HugoKuo Can you please share one beat event which is been used for ingestion to OpenSearch.

[nknize]

@dreamer-89 is right. The error is tripped from the BulkRequestParser.

The new errorOnType parameter was added to help alleviate these bwc errors but since we're relying on upstream Beats here it looks like this wasn't caught before release. The default for errorOnType was set to true when it was merged to main and backported to 2.0 but we never followed up to set this to false for client bwc because there is no REST versioning mechanism in OpenSearch to dynamically switch based on client request. I think a simple short term fix is to unblock this issue is to default errorOnType to false in a 2.0 patch release and follow up w/ a REST API versioning mechanism in 2.2 (or 2.1 if it can make the release date).

Before releasing this as a 2.0.1 patch we need to check that this is the only incompatibility and there aren't more lingering that haven't yet been reported.

[msillence]

same issue I belive with pubsubbeats:

2022-06-16T13:27:43.690Z ERROR elasticsearch/client.go:344 Failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

[msillence]

pubsubbeats now works with 2.0.1 if you set "compatibility.override_main_response_version" : true

[dreamer-89]

Thank you @msillence for the confirmating that fix is working.

@Wadlazky @HugoKuo : Can you please try OpenSearch 2.0.1 with compatibility.override_main_response_version set to true.

[dreamer-89]

A brief note on the fix deployed.
Initially typed end-points were added back but later reverted as it was identified that integration is breaking at multiple places due to version check mismatch (more details in #3530). So, adding back typed end-points was not sufficient. Instead, version override cluster setting change was added back, that spoofs OpenSearch engine to 7.10.2 and temporarily fixes the issue.

[Wadlazky]

Thanks all for taking care of it.
I tried compatibility.override_main_response_version: true in opensearch.yml and it works.
OpenSearch from 1.3.2 to 2.0.1
Filebeat 7.12.1

[HugoKuo]

confirmed it's working with 2.0.1 now.

[nanolonny]

Working for me too in 2.0.1 with compatibility.override_main_response_version set to true. Does not work without this. Please update documentation: https://opensearch.org/docs/latest/clients/agents-and-ingestion-tools/index/
I went through a whole lot of pain dealing with this issue before coming across this Github issue. Updating the main documentation will surely spare others.

[dreamer-89]

Thanks @nanolonny for pointing this and apologies for delay on documentation part. Certainly updated doc will save a lot of time and effort. There is some discussion happened on PR

Pinging @alicejw-aws @hdhalter for visibility.

[dreamer-89]

Documentation issue tracking this: opensearch-project/documentation-website#741

[iqra-shafiq]

I tried compatibility.override_main_response_version: true in opensearch.yml and it works.
Opensearch 2.5.0
Filebeat 7.10.2

[rajatsharma4318]

I was facing the same issue Just added Suppress_Type_Name On in the fluent-bit.conf to make the connection with open-serach.
[OUTPUT]
Name opensearch
Match *
Host localhost
Port 9200
Index fluent-bit-sds-wrapper
tls On
tls.verify Off
http_user admin
http_passwd admin
Suppress_Type_Name On

Ref:https://docs.fluentbit.io/manual/pipeline/outputs/opensearch#action-metadata-contains-an-unknown-parameter-type

[ayodeji-awe]

opensearch.yml
Was this done for a AWS managed Opensearch?

[mithlajkn]

@ayodeji-awe yes... filebeat 7.12.1
aws opensearch : 2.5.0
I tried compatibility.override_main_response_version: true and fixed the issue by

curl -X PUT "http://your-opensearch-host:9200/_cluster/settings" -H "Content-Type: application/json" -d '{
  "persistent": {
    "compatibility.override_main_response_version": true
  }
}'

reference : https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html