Duplicate logs getting created on Opensearch through fluentd

[SohailLS]

Describe the bug

Hi Team,

fluentd is generating logs 5 times on Opensearch.
We have tried implementing below solution with "elasticsearch_genid" and its even able to generate _hash values.... but its not able to remove the duplicates.

https://medium.com/redbox-techblog/tuning-fluentd-retries-avoiding-duplicate-documents-in-elasticsearch-e7cb9630a453.

logs flow is as below
fluentbit -> fluentd -> Opensearch

Earlier we were having only fluent-bit and it was working fine.

After introducing fluentd for multiple log files we started facing the issue.

Please advise.

To Reproduce

Please use the attached configs for fluent-bit and fluentd to reproduce the issue.

3ds-fluentbit.txt
td-agent.txt

Expected behavior

Single logs should get reported on the Opensearch end.

Your Environment

- Fluentd version: 1.16.3
- TD Agent version: 4.5.2
- Operating system: RHEL 8.9
- Kernel version: 4.18.0-513.11.1.el8_9.x86_64
- Fluent-bit version: 2.1.5

Your Configuration

PFA

Your Error Log

No error logs.

Additional context

No response

[SohailLS]

Requesting for a quick solution.
Most of our deliverables are stuck due to this issue.

[daipom]

@SohailLS Sorry for my late response.
Could you please send the issue report to https://github.com/fluent/fluent-plugin-opensearch/issues as well?
Because it looks like the issue of fluent-plugin-opensearch.

https://medium.com/redbox-techblog/tuning-fluentd-retries-avoiding-duplicate-documents-in-elasticsearch-e7cb9630a453 says retrying can cause duplication.
The output plugin fails to send data and retries, but Opensearch is actually properly received the data.
This is what is happening in your environment as well, right?

If so, one thing that could be done on the Fluentd itself (not fluent-plugin-opensearch) is to disable retries.
(There is a concern, though, that it is not possible to retry data that actually failed to send...)

You can set retry_max_times 0 to disable retries. (https://docs.fluentd.org/output#control-retrying)

<match xxx>
  ...
  <buffer xxx>
    ...
    retry_max_times 0
  </buffer>
</match>

Or, you can also have it sent to a secondary just in case.
(This adjusts the parameters to switch to the secondary from the first retry.)

<match xxx>
  ...
  <buffer xxx>
    ...
    retry_type periodic
    retry_max_times 3
    retry_secondary_threshold 0.1
    retry_randomize false
  </buffer>
  <secondary>
    ...
  </secondary>
</match>

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[github-actions]

This issue was automatically closed because of stale in 7 days