Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for ES datastream with empty namespace or dataset #17883

Closed
Hexta opened this issue Jul 6, 2023 · 1 comment · Fixed by #18193
Closed

Support for ES datastream with empty namespace or dataset #17883

Hexta opened this issue Jul 6, 2023 · 1 comment · Fixed by #18193
Labels
sink: elasticsearch Anything `elasticsearch` sink related type: feature A value-adding code addition that introduce new functionality.

Comments

@Hexta
Copy link
Contributor

Hexta commented Jul 6, 2023

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

There is already existing ES index named logs-frontend.

We need to configure Vector to send logs to this index.

Attempted Solutions

Configuring data_stream.namespace = "" leads in ES index name logs-frontend- instead of logs-frontend

Proposal

Replace fixed index name format type-dataset-namespace with:

  • namespace is empty: type-dataset
  • dataset is empty type

Check invariants:

  • non-empty namespace requires non-empty type and dataset
  • non-empty dataset requires non-empty type

References

No response

Version

0.30.0
@Hexta Hexta added the type: feature A value-adding code addition that introduce new functionality. label Jul 6, 2023
@dsmith3197 dsmith3197 added the sink: elasticsearch Anything `elasticsearch` sink related label Jul 6, 2023
@dsmith3197
Copy link
Contributor

@Hexta,

Thanks for pointing this out. This should be a relatively straightforward fix.

vector/src/sinks/elasticsearch/config.rs

Line 464 in e9f21a9

Some(format!("{}-{}-{}", dtype, dataset, namespace))

Hexta added a commit to Hexta/vector that referenced this issue Aug 9, 2023
@Hexta
enhancement(elasticsearch sink): Allow empty data_stream fileds
3263525 
 Add support for ES datastream with empty namespace or dataset.

 closes: vectordotdev#17883

 Signed-off-by: Artur Malchanau <artur.molchanov@bolt.eu>
@Hexta Hexta mentioned this issue Aug 9, 2023
Merged
Hexta added a commit to Hexta/vector that referenced this issue Aug 9, 2023
@Hexta
enhancement(elasticsearch sink): Allow empty data_stream fileds
46d9cba 
 Add support for ES datastream with empty namespace or dataset.

 closes: vectordotdev#17883

 Signed-off-by: Artur Malchanau <artur.molchanov@bolt.eu>
github-merge-queue bot pushed a commit that referenced this issue Aug 10, 2023
@Hexta
enhancement(elasticsearch sink): Allow empty data_stream fields (#18193
1dd7bb1 
)

enhancement(elasticsearch sink): Allow empty data_stream fileds

 Add support for ES datastream with empty namespace or dataset.

 closes: #17883

 Signed-off-by: Artur Malchanau <artur.molchanov@bolt.eu>
neuronull pushed a commit that referenced this issue Aug 16, 2023
@Hexta @neuronull
enhancement(elasticsearch sink): Allow empty data_stream fields (#18193
b63ffa8 
)

enhancement(elasticsearch sink): Allow empty data_stream fileds

 Add support for ES datastream with empty namespace or dataset.

 closes: #17883

 Signed-off-by: Artur Malchanau <artur.molchanov@bolt.eu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sink: elasticsearch Anything `elasticsearch` sink related type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

enhancement(elasticsearch sink): Allow empty data_stream fields Hexta/vector
2 participants
@Hexta @dsmith3197