Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add target option to jdbc input #69

Merged

Conversation

andsel
Copy link
Contributor

@andsel andsel commented May 18, 2021

This PR adds the target option to configure destination field inside the event to the JDBC input.
When ECS is enabled and the target is not specified it log a warn message.

Related #19

@andsel andsel changed the title Feature/enable ecs for jdbc input Add target option to jdbc input May 19, 2021
@andsel andsel marked this pull request as ready for review May 19, 2021 08:58
@elasticsearch-bot elasticsearch-bot self-assigned this May 19, 2021
Copy link
Contributor

@kares kares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 code-wise looking good,
we'll need some input on "standardizing" the missing target message


# target must be populated if ecs_compatibility is not :disabled
if @target.nil? && ecs_compatibility != :disabled
logger.warn("When ECS compatibility is enabled also target option must be valued")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

believe we agreed upon this being logger at the info level (as a compromise since it might be legit to skip setting target in cases where data has no conflicts with the schema).

what I would be also interested, due other plugins, is a standard message we would use across plugins ... draft:
ECS compatibility is enabled but no ``target`` option was specified, it is recommended to set the option to avoid potential schema conflicts (if you're data is ECS compliant or non-conflicting feel free to ignore this message) // cc @yaauie @karenzone

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on standardize, maybe we can put it as constant in the ECS mixin?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kares In another conversation for a different plugin, I believe we discussed emitting this as info, since it isn't exactly something that needs to be resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you recall maybe the plugin?So that I can use the same log info, to be consistent across various plugins

Copy link
Contributor

@kares kares May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

believe we agreed upon this being logger at the info level

yes we agreed on info level logging previously, we do not have another plugin with the "warning" yet.
that's why I proposed a message for review, if accepted we would use it here and maybe move to ECS support mixing for re-use (as @andsel suggested)? (it's going to concern 3 other plugins I can think of right away)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logger.warn("When ECS compatibility is enabled also target option must be valued")
logger.info("ECS compatibility is enabled but no ``target`` option was specified, it is recommended to set the option to avoid potential schema conflicts (if you're data is ECS compliant or non-conflicting feel free to ignore this message)")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your message is perfect, in case of jdbc static and filter we would also add the id of the interested lookups, so in that case it could hit many lines like this, one for each lookup definition

Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
@andsel andsel requested a review from kares May 25, 2021 08:21
Copy link
Contributor

@kares kares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a slight typo that I had in the suggestion

lib/logstash/inputs/jdbc.rb Outdated Show resolved Hide resolved
spec/inputs/jdbc_spec.rb Outdated Show resolved Hide resolved
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
andsel and others added 2 commits May 25, 2021 13:43
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
@andsel andsel force-pushed the feature/enable_ecs_for_jdbc_input branch from eb17aef to 7d58fbe Compare May 25, 2021 12:01
@andsel andsel merged commit 86cac0a into logstash-plugins:master May 25, 2021
This was referenced May 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants