Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load the ES template on connect. #1381

Merged
merged 1 commit into from
Apr 19, 2016
Merged

Conversation

tsg
Copy link
Contributor

@tsg tsg commented Apr 13, 2016

It used to try loading it only once on init, causing bug #1321.
This change moves the call to loadTemplate at connection time, immediately
after successful connection. This has the effect that if overwrite is true,
the template will be loaded on each new established connection.

@tsg
Copy link
Contributor Author

tsg commented Apr 13, 2016

@urso would be good if you could check for side effects of this, or if you can think of a better way to implement it. I had to do two nested closures, so I'd like to simplify it if possible.

Another issue is that currently loadTemplate ignores errors (apart from logging them). We might want to make Connect() fail if the template loading failed, as running Beats without a template is always bad.

@tsg tsg added discuss Issue needs further discussion. review labels Apr 13, 2016
@ruflin ruflin added the libbeat label Apr 13, 2016
@@ -61,6 +62,7 @@ func NewClient(
esURL, index string, proxyURL *url.URL, tls *tls.Config,
username, password string,
params map[string]string,
onConnectCallback *func(client *Client),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we've got so many parameters in NewClient already, we might consider to pass a config struct to clean this up a little. Pretty annoying to adjust tests every single parameter we add (even if not used by tests)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onConnectCallback does not need to be a pointer to a function. Just use func (client *Client). One can check for == nil and != nil on func objects only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, didn't know that.

@urso
Copy link

urso commented Apr 14, 2016

I think error handling in loadTemplate is required. What if shield is available, and beats do not have rights to load template?

@tsg tsg force-pushed the load_template_on_connect branch from 400541c to bcb390c Compare April 15, 2016 20:44
@tsg
Copy link
Contributor Author

tsg commented Apr 18, 2016

Improvements since last review:

  • Error checking and mark the Connect as failed if the template cannot be PUT to Elasticsearch
  • Read the template file on startup and exit with an error if the file cannot be found (also happens when using -configtest).
  • Serialize loadTemplates requests with a Mutex
  • Added an integration test for template loading

@urso, can you have another look?

@urso
Copy link

urso commented Apr 18, 2016

LGTM, but travis seems to complain.

@tsg tsg force-pushed the load_template_on_connect branch from 270ac1d to 9439e4d Compare April 18, 2016 14:58
@tsg
Copy link
Contributor Author

tsg commented Apr 18, 2016

Rebased and added the Changelog item. The test failure in Jenkins doesn't seem related.

@tsg
Copy link
Contributor Author

tsg commented Apr 18, 2016

jenkins, retest it

It used to try loading it only once on init, causing bug elastic#1321.
This change moves the call to loadTemplate at connection time, immediately
after successful connection. This has the effect that if overwrite is true,
the template will be loaded on each new established connection.

The template is read on init time and sent to Elasticsearch at connect time.
This means that if the template path is wrong, it will be discovered at
startup (including `-configtest`).

In case there is an error loading the template, the Connect call fails.

This commit includes an integration test for the behaviour.
@tsg tsg force-pushed the load_template_on_connect branch from 9439e4d to bc53cdb Compare April 18, 2016 21:13
@ruflin ruflin merged commit d601de2 into elastic:master Apr 19, 2016
@tsg tsg deleted the load_template_on_connect branch August 25, 2016 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Issue needs further discussion. libbeat review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants