Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Adding filebeat config to file dataviz #58152

Conversation

jgowdyelastic
Copy link
Member

@jgowdyelastic jgowdyelastic commented Feb 20, 2020

Adds the option to generate a Filebeat configuration for uploading additional data to the recently created index.

Fixes #56737

image

image

Also fixes an issue where lines which do not start with the expected line start regex are incorrectly added to the list of messages to import.
Affecting blank lines in csv or partial lines at the start of a rolled over log.

Checklist

@jgowdyelastic jgowdyelastic force-pushed the adding-filebeat-config-to-file-dataviz branch from 5cc40bc to acaf553 Compare February 24, 2020 08:33
@jgowdyelastic jgowdyelastic marked this pull request as ready for review February 24, 2020 13:09
@jgowdyelastic jgowdyelastic requested a review from a team as a code owner February 24, 2020 13:09
Copy link
Contributor

@walterra walterra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

x-pack/legacy/plugins/ml/public/application/app.tsx Outdated Show resolved Hide resolved
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@jgowdyelastic jgowdyelastic force-pushed the adding-filebeat-config-to-file-dataviz branch from ddc423f to 8ab3b4d Compare February 24, 2020 17:21
Copy link
Contributor

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a few more things that need changing to get it to work with the latest Filebeat. If you could update the PR branch then I can retest tomorrow to make sure I didn't forget any.

Copy link
Contributor

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I tested this with the latest changes and the config works now if you replace the placeholders.

Procedure:

  1. Download the latest Filebeat from https://www.elastic.co/downloads/beats/filebeat
  2. Copy the config proposed by the file uploader to the clipboard using the button provided
  3. Paste it into filebeat.yml
  4. Edit filebeat.yml to replace the placeholders with the actual file name/pattern, ES URL, ES username and password
  5. ./filebeat -c filebeat.yml -e

Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question, otherwise LGTM.

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jgowdyelastic jgowdyelastic merged commit b756cc3 into elastic:master Feb 25, 2020
@jgowdyelastic jgowdyelastic deleted the adding-filebeat-config-to-file-dataviz branch February 25, 2020 13:39
jgowdyelastic added a commit to jgowdyelastic/kibana that referenced this pull request Feb 25, 2020
* [ML] Adding filebeat config to file dataviz

* adding extra help text

* removing commented out code

* adding extra blank line to processors section

* cleaning up types

* moving hosts line out of function

* typo in config text

* updating config based on review

* tiny refactor

* translating paths text
jloleysens added a commit to jloleysens/kibana that referenced this pull request Feb 25, 2020
…re/files-and-filetree

* 'master' of github.com:elastic/kibana: (174 commits)
  [SIEM] Fix unnecessary re-renders on the Overview page (elastic#56587)
  Don't mutate error message (elastic#58452)
  Fix service map popover transaction duration (elastic#58422)
  [ML] Adding filebeat config to file dataviz (elastic#58152)
  [Uptime] Improve refresh handling when generating test data (elastic#58285)
  [Logs / Metrics UI] Remove path prefix from ViewSourceConfigur… (elastic#58238)
  [ML] Functional tests - adjust classification model memory (elastic#58445)
  [ML] Use event.timezone instead of beat.timezone in file upload (elastic#58447)
  [Logs UI] Unskip and stabilitize log column configuration tests (elastic#58392)
  [Telemetry] Separate the license retrieval from the stats in the usage collectors (elastic#57332)
  hide welcome screen for cloud (elastic#58371)
  Move src/legacy/ui/public/notify/app_redirect to kibana_legacy (elastic#58127)
  [ML] Functional tests - stabilize typing during df analytics creation (elastic#58227)
  fix short url in spaces (elastic#58313)
  [SIEM] Upgrades cypress to version 4.0.2 (elastic#58400)
  [Index management] Move to new platform "plugins" folder (elastic#58109)
  [kbn/optimizer] disable parallelization in terser plugin (elastic#58396)
  [Uptime] Delete useless try...catch blocks (elastic#58263)
  [Uptime] Use scripted metric for snapshot calculation (elastic#58247) (elastic#58389)
  [APM] Stabilize agent configuration API (elastic#57767)
  ...

# Conflicts:
#	src/plugins/console/public/application/containers/editor/legacy/console_editor/editor.tsx
jgowdyelastic added a commit that referenced this pull request Feb 25, 2020
* [ML] Adding filebeat config to file dataviz

* adding extra help text

* removing commented out code

* adding extra blank line to processors section

* cleaning up types

* moving hosts line out of function

* typo in config text

* updating config based on review

* tiny refactor

* translating paths text
elasticmachine pushed a commit to dhurley14/kibana that referenced this pull request Feb 25, 2020
…58471)

* [ML] Adding filebeat config to file dataviz

* adding extra help text

* removing commented out code

* adding extra blank line to processors section

* cleaning up types

* moving hosts line out of function

* typo in config text

* updating config based on review

* tiny refactor

* translating paths text
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ML] After file upload completes, display a Filebeat config that could add more data
6 participants