Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] File data viz limiting uploaded doc chunk size #44768

Conversation

jgowdyelastic
Copy link
Member

@jgowdyelastic jgowdyelastic commented Sep 4, 2019

Summary

Adjusts the size of the chunks of docs uploaded to kibana when importing a file.
If the chunk of documents is over 1000000 chars long, it is split into smaller chunks who's average length will be around 1000000 chars.

Also halves the default size of the document chunk to 5000 docs.

Fixes #42114

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

- [ ] This was checked for cross-browser compatibility, including a check against IE11
- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
- [ ] Documentation was added for features that require explanation or tutorials
- [ ] Unit or functional tests were updated or added to match the most common scenarios
- [ ] This was checked for keyboard-only and screenreader accessibility

For maintainers

- [ ] This was checked for breaking API changes and was labeled appropriately
- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@jgowdyelastic jgowdyelastic self-assigned this Sep 4, 2019
@jgowdyelastic jgowdyelastic added :ml bug Fixes for quality problems that affect the customer experience Feature:File and Index Data Viz ML file and index data visualizer release_note:fix review labels Sep 4, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui

@jgowdyelastic jgowdyelastic marked this pull request as ready for review September 4, 2019 16:58
@jgowdyelastic jgowdyelastic requested a review from a team as a code owner September 4, 2019 16:58
@jgowdyelastic jgowdyelastic changed the title [ML] File data viz limitiing upload chunk size [ML] File data viz limiting upload chunk size Sep 4, 2019
@jgowdyelastic jgowdyelastic changed the title [ML] File data viz limiting upload chunk size [ML] File data viz limiting uploaded doc chunk size Sep 4, 2019
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@walterra walterra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a question about createDocumentChunks().

Copy link
Contributor

@alvarezmelissa87 alvarezmelissa87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ⚡️

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@walterra walterra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes LGTM.

@jgowdyelastic jgowdyelastic merged commit 7d22b74 into elastic:master Sep 5, 2019
@jgowdyelastic jgowdyelastic deleted the file-data-viz-limiting-upload-chunk-size branch September 5, 2019 15:55
jgowdyelastic added a commit to jgowdyelastic/kibana that referenced this pull request Sep 5, 2019
* [ML] File data viz limitiing upload chunk size

* adding comments

* refactor

* fixing incorrect overwrite of array
jgowdyelastic added a commit to jgowdyelastic/kibana that referenced this pull request Sep 5, 2019
* [ML] File data viz limitiing upload chunk size

* adding comments

* refactor

* fixing incorrect overwrite of array
jgowdyelastic added a commit that referenced this pull request Sep 5, 2019
* [ML] File data viz limitiing upload chunk size

* adding comments

* refactor

* fixing incorrect overwrite of array
jgowdyelastic added a commit that referenced this pull request Sep 5, 2019
* [ML] File data viz limitiing upload chunk size

* adding comments

* refactor

* fixing incorrect overwrite of array
jloleysens added a commit to jloleysens/kibana that referenced this pull request Sep 6, 2019
…ete-for-distance_feature

* 'master' of github.com:elastic/kibana: (89 commits)
  Replace TSVB timeseries charts with elastic-charts (elastic#33558)
  [TSVB][Top N aggregation] Unable to deal with negative values (elastic#43581)
  [alerting] Adds Action Type configuration support and whitelisting (elastic#44483)
  FTR: fix WebDriver Actions calls (elastic#44605)
  [Code] add NodeRepositoriesService to watch new repositories on local node (elastic#44677)
  [skip-ci][Maps] Improve Maps intro page (elastic#44721)
  [Maps] Update titles and descriptions for data sources (elastic#44833)
  Types + Extract Integration Util (elastic#44433)
  Downgrade log level from info to debug for cases when we cannot handle authentication attempt. (elastic#44933)
  [Reporting] Remove Chome stdout/stderr observables, Add Browser Logger observable (elastic#44359)
  Update Jest script to output coverage (elastic#44447)
  [ftr] support --kibana-install-dir flag (elastic#44552)
  [WATCHER] Allow user to set a threshold value of 0 (elastic#44810)
  Remove injectI18n in dashboard plugin. (elastic#44580)
  [Graph] Save modal (elastic#44261)
  Use external script for the OIDC Implicit flow handler page. (elastic#44866)
  disable router prefixing with pluginId (elastic#44855)
  [SIEM] Fix bug on url + inspect functionality on hosts/hostDetails page (elastic#44671)
  [ML] File data viz limiting uploaded doc chunk size (elastic#44768)
  [code] Append go env variable 'GOCACHE' to go lsp spawn command. (elastic#44864)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:File and Index Data Viz ML file and index data visualizer :ml release_note:fix review v7.4.0 v7.5.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ML] CSV import via Data visualizer is not working with the attached file
5 participants