[Maps] chunk geojson upload to keep import requests under 1MB #93678
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nreese
added
release_note:fix
[Deprecated-Use Team:Presentation]Team:Geo
|
Pinging @elastic/kibana-gis (Team:Geo) |
|
@elasticmachine merge upstream |
thomasneirynck
approved these changes
Mar 8, 2021
💚 Build SucceededMetrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: |
nreese
added a commit
to nreese/kibana
that referenced
this pull request
Mar 8, 2021
…c#93678) * [Maps] chunk geojson upload to keep import requests under 1MB * fix geojson_importer tests * update failure.item to reflect location in file * remove console statement * clean up * return instead of break if upload is no longer active * add unit test for createChunks * update file_upload API Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
nreese
added a commit
that referenced
this pull request
Mar 8, 2021
#93974) * [Maps] chunk geojson upload to keep import requests under 1MB * fix geojson_importer tests * update failure.item to reflect location in file * remove console statement * clean up * return instead of break if upload is no longer active * add unit test for createChunks * update file_upload API Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Should fix #72985
#92620 updated GeoJson importer to stream files in 10MB blocks for importing. Upon trying this in cloud, 10MB blocks were still too large for cloud and result in timeouts. Upon deeper inspection of ML's CSV upload, which can upload large files on cloud, it became apparent that blocks need to be broken into chunks of less then 1MB chunks for importing. There is an edge case where an individual feature is larger than 1MB. In this case, the individual feature can not be broken down any smaller and must be imported in a single call.
Unfortunately, there is no way to test this on cloud without merging and waiting for the next snapshot to be built.