S3 Search Improvements #1391
Merged
S3 Search Improvements #1391
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ssalinas
force-pushed
the
s3_rework
branch
2 times, most recently
from
ef89fc8
to
9c6ad9a
Compare
ssalinas
changed the title
|
FYI, now building this on top of the commits from #1375 . It modifies/moves a bunch of the same files and the merges are a bit horrid otherwise |
ssalinas
changed the title
jets3t -> aws sdk
need @consumes on POST endpoints more s3 pagination tweaks more attempts at better pagination need >= here fix maxPerPage more robust search options add missing file fix for findbugs continuation token format needs group too use isTruncated for ending re-request to respect page size revert re-request for page size missing tokens gives false positive end of content typo
Implement pagination for S3 endpoint using continuation tokens
|
Added the docs for this now, going to merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening this early for any feedback.
Currently, all data about which files to upload to s3, and any possible bucket or key pattern overrides is stored in the executor configuration. This unfortunately means that SingularityService has no idea if it needs to search a separate bucket/pattern to find logs. It can only search what it knows about from it's defaults and the request group overrides.
Notable changes so far:
loggingS3Bucketremoved fromExecutorDataon the deploy. We have overrides by group and by file name for this purpose. If we want a request level override separate from the group one we can add that laterSingularityTaskExecutorDataobject not theExecutorDatafrom the deploy. This way we can pass additional information to the custom executor from system-wide configuration.S3LogResourcewill check the now available s3 additional files to determine if additional buckets need to be searchedStill TODO:
Note - SingularityService needs to be deployed before the executor for this particular change
/cc @tpetr