Attempt to fix AWS jobstore test #2874
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
They try to run on Python 3 now because I dropped the EC2 SSH key dependency for attempting them
adamnovak
force-pushed
the
issues/2873-fix-aws-jobstore-test
branch
from
a2bd903
to
e5ddd64
Compare
…fuse to progress with it off.
|
OK, this gets the tests passing again. I think the problem really was being able to do an upload before versioning got enabled properly, and probably something on Amazon's end (maybe just making things faster?) made it so we started having that happen with appreciable frequency. If we don't see the same problem again, we can eventually remove some of the logging and polling I've added here, and keep just the initial wait for versioning to take effect. |
DailyDreaming
pushed a commit
that referenced
this pull request
Feb 12, 2020
* Add more logging to try and diagnose cases where version and content disappear * Allow AWS job store tests to run when you don't have EC2 * Add comment * Try and make AWS job store tests run through on Python 3 They try to run on Python 3 now because I dropped the EC2 SSH key dependency for attempting them * use actually extant isinstance * Decode bytes when we need them as strings * Fix some Python 3 incompatibilities and watch our pipe to make sure the reader runs * Make sure uploads actually happen * Run flaky test first * Add logging and retries to try and get object versions after upload * Get the key the right way * Don't use removed message exception attribute * Poll the versioning setting, give it some time to take effect, and refuse to progress with it off. * Don't len a BytesIO * Do integration tests last again
DailyDreaming
pushed a commit
that referenced
this pull request
Feb 13, 2020
* Add more logging to try and diagnose cases where version and content disappear * Allow AWS job store tests to run when you don't have EC2 * Add comment * Try and make AWS job store tests run through on Python 3 They try to run on Python 3 now because I dropped the EC2 SSH key dependency for attempting them * use actually extant isinstance * Decode bytes when we need them as strings * Fix some Python 3 incompatibilities and watch our pipe to make sure the reader runs * Make sure uploads actually happen * Run flaky test first * Add logging and retries to try and get object versions after upload * Get the key the right way * Don't use removed message exception attribute * Poll the versioning setting, give it some time to take effect, and refuse to progress with it off. * Don't len a BytesIO * Do integration tests last again
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is my attempt to fix #2873. Right now I can't replicate the failure locally, so I've added some more specific errors/logging to see if I can get more information on failures on Gitlab.