Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workaround for jobs failing with missing S3 credentials #525

Merged
merged 2 commits into from
Oct 22, 2019
Merged

Workaround for jobs failing with missing S3 credentials #525

merged 2 commits into from
Oct 22, 2019

Conversation

gcampax
Copy link
Contributor

@gcampax gcampax commented Oct 22, 2019

Apparently it's a known problem: jtblin/kube2iam#136

@gcampax
training: add artificial delay of one minute before touching S3
1252155 
This ensures that kube2iam is properly started even on freshly
created nodes, and we get the correct credentials.
@gcampax gcampax added bug training Automatic training, dataset generation and update labels Oct 22, 2019
@gcampax gcampax requested a review from jgd5 October 22, 2019 04:45
@gcampax
tests: make use of the new sleep function
9fc1911 
I added it in the util folder so I can use it everywhere, because
I have written it way too many times
@coveralls
Copy link

Pull Request Test Coverage Report for Build 1728

  • 5 of 6 (83.33%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.005%) to 66.828%
Changes Missing Coverage Covered Lines Changed/Added Lines %
training/tasks/train.js 2 3 66.67%
Totals Coverage Status
Change from base Build 1718: 0.005%
Covered Lines: 6758
Relevant Lines: 9514
💛 - Coveralls

jgd5
jgd5 approved these changes Oct 22, 2019
View reviewed changes
Copy link
Contributor

@jgd5 jgd5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. This should work. Another way is to poll http://169.254.169.254/latest/meta-data/iam/security-credentials until we get the s3 role we expect (almond-training-s3-admin).

@gcampax gcampax merged commit 2de81af into master Oct 22, 2019
@gcampax gcampax deleted the wip/workaround-kube2iam branch October 24, 2019 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgd5 jgd5 jgd5 approved these changes

Assignees
No one assigned
Labels
bug training Automatic training, dataset generation and update
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gcampax @coveralls @jgd5