Bumps nomad timeout, adds a few missing dependencies, speeds up Nomad… #617
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… job killing in deploy process.
ghost
assigned 
|
Is this PR going to including killing the telemetry stuff? Ideally just a comment out since it shouuuuuld hopefully be improved in a future Nomad, but we want to disable it for prod for now. |
|
I have added commenting out the graphite code for Nomad lead server along with reverting the EBS-volume discovery changes I made. |
|
Related: hashicorp/nomad#4422 |
|
You have removed the telemetry statsd servers, but not the telemetry configuration from Nomad itself |
…finebio into kurtwheeler/minor-fixes
Miserlou
approved these changes
Sep 14, 2018
kurtwheeler
added a commit
that referenced
this pull request
Jan 10, 2019
#617) * Bumps nomad timeout, adds a few missing dependencies, speeds up Nomad job killing in deploy process. * Comment out graphite from lead nomad server, revert EBS volume discovery change. * Hopefully speed up Nomad job registration time. * Prevent old or local distributions of common from getting installed while deploying. * Remove telemetry terraform blocks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… job killing in deploy process.
Issue Number
N/A came up during crunch.
Purpose/Implementation Notes
Most of these are pretty minor/obvious things. The one significant change is in
deploy.sh. This makes it so we kill all the Nomad jobs in the background and then wait for them so we can do it a lot faster.Here is an image of the CPU utilization of the lead nomad server:
The long period at the start at ~18% was us killing them one by one. After the change to
deploy.shyou can see it peak up to the mid fifties as it kills all the jobs very quickly. I think the last high point which then trails off is probably us re-registering the base jobs.Types of changes
Functional tests
I have tested all of this by running it against production.
Checklist