Even more remote heartbeat improvements #3594
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
billyb2
force-pushed
the
billy/set_remote_builder
branch
3 times, most recently
from
942c367
to
18dd107
Compare
Having the logic for creating a remote builder be in `web` doesn't make sense. The majority of the work can be done in `flyctl`, and as such, remove another point of failure in a different component
Since these operations are just queries (and don't mutate state), we can feel free to retry them as many times as needed
UNIT TEST WHOOOOO
If we get a non 500 error, then there's no point in having multiple seconds waiting, we should just fail.
For whatever reason, if we pass app.Name to destroy the app when we encounter an issue creating the builder, we pass nil (in unit tests). Not sure why, but all good Secondly, we should sleep between API requests
billyb2
force-pushed
the
billy/set_remote_builder
branch
from
a3506f5
to
a4ebb29
Compare
billyb2
force-pushed
the
billy/set_remote_builder
branch
from
c70b7fa
to
88c35b3
Compare
benbjohnson
reviewed
Jun 7, 2024
| return nil, nil, err | ||
| } | ||
|
|
||
| if validateBuilderErr != NoBuilderApp { |
There was a problem hiding this comment.
I believe this block won't occur because NoBuilderApp is an instance of ValidateBuilderError which is checked with the errors.As() above, correct?
There was a problem hiding this comment.
i don't believe so. errors.As just checks if it's a ValidateBuilderError or not, and we only return before this check if it isn't a validation error.
| flapsClient := flapsutil.ClientFromContext(ctx) | ||
|
|
||
| var volumes []fly.Volume | ||
| numRetries := 0 |
There was a problem hiding this comment.
We use a "backoff" library elsewhere for retries. Maybe that's better to use than rolling your own?
There was a problem hiding this comment.
the only issue with the backoff library is that i need to exit early if we don't get a server error, instead of retrying multiple times (for example, if we get a 404, it's likely a user error)
billyb2
force-pushed
the
billy/set_remote_builder
branch
from
9f704ed
to
98f45b5
Compare
benbjohnson
approved these changes
Jun 10, 2024
billyb2
added a commit
that referenced
this pull request
Jun 11, 2024
billyb2
added a commit
that referenced
this pull request
Jun 11, 2024
This reverts commit 4fb6bd4. Ben T made some changes that should let this work again. We'll need to add a preflight test to ensure that this doesn't break in the future, however.
billyb2
added a commit
that referenced
this pull request
Jun 12, 2024
* Restart remote builder machine if it's stopped * Fix tests * Reapply "Even more remote heartbeat improvements (#3594)" This reverts commit 4fb6bd4. Ben T made some changes that should let this work again. We'll need to add a preflight test to ensure that this doesn't break in the future, however. * Add the recreate builder flag This allows users forcing recreating the remote builder, even if it's valid * Add a test for deploying and recreating the remote builder * Catch and trace errors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Summary
What and Why:
Having the logic for creating a remote builder be in
webdoesn't make sense. The majority of the work can be done inflyctl, and as such, remove another point of failure in a different componentHow:
Move all of the logic from EnsureMachineRemoteBuilder from web to flyctl
Related to:
Documentation