Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SP reset timeout may be too short and should be runtime configurable #284

Closed
jgallagher opened this issue Sep 12, 2024 · 0 comments · Fixed by #285
Closed

SP reset timeout may be too short and should be runtime configurable #284

jgallagher opened this issue Sep 12, 2024 · 0 comments · Fixed by #285
Assignees
Milestone

Comments

@jgallagher
Copy link
Collaborator

We currently override whatever retry attempt is configured specifically for SP resets:

// See comment on `SP_RESET_TIME_ALLOWED` above; bump up the retry count
// if we're trying to trigger an SP reset.
let calc_reset_attempts = || {
let time_desired = SP_RESET_TIME_ALLOWED.as_millis();
let per_attempt = self.per_attempt_timeout.as_millis().max(1);
((time_desired + per_attempt - 1) / per_attempt) as usize
};

to allow up to 30 seconds, particularly for sidecar SPs which have historically taken "a while".

We saw a sidecar SP reset timeout today during a mupdate, and a subsequent retry succeeded but took 25 seconds. A likely explanation for the timeout is that there's some variance and 30 seconds was just too short. We should bump this 30 seconds up, and ideally also allow reset timeouts to be configured at runtime like the retry timeouts are for every other command.

@jgallagher jgallagher added this to the 11 milestone Sep 12, 2024
@jgallagher jgallagher self-assigned this Sep 12, 2024
jgallagher added a commit to oxidecomputer/omicron that referenced this issue Sep 13, 2024
All the changes are in `sp-sim` (mostly related to API changes to
support tech port unlocking).

Opening this on its own to reduce diff churn for upcoming changes to
deal with
oxidecomputer/management-gateway-service#284.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant