You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What steps did you take and what happened:
[A clear and concise description of what the bug is.]
tl;dr - if your sonobuoy aggregator and sonobuoy plugin pods are running on separate hosts, and the sonobuoy plugin's host dies, then the sonobuoy aggregator will get stuck with the following message and keep infinitely retrying until timeout:
time="2024-06-28T21:49:39Z" level=error msg="could not find pod created by plugin my-plugin-test, will retry: no pods were created by plugin my-plugin-test"
sonobuoy run with a test that takes >few minutes to finish
wait for the sonobuoy pod to create the plugin pod (e.g. sonobuoy-my-plugin-test)
force delete the node that sonobuoy-my-plugin-test is running on. it MUST be a different node than the sonobuoy pod.
check the logs of the sonobuoy pod.
What did you expect to happen:
it would be good if sonobuoy re-created the plugin pods. perhaps we could add a timeout for this check and try to re-create the pods if it times out.
alternatively, we can have the parent caller of sonobuoy run do the retry, but i'm wondering if there's a better way to do this in sonobuoy itself.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
Sonobuoy version: 0.56.15
Kubernetes version: (use kubectl version): 1.22
Kubernetes installer & version:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): ubuntu 20.04
Sonobuoy tarball (which contains * below)
The text was updated successfully, but these errors were encountered:
What steps did you take and what happened:
[A clear and concise description of what the bug is.]
tl;dr - if your sonobuoy aggregator and sonobuoy plugin pods are running on separate hosts, and the sonobuoy plugin's host dies, then the sonobuoy aggregator will get stuck with the following message and keep infinitely retrying until timeout:
sonobuoy run
with a test that takes >few minutes to finishsonobuoy
pod to create the plugin pod (e.g.sonobuoy-my-plugin-test
)sonobuoy-my-plugin-test
is running on. it MUST be a different node than thesonobuoy
pod.sonobuoy
pod.What did you expect to happen:
it would be good if sonobuoy re-created the plugin pods. perhaps we could add a timeout for this check and try to re-create the pods if it times out.
alternatively, we can have the parent caller of
sonobuoy run
do the retry, but i'm wondering if there's a better way to do this in sonobuoy itself.Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
): 1.22/etc/os-release
): ubuntu 20.04The text was updated successfully, but these errors were encountered: