-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermittent CI/local issues with promtail #271
Comments
We may want to investigate if |
From the events from a failing job (https://github.com/defenseunicorns/uds-core/actions/runs/8333847348/job/22814109664?pr=156):
|
Just for posterity, dumping
|
From that dump the |
Tentatively closing this one based on current CI results - have not seen this since the k3s update - #426. |
Describe what should be investigated or refactored
We have seen occasional issues in CI/local dev with promtail where it takes a while to become healthy (or worst case doesn't). In at least a few of these cases we have seen the issue
too many open files
which typically is caused by cert ulimit/sysctl params not being set properly.Links to any relevant code
It may be worth including these settings in CI by default, along with documenting these as common needs for environments. Unsure how this would look for local dev - potentially we can enable this with a job in uds-k3d to set these on the node?
Additional context
This was a common issue in Big Bang with promtail/fluentbit as well as some of the pieces living on top of "core".
The text was updated successfully, but these errors were encountered: