Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding note about runner GC on task pod #8829

Merged
merged 5 commits into from
Jul 5, 2024
Merged

adding note about runner GC on task pod #8829

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
41b9404
adding note about runner GC on task pod
soulchips Jun 26, 2024
fc26092
Update container-runner.adoc
soulchips Jun 27, 2024
5a1d97d
Update jekyll/_cci2/container-runner.adoc
soulchips Jul 5, 2024
9b53332
Update jekyll/_cci2/container-runner.adoc
soulchips Jul 5, 2024
6308466
Merge branch 'master' into onprem-276/runner-garbage-collection-docs
rosieyohannan Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion jekyll/_cci2/container-runner.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,11 @@ NOTE: Cluster-wide permissions are used by container runner to autodetect the OS
[#garbage-collection]
== Garbage collection

Each container runner has a garbage collector which will ensure any pods and secrets with the label `app.kubernetes.io/managed-by=circleci-container-agent` left dangling in the cluster are removed. By default this will remove all jobs older than five hours and five minutes. This can be shortened or lengthened via the `agent.kubeGCThreshold` parameter. However, if you do shorten the garbage collection (GC) frequency, also shorten the max task run time via the `agent.maxRunTime` parameter to be a value smaller than the new GC frequency. Otherwise a running task pod could be removed by the GC.
Each container runner has a garbage collector. The garbage collector ensures the removal of any pods and secrets with the label `app.kubernetes.io/managed-by=circleci-container-agent` that are left dangling in the cluster. By default, the garbage collector removes all jobs older than five hours and five minutes. This time limit can be shortened or lengthened via the `agent.gc.threshold` parameter. However, if you do shorten the garbage collection frequency, you must also shorten the maximum task run time via the `agent.maxRunTime` parameter to be a value smaller than the new garbage collection frequency.

CAUTION: If you change the garbage collection threshold but do **not** keep the max task run time lower than the garbage collection frequency, a running task pod could be removed by the garbage collector.

The garbage collector may remove some objects sooner than the threshold. Task pods have a liveness probe that checks for a running task-agent process. Once a task completes or fails, the task-agent process will stop running and the liveness probe will fail, which will trigger GC.

Container runner will drain and restart cleanly when sent a termination signal. Container runner will not automatically attempt to launch a task that fails to start. This can be done in the CircleCI web app.

Expand Down