Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug image and cache dump script #1541

Merged
merged 1 commit into from
Jan 3, 2024
Merged

Debug image and cache dump script #1541

merged 1 commit into from
Jan 3, 2024

Conversation

alculquicondor
Copy link
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it:

  • Configuration to build a debug image in the pipeline.
  • A script to produce a dump of the cache into the kueue Pod logs

Which issue(s) this PR fixes:

Fix #1437

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The image gcr.io/k8s-staging-kueue/debug:main, along with the script ./hack/dump_cache.sh can be used to trigger a dump of the internal cache into the logs.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 3, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify Netlify
Copy link

netlify bot commented Jan 3, 2024 •
edited

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit 94c5fbe
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6595a0526a87700007c593fd

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 3, 2024
@alculquicondor
Copy link
Contributor Author

/assign @tenzen-y

tenzen-y
tenzen-y reviewed Jan 3, 2024
View reviewed changes
Copy link
Member

@tenzen-y tenzen-y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally lgtm.

hack/debugpod/Dockerfile Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this image? Can we use the debian:stable image?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alculquicondor ping :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't set the user, the ephemeral pod creation fails because of the tight Pod security context (can't run as root).

Ideally, kubectl debug should allow to set a user id, but it doesn't.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we just need a slim image to be able to send kill

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, kubectl debug should allow to set a user id, but it doesn't.

Oh, I didn't know that. Thank you for clarifying.

hack/dump_cache.sh Outdated Show resolved Hide resolved
hack/dump_cache.sh Show resolved Hide resolved
hack/dump_cache.sh Outdated
NAMESPACE=${NAMESPACE:-kueue-system}
DEBUG_IMAGE=${DEBUG_IMAGE:-gcr.io/k8s-staging-kueue/debug:main}

leader=$(kubectl get lease -n ${NAMESPACE} -o jsonpath='{.items[0].spec.holderIdentity}' | cut -d '_' -f 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we obtain the leader name from c1f6bfd2.kueue.x-k8s.io? If we deploy the manager for custom jobs separate from the kueue-manager, this script probably doesn't work well.

kueue/config/components/manager/controller_manager_config.yaml

Line 12 in 7bf4e92

resourceName: c1f6bfd2.kueue.x-k8s.io

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@alculquicondor
Debug image and cache dump script
94c5fbe 
Change-Id: I7e0bdb0acbeb04a94cea8a574abd340aed81de0b
@alculquicondor
Copy link
Contributor Author

/retest

Opened flaky report #1543

tenzen-y
tenzen-y reviewed Jan 3, 2024
View reviewed changes
Copy link
Member

@tenzen-y tenzen-y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 3, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 0fe8f925a8f5b7d67b604d5309135a63d47e9b6b

@k8s-ci-robot k8s-ci-robot merged commit d5c83e3 into kubernetes-sigs:main Jan 3, 2024
14 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v0.6 milestone Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tenzen-y tenzen-y tenzen-y left review comments

@denkensk denkensk Awaiting requested review from denkensk

@kerthcet kerthcet Awaiting requested review from kerthcet

Assignees

@tenzen-y tenzen-y

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v0.6
Development

Successfully merging this pull request may close these issues.

Implement cache dump via signals
3 participants
@alculquicondor @k8s-ci-robot @tenzen-y