Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add logging in case we have minimum memory recommendations #299

Merged
merged 1 commit into from
Mar 11, 2024
Merged

Add logging in case we have minimum memory recommendations #299

merged 1 commit into from
Mar 11, 2024

Conversation

voelzmo
Copy link
Member

@voelzmo voelzmo commented Mar 8, 2024 •
edited
Loading

What this PR does / why we need it:
Add logging in case we have minimum memory recommendations. This happens from time to time and we need more information to understand why. This gives us all the internal recommender state, such that we can derive from there, how to fix this.

This is what the debug output currently looks like

W0311 09:54:11.860225       1 estimator.go:157] Computed resources for memory were below minimum! Computed 5886, minimum is 262144000.
W0311 09:54:11.860234       1 estimator.go:180] Here's the checkpoint/state: {
  "lastUpdateTime": "2024-03-11T09:54:11Z",
  "version": "v3",
  "cpuHistogram": {
    "referenceTimestamp": "2024-03-11T00:00:00Z",
    "bucketWeights": {
      "26": 10000
    },
    "totalWeight": 0.26620445628136635
  },
  "memoryHistogram": {
    "referenceTimestamp": "2024-03-12T00:00:00Z",
    "bucketWeights": {
      "0": 10000
    },
    "totalWeight": 2.6620445628136635
  },
  "firstSampleStart": "2024-03-11T09:54:02Z",
  "lastSampleStart": "2024-03-11T09:54:04Z",
  "totalSamplesCount": 2
}
W0311 09:54:11.860235       1 estimator.go:157] Computed resources for cpu were below minimum! Computed 0, minimum is 25.
W0311 09:54:11.860241       1 estimator.go:180] Here's the checkpoint/state: {
  "lastUpdateTime": "2024-03-11T09:54:11Z",
  "version": "v3",
  "cpuHistogram": {
    "referenceTimestamp": "2024-03-11T00:00:00Z",
    "bucketWeights": {
      "26": 10000
    },
    "totalWeight": 0.26620445628136635
  },
  "memoryHistogram": {
    "referenceTimestamp": "2024-03-12T00:00:00Z",
    "bucketWeights": {
      "0": 10000
    },
    "totalWeight": 2.6620445628136635
  },
  "firstSampleStart": "2024-03-11T09:54:02Z",
  "lastSampleStart": "2024-03-11T09:54:04Z",
  "totalSamplesCount": 2
}

Release note:

Add logging in case we have minimum memory recommendations

@gardener-robot gardener-robot added needs/review Needs review size/xs Size of pull request is tiny (see gardener-robot robot/bots/size.py) labels Mar 8, 2024
@voelzmo
Copy link
Member Author

voelzmo commented Mar 8, 2024

/invite @andrerun

@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Mar 8, 2024
@gardener-robot
Copy link

@voelzmo Command "/invite @andrerun" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/autoscaler repository.".

Additional Information 
Redacted in public. Check backend logs.

@voelzmo voelzmo force-pushed the enh/add-logging-for-minimum-recommendations branch from d141eb6 to b82c8d2 Compare March 8, 2024 12:42
andrerun
andrerun approved these changes Mar 11, 2024
View reviewed changes
Copy link

@andrerun andrerun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Check if the one question I posted requires action, then ship it.

vertical-pod-autoscaler/pkg/recommender/logic/estimator.go
if err != nil {
klog.Errorf("Error during marshalling checkpoint: %s", err)
}
klog.Warningf("Here's the checkpoint/state: %s", prettyCheckpoint)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need a line break before the checkpoint content?

@voelzmo voelzmo merged commit f838d65 into gardener:rel-vertical-pod-autoscaler Mar 11, 2024
5 checks passed
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Mar 11, 2024
@voelzmo voelzmo mentioned this pull request Apr 19, 2024
Merged
@ialidzhikov ialidzhikov mentioned this pull request Aug 14, 2024
Merged
@plkokanov plkokanov mentioned this pull request Aug 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrerun andrerun andrerun approved these changes

@himanshu-kun himanshu-kun Awaiting requested review from himanshu-kun

@unmarshall unmarshall Awaiting requested review from unmarshall unmarshall is a code owner

@rishabh-11 rishabh-11 Awaiting requested review from rishabh-11 rishabh-11 is a code owner

Assignees
No one assigned
Labels
needs/review Needs review reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) size/xs Size of pull request is tiny (see gardener-robot robot/bots/size.py) status/closed Issue is closed (either delivered or triaged)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@voelzmo @gardener-robot @andrerun @gardener-robot-ci-1