Bug 1925061: disable using MADV_FREE to release unused memory

[paulfantom]

As in title. This is for testing a possible solution to OOMKilling prometheus during upgrades.

/cc @s-urbaniak @sdodson @bparees

[openshift-ci-robot]

@paulfantom: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

*: disable using MADV_FREE to release unused memory

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bparees]

pre-emptively group-lead-approving so you can merge it if it helps.

[openshift-ci-robot]

@paulfantom: This pull request references Bugzilla bug 1925061, which is invalid:

• expected the bug to target the "4.7.0" release, but it targets "---" instead
• expected Bugzilla bug 1925061 to depend on a bug targeting a release in 4.8.0 and in one of the following states: MODIFIED, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1925061: disable using MADV_FREE to release unused memory

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[paulfantom]

/bugzilla refresh

[openshift-ci-robot]

@paulfantom: This pull request references Bugzilla bug 1925061, which is invalid:

• expected the bug to target the "4.7.0" release, but it targets "---" instead
• expected Bugzilla bug 1925061 to depend on a bug targeting a release in 4.8.0 and in one of the following states: MODIFIED, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rphillips]

/lgtm

[openshift-ci-robot]

@rphillips: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: paulfantom, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [paulfantom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rphillips]

/hold until ready

[smarterclayton]

I don't see how MADV_FREE fixes the issue. This is real memory that prometheus has to track. I still see a doubling of prometheus memory use over the entire run

[rphillips]

MADV_FREE may decrease the memory usage on compaction... it is worth the test to see if it helps.

[smarterclayton]

Upgrade is in the middle, I still see 2x growth in upgrade.

[simonpasquier]

IIUC MADV_FREE should already be set by the base image in 4.7 thanks to openshift/images#61?

[rphillips]

@simonpasquier I just checked my 4.7 cluster and the madvdontneed option is not enabled.

It looks like enabling the option might be recommended from upstream now: prometheus#8357 (comment)

[rphillips]

I tweaked the registry.ci.openshift.org/ocp/4.7:base image which is not being used here.

We should get this PR into 4.7 and 4.8.

[simonpasquier]

I tweaked the registry.ci.openshift.org/ocp/4.7:base image which is not being used here.

Right i was confused about which base image is used for Prometheus. It shouldn't hurt to use MADV_DONTNEED but I'm afraid it won't help much for bug 1925061 since we're not seeing memory increase due to compaction (or any other transient task which should release memory as fast as possible once it's completed).

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[paulfantom]

/close

[openshift-ci]

@paulfantom: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[paulfantom]

/reopen

[openshift-ci]

@paulfantom: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@paulfantom: This pull request references Bugzilla bug 1925061, which is invalid:

• expected the bug to target the "4.7.z" release, but it targets "4.8.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1925061: disable using MADV_FREE to release unused memory

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: paulfantom, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [paulfantom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@paulfantom: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-upgrade	132567c	link	/test e2e-aws-upgrade
ci/prow/e2e-aws	132567c	link	/test e2e-aws

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.