New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Memory Controller #7233

Merged

kunga merged 14 commits into ydb-platform:main from kunga:memory-controller-chapter-two

Jul 31, 2024

Member

kunga commented Jul 30, 2024 •

edited

Loading

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Additional information

Use NMemory::IProcessMemoryInfoProvider where is possible
Calculate AllocatorCachesMemory metric precise
Obtain node MemLimit and MemAvailable from
Pass detailed memory stats to NodeWhiteboard for further rendering in UI
Calculate external (non YDB) consumption in case without CGroupMemLimit
Pass HardLimit from MemoryController to other components
Remove excess Bytes suffix from metric names

In addition to #6587

Connected with ydb-platform/ydb-embedded-ui#1091

kunga added 13 commits

July 30, 2024 07:05


          use process memory provider in monitor

ed4915f


          pass allocator caches memory

a44458a


          fix min

91ecc5e


          extract mem info

df0ffc1


          treat memtotal as hard limit

4cc1fa4


          measure external consumption

3fc55a8


          fix memTotalHardLimit

0eaf4c5


          new node whiteboard mem stats

dc7eb8a


          fix ext_counters.cpp

5baa002


          aggregate mem stats by tenant

caa9c0f


          fix proto

94a6486


          clean up

2981d53


          clean up

aa549be

kunga marked this pull request as draft

July 30, 2024 07:08

github-actions bot added the not-for-changelog label

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 07:09:35 UTC Pre-commit check for e0ca5c4 has started.
⚪ 2024-07-30 07:18:49 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 10:05:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75033	61458	0	6	13555	14

⚪ 2024-07-30 10:14:00 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 10:16:27 UTC ydbd size 8.1 GiB changed* by +1.8 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `c0f561e`	merge: `e0ca5c4`	diff	diff %
ydbd size	8 677 238 176 Bytes	8 679 117 304 Bytes	+1.8 MiB	+0.022%
ydbd stripped size	471 843 808 Bytes	471 892 256 Bytes	+47.3 KiB	+0.010%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-07-30 10:16:28 UTC Check cancelled

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 07:09:58 UTC Pre-commit check for e0ca5c4 has started.
⚪ 2024-07-30 07:18:48 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-30 07:59:04 UTC Build successful.

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 07:10:00 UTC Pre-commit check for e0ca5c4 has started.
⚪ 2024-07-30 07:19:12 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-30 10:11:00 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13560	12958	0	90	342	170

🟢 2024-07-30 10:12:20 UTC Build successful.
🟡 2024-07-30 10:13:25 UTC ydbd size 5.2 GiB changed* by +290.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `9e244eb`	merge: `e0ca5c4`	diff	diff %
ydbd size	5 571 073 688 Bytes	5 571 370 888 Bytes	+290.2 KiB	+0.005%
ydbd stripped size	1 199 201 776 Bytes	1 199 262 000 Bytes	+58.8 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot added not-for-changelog and removed not-for-changelog labels

kunga self-assigned this


          remove proc stat assert

9c942a9

kunga marked this pull request as ready for review

July 30, 2024 10:16

kunga requested a review from alexd65536

July 30, 2024 10:17

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 10:26:43 UTC Pre-commit check for 96b91ac has started.
⚪ 2024-07-30 10:29:47 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-30 12:47:17 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13542	12930	0	103	337	172

🟢 2024-07-30 12:48:39 UTC Build successful.
🟡 2024-07-30 12:49:33 UTC ydbd size 5.2 GiB changed* by +290.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0138b90`	merge: `96b91ac`	diff	diff %
ydbd size	5 570 781 208 Bytes	5 571 078 416 Bytes	+290.2 KiB	+0.005%
ydbd stripped size	1 199 150 352 Bytes	1 199 210 576 Bytes	+58.8 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 10:26:56 UTC Pre-commit check for 96b91ac has started.
⚪ 2024-07-30 10:30:04 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-30 10:35:32 UTC Build successful.

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 10:31:34 UTC Pre-commit check for 96b91ac has started.
⚪ 2024-07-30 10:34:37 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 12:42:42 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75034	61464	0	2	13554	14

⚪ 2024-07-30 12:50:42 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 12:59:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18 (only retried tests)	6	0	3	0	9

⚪ 2024-07-30 13:06:23 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-07-30 13:14:15 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13 (only retried tests)	2	0	3	0	8

🟡 2024-07-30 13:20:46 UTC ydbd size 8.1 GiB changed* by +490.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0138b90`	merge: `96b91ac`	diff	diff %
ydbd size	8 678 040 144 Bytes	8 678 542 440 Bytes	+490.5 KiB	+0.006%
ydbd stripped size	471 852 768 Bytes	471 879 648 Bytes	+26.2 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

alexd65536 approved these changes

View reviewed changes

kunga added the rebase-and-check label

github-actions bot removed the rebase-and-check label

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 15:25:03 UTC Pre-commit check for ef9e3d2 has started.
⚪ 2024-07-30 15:28:02 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-30 15:38:45 UTC Build successful.

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 15:26:40 UTC Pre-commit check for ef9e3d2 has started.
⚪ 2024-07-30 15:30:00 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-30 18:03:57 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13562	12959	0	92	338	173

🟢 2024-07-30 18:05:17 UTC Build successful.
🟡 2024-07-30 18:05:53 UTC ydbd size 5.2 GiB changed* by +286.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ee419db`	merge: `ef9e3d2`	diff	diff %
ydbd size	5 571 118 576 Bytes	5 571 411 680 Bytes	+286.2 KiB	+0.005%
ydbd stripped size	1 199 220 976 Bytes	1 199 277 104 Bytes	+54.8 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot commented Jul 30, 2024 •

edited

Loading

⚪ 2024-07-30 15:26:40 UTC Pre-commit check for ef9e3d2 has started.
⚪ 2024-07-30 15:30:00 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 17:51:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75053	61466	0	3	13567	17

⚪ 2024-07-30 17:59:42 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 18:10:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
23 (only retried tests)	9	0	2	4	8

⚪ 2024-07-30 18:17:36 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-07-30 18:25:53 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14 (only retried tests)	3	0	3	0	8

🟢 2024-07-30 18:33:16 UTC Build successful.
🟡 2024-07-30 18:33:55 UTC ydbd size 8.1 GiB changed* by +482.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ee419db`	merge: `ef9e3d2`	diff	diff %
ydbd size	8 678 462 728 Bytes	8 678 956 752 Bytes	+482.4 KiB	+0.006%
ydbd stripped size	471 881 056 Bytes	471 899 680 Bytes	+18.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

kunga added the rebase-and-check label

github-actions bot removed the rebase-and-check label

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 10:36:24 UTC Pre-commit check for 2d9b83f has started.
⚪ 2024-07-31 10:39:32 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-31 11:18:18 UTC Build successful.

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 10:38:55 UTC Pre-commit check for 2d9b83f has started.
⚪ 2024-07-31 10:42:10 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-31 13:56:17 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13541	13464	0	23	36	18

🟢 2024-07-31 13:57:58 UTC Build successful.
🔴 2024-07-31 13:58:29 UTC ydbd size 5.5 GiB changed* by +276.0 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash	main: `e464b73`	merge: `2d9b83f`	diff	diff %
ydbd size	5 571 429 760 Bytes	5 860 793 624 Bytes	+276.0 MiB	+5.194%
ydbd stripped size	1 199 304 144 Bytes	1 467 659 632 Bytes	+255.9 MiB	+22.376%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 10:38:56 UTC Pre-commit check for 2d9b83f has started.
⚪ 2024-07-31 10:43:40 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-31 13:46:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75054	61470	0	4	13567	13

⚪ 2024-07-31 13:55:00 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-31 14:06:04 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
21 (only retried tests)	6	0	3	4	8

⚪ 2024-07-31 14:13:26 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-07-31 14:21:50 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15 (only retried tests)	3	0	4	0	8

🟢 2024-07-31 14:29:17 UTC Build successful.
🟡 2024-07-31 14:29:55 UTC ydbd size 8.1 GiB changed* by +482.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `e464b73`	merge: `2d9b83f`	diff	diff %
ydbd size	8 678 737 472 Bytes	8 679 231 592 Bytes	+482.5 KiB	+0.006%
ydbd stripped size	471 916 032 Bytes	471 934 752 Bytes	+18.3 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

kunga added the rebase-and-check label

github-actions bot removed the rebase-and-check label

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 14:36:29 UTC Pre-commit check for 9a6507b has started.
⚪ 2024-07-31 14:44:56 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-31 14:57:20 UTC Build successful.

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 14:36:41 UTC Pre-commit check for 9a6507b has started.
⚪ 2024-07-31 14:42:29 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-31 16:55:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75091	61486	0	4	13588	13

⚪ 2024-07-31 17:03:50 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-31 17:12:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19 (only retried tests)	8	0	2	0	9

⚪ 2024-07-31 17:19:44 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-07-31 17:27:53 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12 (only retried tests)	3	0	1	0	8

🟢 2024-07-31 17:34:58 UTC Build successful.
🟡 2024-07-31 17:35:32 UTC ydbd size 8.1 GiB changed* by +478.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1dbe46b`	merge: `9a6507b`	diff	diff %
ydbd size	8 679 965 544 Bytes	8 680 455 576 Bytes	+478.5 KiB	+0.006%
ydbd stripped size	471 972 000 Bytes	471 986 656 Bytes	+14.3 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot commented Jul 31, 2024 •

edited

Loading

⚪ 2024-07-31 14:36:49 UTC Pre-commit check for 9a6507b has started.
⚪ 2024-07-31 14:45:23 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-31 16:46:48 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13524	13453	0	15	41	15

🟢 2024-07-31 16:48:15 UTC Build successful.
🟡 2024-07-31 16:48:49 UTC ydbd size 5.5 GiB changed* by +291.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1dbe46b`	merge: `9a6507b`	diff	diff %
ydbd size	5 861 242 928 Bytes	5 861 541 520 Bytes	+291.6 KiB	+0.005%
ydbd stripped size	1 467 761 232 Bytes	1 467 821 904 Bytes	+59.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

kunga merged commit 27b7968 into ydb-platform:main

17 of 27 checks passed

kunga mentioned this pull request

Remove MemoryStats from SystemStateInfo #7378

Merged

StekPerepolnen mentioned this pull request

tenantinfo handler added DatabaseStorage #7554

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

not-for-changelog