Memory Controller #6587

kunga · 2024-07-11T17:42:10Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Additional information

https://wiki.yandex-team.ru/kikimr/design/localdb/avtoopredelenie-razmera-shared-cache/#memory-controller

Example 1

Many writes to 3000 DataShards lead to force MemTable compactions

https://nda.ya.ru/t/t1oQO5o8779CHW

ydb kv init --min-partitions 3000 --auto-partition 0 --cols 10 --int-cols 10 --key-cols 3 --init-upserts 0
ydb kv run upsert --cols 10 --int-cols 10 --key-cols 3 -t 50 --rows 2 -s 100000

Example 2

Many writes to 10 DataShards lead to intensive SharedCache usage

https://nda.ya.ru/t/UWnr4ITy779Ex9

ydb kv init --min-partitions 10 --auto-partition 0 --cols 10 --int-cols 10 --key-cols 3 --init-upserts 0
ydb kv run upsert --cols 10 --int-cols 10 --key-cols 3 -t 50 --rows 10 -s 100000

github-actions · 2024-07-11T17:45:26Z

⚪ 2024-07-11 17:45:26 UTC Pre-commit check for 199eab2 has started.
⚪ 2024-07-11 17:48:31 UTC Build linux-x86_64-release-asan is running...
🔴 2024-07-11 17:56:25 UTC Build failed. see the build logs.
🔴 2024-07-11 17:56:26 UTC Tests run skipped.

github-actions · 2024-07-11T17:50:07Z

⚪ 2024-07-11 17:50:06 UTC Pre-commit check for 199eab2 has started.
⚪ 2024-07-11 17:53:36 UTC Build linux-x86_64-release-clang14 is running...
🔴 2024-07-11 18:02:24 UTC Build failed. see the build logs.

github-actions · 2024-07-11T17:59:02Z

⚪ 2024-07-11 17:59:02 UTC Pre-commit check for 199eab2 has started.
⚪ 2024-07-11 18:02:27 UTC Build linux-x86_64-relwithdebinfo is running...
🔴 2024-07-11 18:12:24 UTC Build failed. see the build logs.
🔴 2024-07-11 18:12:25 UTC Tests run skipped.

github-actions · 2024-07-12T17:10:20Z

⚪ 2024-07-12 17:10:19 UTC Pre-commit check for 6cb9936 has started.
⚪ 2024-07-12 17:13:38 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-12 17:31:25 UTC Build failed. see the logs.

github-actions · 2024-07-12T17:10:48Z

⚪ 2024-07-12 17:10:48 UTC Pre-commit check for 6cb9936 has started.
⚪ 2024-07-12 17:14:28 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-12 17:39:17 UTC Build failed. see the logs.
🔴 2024-07-12 17:39:21 UTC Test run completed, no test results found for commit 701df20.

github-actions · 2024-07-12T17:17:02Z

⚪ 2024-07-12 17:17:02 UTC Pre-commit check for 6cb9936 has started.
⚪ 2024-07-12 17:20:09 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-12 17:42:54 UTC Build failed. see the logs.
🔴 2024-07-12 17:42:58 UTC Test run completed, no test results found for commit 701df20.

github-actions · 2024-07-16T14:07:37Z

⚪ 2024-07-16 14:07:36 UTC Pre-commit check for 781c757 has started.
⚪ 2024-07-16 14:10:44 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 14:28:05 UTC Test run completed, no test results found for commit ed966c3.
⚫ 2024-07-16 14:28:17 UTC Check cancelled

github-actions · 2024-07-16T14:07:49Z

⚪ 2024-07-16 14:07:48 UTC Pre-commit check for 781c757 has started.
⚪ 2024-07-16 14:11:01 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 14:28:14 UTC Test run completed, no test results found for commit ed966c3.
⚫ 2024-07-16 14:28:37 UTC Check cancelled

github-actions · 2024-07-16T14:09:30Z

⚪ 2024-07-16 14:09:29 UTC Pre-commit check for 781c757 has started.
⚪ 2024-07-16 14:12:34 UTC Build+Tests linux-x86_64-release-clang14 is running...
⚫ 2024-07-16 14:28:34 UTC Check cancelled

github-actions · 2024-07-16T14:30:09Z

⚪ 2024-07-16 14:30:09 UTC Pre-commit check for 70ee1cd has started.
⚪ 2024-07-16 14:33:19 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 14:39:45 UTC Build failed. see the logs.
🔴 2024-07-16 14:39:49 UTC Test run completed, no test results found for commit f4f1616.

github-actions · 2024-07-16T14:30:12Z

⚪ 2024-07-16 14:30:11 UTC Pre-commit check for 70ee1cd has started.
⚪ 2024-07-16 14:33:24 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-16 14:40:32 UTC Build failed. see the logs.

github-actions · 2024-07-16T14:31:46Z

⚪ 2024-07-16 14:31:46 UTC Pre-commit check for 70ee1cd has started.
⚪ 2024-07-16 14:35:24 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 14:49:51 UTC Build failed. see the logs.
🔴 2024-07-16 14:49:56 UTC Test run completed, no test results found for commit f4f1616.

github-actions · 2024-07-16T15:49:42Z

⚪ 2024-07-16 15:49:42 UTC Pre-commit check for e20ef44 has started.
⚪ 2024-07-16 15:53:23 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 16:13:11 UTC Test run completed, no test results found for commit a2130d7.
⚫ 2024-07-16 16:13:19 UTC Check cancelled

github-actions · 2024-07-16T15:55:37Z

⚪ 2024-07-16 15:55:37 UTC Pre-commit check for e20ef44 has started.
⚪ 2024-07-16 15:59:20 UTC Build+Tests linux-x86_64-release-clang14 is running...
⚫ 2024-07-16 16:13:18 UTC Check cancelled

github-actions · 2024-07-16T16:00:16Z

⚪ 2024-07-16 16:00:15 UTC Pre-commit check for e20ef44 has started.
⚪ 2024-07-16 16:03:25 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 16:13:12 UTC Test run completed, no test results found for commit a2130d7.
⚫ 2024-07-16 16:13:34 UTC Check cancelled

github-actions · 2024-07-16T16:15:01Z

⚪ 2024-07-16 16:15:01 UTC Pre-commit check for 096203a has started.
⚪ 2024-07-16 16:18:08 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-16 16:40:16 UTC Build failed. see the logs.

github-actions · 2024-07-16T16:15:03Z

⚪ 2024-07-16 16:15:02 UTC Pre-commit check for 096203a has started.
⚪ 2024-07-16 16:18:08 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 16:41:02 UTC Build failed. see the logs.
🔴 2024-07-16 16:41:07 UTC Test run completed, no test results found for commit fcbe9b8.

github-actions · 2024-07-16T16:15:49Z

⚪ 2024-07-16 16:15:49 UTC Pre-commit check for 096203a has started.
⚪ 2024-07-16 16:19:27 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 16:53:00 UTC Build failed. see the logs.
🔴 2024-07-16 16:53:05 UTC Test run completed, no test results found for commit fcbe9b8.

github-actions · 2024-07-16T17:08:50Z

⚪ 2024-07-16 17:08:50 UTC Pre-commit check for 2b6bdcd has started.
⚪ 2024-07-16 17:11:56 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 17:52:15 UTC Build failed. see the logs.
🔴 2024-07-16 17:52:19 UTC Test run completed, no test results found for commit 7b0d213.

github-actions · 2024-07-16T17:10:08Z

⚪ 2024-07-16 17:10:08 UTC Pre-commit check for 2b6bdcd has started.
⚪ 2024-07-16 17:13:48 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-16 17:54:28 UTC Build failed. see the logs.

github-actions · 2024-07-16T17:10:43Z

⚪ 2024-07-16 17:10:43 UTC Pre-commit check for 2b6bdcd has started.
⚪ 2024-07-16 17:13:48 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 18:12:26 UTC Build failed. see the logs.
🔴 2024-07-16 18:12:33 UTC Test run completed, no test results found for commit 7b0d213.

github-actions · 2024-07-17T09:37:00Z

⚪ 2024-07-17 09:37:00 UTC Pre-commit check for 0ff107b has started.
⚪ 2024-07-17 09:40:16 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-17 10:47:52 UTC Build failed. see the logs.
🔴 2024-07-17 10:47:58 UTC Test run completed, no test results found for commit 7e374c5.

github-actions · 2024-07-17T09:37:04Z

⚪ 2024-07-17 09:37:03 UTC Pre-commit check for 0ff107b has started.
⚪ 2024-07-17 09:40:18 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-17 10:22:38 UTC Build failed. see the logs.

github-actions · 2024-07-17T09:37:07Z

⚪ 2024-07-17 09:37:07 UTC Pre-commit check for 0ff107b has started.
⚪ 2024-07-17 09:40:14 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-17 10:21:22 UTC Build failed. see the logs.
🔴 2024-07-17 10:21:27 UTC Test run completed, no test results found for commit 7e374c5.

github-actions · 2024-07-29T20:52:04Z

⚪ 2024-07-29 20:52:04 UTC Pre-commit check for 6c55da3 has started.
⚪ 2024-07-29 20:55:31 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-29 21:39:07 UTC Build successful.

github-actions · 2024-07-29T20:52:09Z

⚪ 2024-07-29 20:52:09 UTC Pre-commit check for 6c55da3 has started.
⚪ 2024-07-29 20:55:31 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-29 23:35:46 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13565	12966	0	87	333	179

🟢 2024-07-29 23:37:06 UTC Build successful.
🟡 2024-07-29 23:37:59 UTC ydbd size 5.2 GiB changed* by +677.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `f186ded`	merge: `6c55da3`	diff	diff %
ydbd size	5 570 111 808 Bytes	5 570 805 432 Bytes	+677.4 KiB	+0.012%
ydbd stripped size	1 199 062 832 Bytes	1 199 170 736 Bytes	+105.4 KiB	+0.009%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-29T20:52:12Z

⚪ 2024-07-29 20:52:12 UTC Pre-commit check for 6c55da3 has started.
⚪ 2024-07-29 20:55:30 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-29 23:32:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75037	61472	0	1	13551	13

⚪ 2024-07-29 23:40:23 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-07-29 23:49:10 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14 (only retried tests)	5	0	0	1	8

🟢 2024-07-29 23:56:16 UTC Build successful.
🟡 2024-07-29 23:56:47 UTC ydbd size 8.1 GiB changed* by +1.3 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `f186ded`	merge: `6c55da3`	diff	diff %
ydbd size	8 676 595 120 Bytes	8 677 984 288 Bytes	+1.3 MiB	+0.016%
ydbd stripped size	471 822 560 Bytes	471 856 480 Bytes	+33.1 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot added new-feature not-for-changelog labels Jul 11, 2024

kunga removed the new-feature label Jul 12, 2024

kunga changed the title ~~Memory-controller~~ Memory Controller Jul 16, 2024

kunga force-pushed the memory-controller branch from 701df20 to ed966c3 Compare July 16, 2024 14:06

kunga self-assigned this Jul 16, 2024

kunga added 21 commits July 29, 2024 20:48

ydbd peerdir

146eb56

fixing build

40defe7

recurse for

8a6cef1

actor tests

b23bda6

optional config limit

72a234d

fix build

b52950a

better ya.make

b33ad13

fix actor id

a9b3f48

todo

3ecc495

metric fixes

4d32a3f

clean up

4bd7585

fix counters

84eb372

todo

f6c7ab6

reservation counter

372912a

other consumption

08c462c

pass config limit to memory controller

5296c42

fix registered handle

d135570

fix TObjectStorageListingTest::ManyDeletes

4b05269

don't apply mem limit without hard limit

10f6c95

cr fixes

d9e6ab7

add config and counters tests

6639983

kunga dismissed azevaykin’s stale review via 6639983 July 29, 2024 20:48

kunga force-pushed the memory-controller branch from 3f09792 to 6639983 Compare July 29, 2024 20:48

azevaykin approved these changes Jul 30, 2024

View reviewed changes

kunga merged commit 9e244eb into ydb-platform:main Jul 30, 2024
10 of 12 checks passed

kunga mentioned this pull request Jul 30, 2024

Improve Memory Controller #7233

Merged

StekPerepolnen mentioned this pull request Aug 8, 2024

tenantinfo handler added DatabaseStorage #7554

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory Controller #6587

Memory Controller #6587

kunga commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading

Memory Controller #6587

Memory Controller #6587

Conversation

kunga commented Jul 11, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

Example 1

Example 2

github-actions bot commented Jul 11, 2024 • edited Loading

github-actions bot commented Jul 11, 2024 • edited Loading

github-actions bot commented Jul 11, 2024 • edited Loading

github-actions bot commented Jul 12, 2024 • edited Loading

github-actions bot commented Jul 12, 2024 • edited Loading

github-actions bot commented Jul 12, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 16, 2024 • edited Loading

github-actions bot commented Jul 17, 2024 • edited Loading

github-actions bot commented Jul 17, 2024 • edited Loading

github-actions bot commented Jul 17, 2024 • edited Loading

github-actions bot commented Jul 29, 2024 • edited Loading

github-actions bot commented Jul 29, 2024 • edited Loading

github-actions bot commented Jul 29, 2024 • edited Loading

kunga commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024 •

edited

Loading