fix for excessive gen0 in high memory load situation #61884

Maoni0 · 2021-11-21T01:46:02Z

in high memory load situations, when we had to wait for a BGC to finish, we switched to preemptive mode which means GCs could have occurred and replenished the budget. but when we come back from the wait, we use the previous decision on whether the budget was exceeded or not. this causes us to trigger GC incorrect and you would see GCs triggered when the allocated bytes are tiny.

repro -

set complus_GCHeapCount=8
set complus_gcServer=1

corerun \saved\GCPerfSim.dll -tc 32 -tagb 3000 -tlgb 3 -lohar 0 -sohsi 50 -lohsi 100 -pohsi 0 -sohpi 30 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time -printEveryNthIter 300000

when memory load is high (I made it 97%), using GC realmon this is the output - the gen0 alloc (mb) column shows some gen0 GCs triggered when it's < 1mb or just a few mb instead of hundreds of mb -

C:\realmon\src\GCRealTimeMon\bin\Release\net6.0>C:\realmon\src\GCRealTimeMon\bin\Release\net6.0\GCRealTimeMon.exe -p 13508
------- press s for current stats or any other key to exit -------

Monitoring process with name: corerun and pid: 13508
GC#     index |            type |   gen | pause (ms) |                reason | gen0 alloc (mb) | LOH size (mb) | peak/after | gen2 size (mb) |
----------------------------------------------------------------------------------------------------------------------------------------------
GC#       401 | NonConcurrentGC |     0 |      16.15 |            AllocSmall |           0.000 |        12.895 |       1.00 |      7,254.638 |
GC#       402 | NonConcurrentGC |     1 |      25.09 |            AllocSmall |         393.777 |        12.895 |       1.00 |      7,254.638 |
GC#      2278 | NonConcurrentGC |     0 |      26.29 |            AllocSmall |         378.853 |        12.895 |       1.00 |      3,219.551 |
GC#      2279 | NonConcurrentGC |     0 |      24.52 |            AllocSmall |         422.564 |        12.895 |       1.00 |      3,219.551 |
GC#      2280 | NonConcurrentGC |     0 |      24.40 |            AllocSmall |         420.125 |        12.895 |       1.00 |      3,219.551 |
GC#      2281 | NonConcurrentGC |     0 |      24.86 |            AllocSmall |         429.455 |        12.895 |       1.00 |      3,219.551 |
GC#      2283 | NonConcurrentGC |     0 |      26.49 |            AllocSmall |         581.462 |        12.895 |       1.00 |      3,219.551 |
GC#      2282 |    BackgroundGC |     2 |      14.90 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,219.542 |
GC#      2284 | NonConcurrentGC |     2 |     333.70 |            AllocSmall |          14.501 |        12.895 |       1.04 |      3,045.020 |
GC#      2285 | NonConcurrentGC |     0 |      34.91 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,045.020 |
GC#      2286 | NonConcurrentGC |     0 |      17.15 |            AllocSmall |           0.013 |        12.895 |       1.00 |      3,045.020 |
GC#      2287 | NonConcurrentGC |     0 |      17.03 |            AllocSmall |           0.090 |        12.895 |       1.00 |      3,045.020 |
GC#      2288 | NonConcurrentGC |     0 |      16.98 |            AllocSmall |           0.087 |        12.895 |       1.00 |      3,045.020 |
GC#      2289 | NonConcurrentGC |     0 |      17.04 |            AllocSmall |           0.346 |        12.895 |       1.00 |      3,045.020 |
GC#      2290 | NonConcurrentGC |     0 |      16.91 |            AllocSmall |           0.228 |        12.895 |       1.00 |      3,045.020 |
GC#      2291 | NonConcurrentGC |     0 |      17.26 |            AllocSmall |           0.991 |        12.895 |       1.00 |      3,045.020 |
GC#      2292 | NonConcurrentGC |     0 |      17.10 |            AllocSmall |           0.189 |        12.895 |       1.00 |      3,045.020 |
GC#      2293 | NonConcurrentGC |     0 |      16.97 |            AllocSmall |           0.581 |        12.895 |       1.00 |      3,045.020 |
GC#      2294 | NonConcurrentGC |     0 |      17.26 |            AllocSmall |           0.474 |        12.895 |       1.00 |      3,045.020 |
GC#      2295 | NonConcurrentGC |     0 |      18.22 |            AllocSmall |           7.915 |        12.895 |       1.00 |      3,045.020 |
GC#      2296 | NonConcurrentGC |     0 |      23.71 |            AllocSmall |         440.564 |        12.895 |       1.00 |      3,045.020 |
GC#      2297 | NonConcurrentGC |     0 |      23.86 |            AllocSmall |         433.488 |        12.895 |       1.00 |      3,045.020 |
GC#      2298 | NonConcurrentGC |     0 |      24.30 |            AllocSmall |         437.295 |        12.895 |       1.00 |      3,045.020 |
GC#      2299 | NonConcurrentGC |     0 |      24.17 |            AllocSmall |         422.302 |        12.895 |       1.00 |      3,045.020 |
GC#      2300 | NonConcurrentGC |     1 |      39.35 |            AllocSmall |         423.940 |        12.895 |       0.98 |      3,126.880 |
GC#      2301 | NonConcurrentGC |     0 |      24.23 |            AllocSmall |         413.198 |        12.895 |       1.00 |      3,126.880 |
GC#      2302 | NonConcurrentGC |     0 |      26.61 |            AllocSmall |         439.262 |        12.895 |       1.00 |      3,126.880 |
GC#      2303 | NonConcurrentGC |     0 |      25.65 |            AllocSmall |         416.549 |        12.895 |       1.00 |      3,126.880 |
GC#      2304 | NonConcurrentGC |     0 |      25.17 |            AllocSmall |         405.923 |        12.895 |       1.00 |      3,126.880 |
GC#      2305 | NonConcurrentGC |     0 |      24.96 |            AllocSmall |         438.479 |        12.895 |       1.00 |      3,126.880 |
GC#      2306 | NonConcurrentGC |     1 |      42.12 |            AllocSmall |         438.927 |        12.895 |       0.98 |      3,221.068 |
GC#      2307 | NonConcurrentGC |     0 |      24.27 |            AllocSmall |         425.704 |        12.895 |       1.00 |      3,221.068 |
GC#      2308 | NonConcurrentGC |     0 |      24.17 |            AllocSmall |         439.244 |        12.895 |       1.00 |      3,221.068 |
GC#      2309 | NonConcurrentGC |     0 |      24.67 |            AllocSmall |         416.837 |        12.895 |       1.00 |      3,221.068 |
GC#      2310 | NonConcurrentGC |     0 |      25.06 |            AllocSmall |         425.330 |        12.895 |       1.00 |      3,221.068 |
GC#      2311 | NonConcurrentGC |     0 |      26.25 |            AllocSmall |         448.383 |        12.895 |       1.00 |      3,221.068 |
GC#      2313 | NonConcurrentGC |     0 |      26.85 |            AllocSmall |         571.748 |        12.895 |       1.00 |      3,221.068 |
GC#      2312 |    BackgroundGC |     2 |      12.65 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,221.062 |
GC#      2314 | NonConcurrentGC |     2 |     358.62 |            AllocSmall |          14.904 |        12.895 |       1.04 |      3,044.236 |
GC#      2315 | NonConcurrentGC |     0 |      35.18 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,044.236 |
GC#      2316 | NonConcurrentGC |     0 |      18.83 |            AllocSmall |           0.022 |        12.895 |       1.00 |      3,044.236 |
GC#      2317 | NonConcurrentGC |     0 |      17.50 |            AllocSmall |           0.019 |        12.895 |       1.00 |      3,044.236 |
GC#      2318 | NonConcurrentGC |     0 |      17.01 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,044.236 |
GC#      2319 | NonConcurrentGC |     0 |      16.94 |            AllocSmall |           0.036 |        12.895 |       1.00 |      3,044.236 |
GC#      2320 | NonConcurrentGC |     0 |      17.61 |            AllocSmall |           0.309 |        12.895 |       1.00 |      3,044.236 |
GC#      2321 | NonConcurrentGC |     0 |      17.09 |            AllocSmall |           0.103 |        12.895 |       1.00 |      3,044.236 |
GC#      2322 | NonConcurrentGC |     0 |      17.32 |            AllocSmall |           4.160 |        12.895 |       1.00 |      3,044.236 |
GC#      2323 | NonConcurrentGC |     0 |      17.03 |            AllocSmall |           1.006 |        12.895 |       1.00 |      3,044.236 |
GC#      2324 | NonConcurrentGC |     0 |      23.48 |            AllocSmall |         442.973 |        12.895 |       1.00 |      3,044.236 |

with fix this no longer repros.

also got rid of the allocations_allowed check that's unused.

@PeterSolMS I did put in a bit more logic to detect whether I should do the check again. PTAL.

ghost · 2021-11-21T01:46:10Z

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

in high memory load situations, when we had to wait for a BGC to finish, we switch to preemptive mode which means GCs can occur and replenished the budget. but when we come back from the wait, we use the previous decision on whether the budget was exceeded or not. this causes us to trigger GC incorrect and you would see GCs triggered when the allocated bytes are tiny.

repro -

set complus_GCHeapCount=8
set complus_gcServer=1

corerun \saved\GCPerfSim.dll -tc 32 -tagb 3000 -tlgb 3 -lohar 0 -sohsi 50 -lohsi 100 -pohsi 0 -sohpi 30 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time -printEveryNthIter 300000

when memory load is high (I made it 97%), using GC realmon this is the output - the gen0 alloc (mb) column shows some gen0 GCs triggered when it's < 1mb or just a few mb instead of hundreds of mb -

C:\realmon\src\GCRealTimeMon\bin\Release\net6.0>C:\realmon\src\GCRealTimeMon\bin\Release\net6.0\GCRealTimeMon.exe -p 13508
------- press s for current stats or any other key to exit -------

Monitoring process with name: corerun and pid: 13508
GC#     index |            type |   gen | pause (ms) |                reason | gen0 alloc (mb) | LOH size (mb) | peak/after | gen2 size (mb) |
----------------------------------------------------------------------------------------------------------------------------------------------
GC#       401 | NonConcurrentGC |     0 |      16.15 |            AllocSmall |           0.000 |        12.895 |       1.00 |      7,254.638 |
GC#       402 | NonConcurrentGC |     1 |      25.09 |            AllocSmall |         393.777 |        12.895 |       1.00 |      7,254.638 |
GC#      2278 | NonConcurrentGC |     0 |      26.29 |            AllocSmall |         378.853 |        12.895 |       1.00 |      3,219.551 |
GC#      2279 | NonConcurrentGC |     0 |      24.52 |            AllocSmall |         422.564 |        12.895 |       1.00 |      3,219.551 |
GC#      2280 | NonConcurrentGC |     0 |      24.40 |            AllocSmall |         420.125 |        12.895 |       1.00 |      3,219.551 |
GC#      2281 | NonConcurrentGC |     0 |      24.86 |            AllocSmall |         429.455 |        12.895 |       1.00 |      3,219.551 |
GC#      2283 | NonConcurrentGC |     0 |      26.49 |            AllocSmall |         581.462 |        12.895 |       1.00 |      3,219.551 |
GC#      2282 |    BackgroundGC |     2 |      14.90 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,219.542 |
GC#      2284 | NonConcurrentGC |     2 |     333.70 |            AllocSmall |          14.501 |        12.895 |       1.04 |      3,045.020 |
GC#      2285 | NonConcurrentGC |     0 |      34.91 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,045.020 |
GC#      2286 | NonConcurrentGC |     0 |      17.15 |            AllocSmall |           0.013 |        12.895 |       1.00 |      3,045.020 |
GC#      2287 | NonConcurrentGC |     0 |      17.03 |            AllocSmall |           0.090 |        12.895 |       1.00 |      3,045.020 |
GC#      2288 | NonConcurrentGC |     0 |      16.98 |            AllocSmall |           0.087 |        12.895 |       1.00 |      3,045.020 |
GC#      2289 | NonConcurrentGC |     0 |      17.04 |            AllocSmall |           0.346 |        12.895 |       1.00 |      3,045.020 |
GC#      2290 | NonConcurrentGC |     0 |      16.91 |            AllocSmall |           0.228 |        12.895 |       1.00 |      3,045.020 |
GC#      2291 | NonConcurrentGC |     0 |      17.26 |            AllocSmall |           0.991 |        12.895 |       1.00 |      3,045.020 |
GC#      2292 | NonConcurrentGC |     0 |      17.10 |            AllocSmall |           0.189 |        12.895 |       1.00 |      3,045.020 |
GC#      2293 | NonConcurrentGC |     0 |      16.97 |            AllocSmall |           0.581 |        12.895 |       1.00 |      3,045.020 |
GC#      2294 | NonConcurrentGC |     0 |      17.26 |            AllocSmall |           0.474 |        12.895 |       1.00 |      3,045.020 |
GC#      2295 | NonConcurrentGC |     0 |      18.22 |            AllocSmall |           7.915 |        12.895 |       1.00 |      3,045.020 |
GC#      2296 | NonConcurrentGC |     0 |      23.71 |            AllocSmall |         440.564 |        12.895 |       1.00 |      3,045.020 |
GC#      2297 | NonConcurrentGC |     0 |      23.86 |            AllocSmall |         433.488 |        12.895 |       1.00 |      3,045.020 |
GC#      2298 | NonConcurrentGC |     0 |      24.30 |            AllocSmall |         437.295 |        12.895 |       1.00 |      3,045.020 |
GC#      2299 | NonConcurrentGC |     0 |      24.17 |            AllocSmall |         422.302 |        12.895 |       1.00 |      3,045.020 |
GC#      2300 | NonConcurrentGC |     1 |      39.35 |            AllocSmall |         423.940 |        12.895 |       0.98 |      3,126.880 |
GC#      2301 | NonConcurrentGC |     0 |      24.23 |            AllocSmall |         413.198 |        12.895 |       1.00 |      3,126.880 |
GC#      2302 | NonConcurrentGC |     0 |      26.61 |            AllocSmall |         439.262 |        12.895 |       1.00 |      3,126.880 |
GC#      2303 | NonConcurrentGC |     0 |      25.65 |            AllocSmall |         416.549 |        12.895 |       1.00 |      3,126.880 |
GC#      2304 | NonConcurrentGC |     0 |      25.17 |            AllocSmall |         405.923 |        12.895 |       1.00 |      3,126.880 |
GC#      2305 | NonConcurrentGC |     0 |      24.96 |            AllocSmall |         438.479 |        12.895 |       1.00 |      3,126.880 |
GC#      2306 | NonConcurrentGC |     1 |      42.12 |            AllocSmall |         438.927 |        12.895 |       0.98 |      3,221.068 |
GC#      2307 | NonConcurrentGC |     0 |      24.27 |            AllocSmall |         425.704 |        12.895 |       1.00 |      3,221.068 |
GC#      2308 | NonConcurrentGC |     0 |      24.17 |            AllocSmall |         439.244 |        12.895 |       1.00 |      3,221.068 |
GC#      2309 | NonConcurrentGC |     0 |      24.67 |            AllocSmall |         416.837 |        12.895 |       1.00 |      3,221.068 |
GC#      2310 | NonConcurrentGC |     0 |      25.06 |            AllocSmall |         425.330 |        12.895 |       1.00 |      3,221.068 |
GC#      2311 | NonConcurrentGC |     0 |      26.25 |            AllocSmall |         448.383 |        12.895 |       1.00 |      3,221.068 |
GC#      2313 | NonConcurrentGC |     0 |      26.85 |            AllocSmall |         571.748 |        12.895 |       1.00 |      3,221.068 |
GC#      2312 |    BackgroundGC |     2 |      12.65 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,221.062 |
GC#      2314 | NonConcurrentGC |     2 |     358.62 |            AllocSmall |          14.904 |        12.895 |       1.04 |      3,044.236 |
GC#      2315 | NonConcurrentGC |     0 |      35.18 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,044.236 |
GC#      2316 | NonConcurrentGC |     0 |      18.83 |            AllocSmall |           0.022 |        12.895 |       1.00 |      3,044.236 |
GC#      2317 | NonConcurrentGC |     0 |      17.50 |            AllocSmall |           0.019 |        12.895 |       1.00 |      3,044.236 |
GC#      2318 | NonConcurrentGC |     0 |      17.01 |            AllocSmall |           0.000 |        12.895 |       1.00 |      3,044.236 |
GC#      2319 | NonConcurrentGC |     0 |      16.94 |            AllocSmall |           0.036 |        12.895 |       1.00 |      3,044.236 |
GC#      2320 | NonConcurrentGC |     0 |      17.61 |            AllocSmall |           0.309 |        12.895 |       1.00 |      3,044.236 |
GC#      2321 | NonConcurrentGC |     0 |      17.09 |            AllocSmall |           0.103 |        12.895 |       1.00 |      3,044.236 |
GC#      2322 | NonConcurrentGC |     0 |      17.32 |            AllocSmall |           4.160 |        12.895 |       1.00 |      3,044.236 |
GC#      2323 | NonConcurrentGC |     0 |      17.03 |            AllocSmall |           1.006 |        12.895 |       1.00 |      3,044.236 |
GC#      2324 | NonConcurrentGC |     0 |      23.48 |            AllocSmall |         442.973 |        12.895 |       1.00 |      3,044.236 |

with fix this no longer repros.

also got rid of the allocations_allowed check that's unused.

@PeterSolMS I did put it a bit more logic to detect whether I should do the check again. PTAL.

Author:	Maoni0
Assignees:	Maoni0
Labels:	`area-GC-coreclr`
Milestone:	.NET 7.0

src/coreclr/gc/gc.cpp

PeterSolMS

LGTM, but note suggestion to make it more readable...

fix for excessive gen0 in high memory load situation

9c0397a

Maoni0 added the area-GC-coreclr label Nov 21, 2021

Maoni0 added this to the .NET 7.0 milestone Nov 21, 2021

Maoni0 self-assigned this Nov 21, 2021

jeffhandley modified the milestones: .NET 7.0, 7.0.0 Nov 22, 2021

PeterSolMS reviewed Nov 24, 2021

View reviewed changes

src/coreclr/gc/gc.cpp Show resolved Hide resolved

PeterSolMS reviewed Nov 24, 2021

View reviewed changes

src/coreclr/gc/gc.cpp Show resolved Hide resolved

PeterSolMS approved these changes Nov 24, 2021

View reviewed changes

cr feedback

d982a32

Maoni0 merged commit 9ede9a2 into dotnet:main Nov 26, 2021

ghost locked as resolved and limited conversation to collaborators Dec 27, 2021

Maoni0 deleted the excessive_gen0 branch January 26, 2022 05:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix for excessive gen0 in high memory load situation #61884

fix for excessive gen0 in high memory load situation #61884

Maoni0 commented Nov 21, 2021 •

edited

Loading

ghost commented Nov 21, 2021

PeterSolMS left a comment

fix for excessive gen0 in high memory load situation #61884

fix for excessive gen0 in high memory load situation #61884

Conversation

Maoni0 commented Nov 21, 2021 • edited Loading

ghost commented Nov 21, 2021

PeterSolMS left a comment

Choose a reason for hiding this comment

Maoni0 commented Nov 21, 2021 •

edited

Loading