Ice memory reduction - enable ice stats flag #1947

disa6302 · 2024-03-08T23:10:34Z

Issue #, if available:

What was changed?

Switched to dynamic allocation for all ICE diagnostic members in IceAgent and added a new flag to allow it to be enabled through kvsRtcConfiguration. Disabled by default.
Modified sizes of string members in some of these structs to reduce allocation size.

Why was it changed?

To save memory. ICE diagnostics need not be a mandatory requirement, but must be made configurable to allow opt-in.

How was it changed?

Added new field in KvsRtcConfiguration -> enableIceStats
Switched to dynamic members in IceAgent struct.
Set the flag to default.

Memory consumption before this change was at 4.05MB. Also, the sizes of the structs before this change:

Size of ICE agent: 167184 bytes
Size of ICE candidate pair: 264 bytes

Memory consumption after this change (with disabled ICE diagnostics) is 3.85MB. There was about 200KB reduction in size. Size of the structs after this change:

Size of ICE agent: 22712 bytes
Size of ice candidate pair: 88 bytes

What testing was done for the changes?

Modified the flags to confirm it all works as expected.
Metrics unit tests run to confirm the tests run as expected.
Added a new unit test with disabled ICE stats.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This reverts commit ad7d02f.

This reverts commit bf9a2ee.

codecov-commenter · 2024-03-11T18:16:26Z

Codecov Report

Attention: Patch coverage is 32.38636% with 119 lines in your changes are missing coverage. Please review.

Project coverage is 75.22%. Comparing base (0e56515) to head (095386c).
Report is 3 commits behind head on develop.

❗ Current head 095386c differs from pull request most recent head 563a88c. Consider uploading reports for the commit 563a88c to get more accurate results

Files	Patch %	Lines
src/source/Ice/IceAgent.c	38.77%	60 Missing ⚠️
samples/Common.c	0.00%	35 Missing ⚠️
src/source/Metrics/Metrics.c	44.18%	24 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1947      +/-   ##
===========================================
- Coverage    75.62%   75.22%   -0.41%     
===========================================
  Files           49       49              
  Lines        13839    13875      +36     
===========================================
- Hits         10466    10437      -29     
- Misses        3373     3438      +65

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/source/Ice/IceAgent.c

src/source/Metrics/Metrics.c

samples/Common.c

sirknightj · 2024-10-09T17:18:59Z

src/source/Ice/IceAgent.c

@@ -88,7 +92,7 @@ STATUS createIceAgent(PCHAR username, PCHAR password, PIceAgentCallbacks pIceAge

    // Pre-allocate stun packets

-    // no other attributes needed: https://tools.ietf.org/html/rfc8445#section-11
+    // no other attribtues needed: https://tools.ietf.org/html/rfc8445#section-11


* Change params size * Use dyanmic allocation and flag for ice stats * Debug 1 * Revert "Debug 1" This reverts commit ad7d02f. * Revert "Use dyanmic allocation and flag for ice stats" This reverts commit bf9a2ee. * Working version * enable flag in samples * Add unit test * Fix bug * README for the flag * Update readme

* Ice memory reduction - enable ice stats flag (#1947) * Change params size * Use dyanmic allocation and flag for ice stats * Debug 1 * Revert "Debug 1" This reverts commit ad7d02f. * Revert "Use dyanmic allocation and flag for ice stats" This reverts commit bf9a2ee. * Working version * enable flag in samples * Add unit test * Fix bug * README for the flag * Update readme * Fix readme typos * clang format * Add ENABLE_STATS_CALCULATION_CONTROL CMake option and compiler flag. Update ReadMe. * Add IceStatsControlOn tests * Add CI test forENABLE_STATS_CALCULATION_CONTROL=TRUE * Clang formatting * Remove unused iceAgentAddConfig * Remove unintended readme merge changes "Controlling RTP rolling buffer capacity" * Address comments, add null check * Cleanup unused variable * Rename and negate variable * Move setting of enableIceStats to fix segfault * Revert member size savings changes (to be applied in separate PR) * Move other instance setting of enableIceStats, clang format. * Fix setting of enableIceStats in tests * Address comment --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu>

* Reduce hash table size for SDP transceivers (#2067) * Reduce hash table size for SDP transceivers (#1914) * Modify hash table sizes * Add logs * Fix hash table size to be set according to number of m-lines and unique supported codecs * CI branch * Update actions to use node20 * Unit test for fake transceiver * Cleanup * Fix build failure * Fix unit test * Initialize to 0 * PR description check: Reduce minimum character count and print the current character count * Use PR number instead of manually constructing link, add fetch error logging * Print out the PR link * Debug the CI * Fix the extraction logic * Clang-format --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * Add job to run the sample and check for memory leak (#2071) * Add job to run the sample and check for memory leak * Simulate a memory leak * MEMALLOC instead of MALLOC * Remove simulated memory leak * Print the exit statuses in case of failure * Move offer from stack to heap (#2070) * Ensure remote description does not live in memory for viewer duration (#1915) * Ensure remote description does not live in memory for viewer duration * Fix memory leak * Free the allocated memory upon failure, also add missing enter, leave and chk_log_error in peerconnection * Working reduced stack sizes. Adjusted default stack size to 64kb, moved Offer/Answer messages to be on heap instead of stack * Revert setting the stack sizes by default; Samples: revert disableSenderSideBandwidthEstimation to TRUE by default * Add missing malloc out of memory checks * Undo samples/Common.c changes * Add chk_log_err to freeJitterBuffer and freeKvsRtpTransceiver * Address comments: add additional null checks and adjust comments * Clang-format --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> Co-authored-by: James Delaplane <delaplan@amazon.com> * Memory Optimization - ICE Agent Stats (#1947) (#2074) * Ice memory reduction - enable ice stats flag (#1947) * Change params size * Use dyanmic allocation and flag for ice stats * Debug 1 * Revert "Debug 1" This reverts commit ad7d02f. * Revert "Use dyanmic allocation and flag for ice stats" This reverts commit bf9a2ee. * Working version * enable flag in samples * Add unit test * Fix bug * README for the flag * Update readme * Fix readme typos * clang format * Add ENABLE_STATS_CALCULATION_CONTROL CMake option and compiler flag. Update ReadMe. * Add IceStatsControlOn tests * Add CI test forENABLE_STATS_CALCULATION_CONTROL=TRUE * Clang formatting * Remove unused iceAgentAddConfig * Remove unintended readme merge changes "Controlling RTP rolling buffer capacity" * Address comments, add null check * Cleanup unused variable * Rename and negate variable * Move setting of enableIceStats to fix segfault * Revert member size savings changes (to be applied in separate PR) * Move other instance setting of enableIceStats, clang format. * Fix setting of enableIceStats in tests * Address comment --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * Memory Optimization - Refactor TWCC (#1934) (#2075) * Refactor TWCC (#1934) * Remove stackqueue usage * Use hashtable instead..working logic * Cleanup, increase hash table size, fix loop bug * method 2, array of pointers * Add rolling window * rolling window with hashtable * hash table with rw * Fix bug * Fix twcc unit test * Cleanup rw logic * Cleanup * Cleanup logic * Update README * unused var fix * Use defines for hash table size * Address comments, disable TWCC by default * readme * Fix windows gst issue * Comments * Remove enableIceStats merge conflict * Address comments * Fix left over typo * Address comments, refactor updaing of TWCC hash table in onRtcpTwccPacket for testing * Clang format, fix compiler Werror * Correct typos * Fix "expression result unused" Werrors * Remove unused variable * Address comments * Fix endless loop * Address comment * Add removal of null hashTable items, add test for null item in hashTable * Address comments * Cleanup freeing of KvsPeerConnection * Clang format --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * Shorten Stats String Lengths (#2079) * Memory Optimization - Rolling Buffer (#2077) * Rb size config (#1921) * Reduce default size * Rolling buffer configurability * Add configurability * Allow setting parameter through trx instead of config * Add UT, fix the members to DOUBLE * Remove unnecessary fields * Rolling buffer readme update * Use enums for min values * Add usage example in samples * Fix unit test * Memset init in UTs * Modify the bitrates in the samples based on codec used in our samples * Modify unit tests, add video/audio specific defaults * Add missing null check * Readme update * Address nits * Clang fix * Logs for RB * set explicit type cast * add max values * Refactor RB logic, readme update, sample cleanup * nits * Fix unit tests * return * update description in header * fix unused var * minor changes * typo fix * review comments * sdp, stun, rtp * comments * cancelled builds * log status * sctp * Revert unrelated sample changes * Revert unrelated CMake changes * Revert unrelated test changes * Revert unrelated ReadMe changes * Fix DEFAULT_MTU_SIZE variable name * Cleanup merge changes * Fixup and comment on Rtp.c * Fixup Rtp.h * Fixup ReadMe * Clang format * Revert all sample changes * Address comments * Add back accidentally deleted line * Address comment * Update README.md --------- Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * Add option to specify stack size to pthread_create (#2073) * Thread stack sizes (#1924) * Working reduced stack sizes. Adjusted default stack size to 64kb, moved Offer/Answer messages to be on heap instead of stack * Custom stack sizes in Common.c * CMake flag working and added to README * Adjust connection listener stack size for datachannels * Change stack size for media related threads and connection listener again. Add README notes to highlight this potential problem threads for other applications * Unnecessary memset * nit picks * Update CMakeLists.txt fix for ZLIB since it is not found on Windows, that way we can verify the stack side change on Windows platform, previously it was not running at all due to build fail. * Update CMake dependency back to develop since the other PRs have merged * README update --------- Co-authored-by: Hassan Sahibzada <hsahibza@amazon.com> Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * Stack size configurablility * Revert setting specific thread stack sizes in the sample and SDK * Check develop branch * Check stack size is forwarded to PIC correctly * Set e * Remove duplicate mkdir build * Print out CMake logs in case of error * Missing fi * Remove square brackets * Dont build samples * Build samples * Pass the stack size down and dont build samples * Remove constrained device * Bump kvsCommonLws from 1.5.4RC to 1.5.4 * Add v * Update README.md --------- Co-authored-by: jdelapla <delaplan@amazon.com> Co-authored-by: Hassan Sahibzada <hsahibza@amazon.com> Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> * SDP Attribute Fixes (#2082) * fix nack pli overwrite bug (#2035) * fix nack pli overwrite bug * create generic strings instead of hard coding * remove stale comment in tests * Check that the sdp lines are not missing, append SDP_LINE_SEPARATOR to lines checked * Remove duplicate session attribute lines, add missing attribute name * Clang format --------- Co-authored-by: Hassan Sahibzada <hsahibza@amazon.com> * ReadMe Fixes (#2083) * Add copy code buttons to github clone commands (#1928) * Add copy code buttons to github clone commands * Add language specifier * build -> built (#1925) * Update clone command * Remove Duplicate ReadMe Lines (#2085) * Update README.md * Update make commad * Bump project version in CMake (#2086) * Upgrade to macos-13 and 15, use DRY for Mac jobs (#2090) * Upgrade to macos-13 and 15, use DRY for Mac jobs, use docker for ubuntu jobs * Revert Ubuntu docker changes * Remove Mac+GCC+Tests combo * Install GStreamer for the mac jobs to check the GStreamer sample builds * Remove unnecessary installs * Build and run the tests without AWS SDK * Add D * Samples: Fit the RTP transceiver rolling buffers to fit the set of sample frames (#2089) * Properly fit the rolling buffers to fit the set of sample frames * Clang-format * Adjust the comment for the rolling buffer configuration for the sample frames * Bump the version from 1.11.1 to 1.12.0 (#2092) * Specify configureTransceiverRollingBuffer as a Public API --------- Co-authored-by: sirknightj <jggunawa@amazon.com> Co-authored-by: Divya Sampath Kumar <disa6302@colorado.edu> Co-authored-by: James Delaplane <delaplan@amazon.com> Co-authored-by: Hassan Sahibzada <hsahibza@amazon.com>

disa6302 added 8 commits March 7, 2024 12:49

Change params size

666cf82

Use dyanmic allocation and flag for ice stats

bf9a2ee

Debug 1

ad7d02f

Revert "Debug 1"

5ddcf3c

This reverts commit ad7d02f.

Revert "Use dyanmic allocation and flag for ice stats"

08f0ac7

This reverts commit bf9a2ee.

Working version

dcd62b7

enable flag in samples

507fe0d

Add unit test

e023092

disa6302 force-pushed the ice-memory-reduction branch from a99d8ab to e023092 Compare March 8, 2024 23:15

disa6302 added 2 commits March 8, 2024 16:13

Fix bug

84c70fb

README for the flag

c9a9d43

disa6302 marked this pull request as ready for review March 11, 2024 21:36

disa6302 changed the title ~~Ice memory reduction~~ Ice memory reduction - enable ice stats flag Mar 11, 2024

stefankiesz reviewed Mar 12, 2024

View reviewed changes

src/source/Ice/IceAgent.c Show resolved Hide resolved

src/source/Ice/IceAgent.c Show resolved Hide resolved

src/source/Ice/IceAgent.c Show resolved Hide resolved

src/source/Metrics/Metrics.c Outdated Show resolved Hide resolved

samples/Common.c Show resolved Hide resolved

disa6302 force-pushed the ice-memory-reduction branch from c3f6650 to cdb36bd Compare March 13, 2024 16:17

jdelapla previously approved these changes Mar 18, 2024

View reviewed changes

Update readme

563a88c

disa6302 dismissed jdelapla’s stale review via 563a88c March 18, 2024 18:47

disa6302 force-pushed the ice-memory-reduction branch from cdb36bd to 563a88c Compare March 18, 2024 18:47

stefankiesz approved these changes Mar 18, 2024

View reviewed changes

jdelapla approved these changes Mar 18, 2024

View reviewed changes

disa6302 merged commit 7be31c1 into develop Mar 18, 2024
30 checks passed

disa6302 deleted the ice-memory-reduction branch March 18, 2024 20:04

sirknightj reviewed Oct 9, 2024

View reviewed changes

stefankiesz mentioned this pull request Nov 1, 2024

Memory Optimization - ICE Agent Stats (#1947) #2074

Merged

stefankiesz mentioned this pull request Nov 13, 2024

Shorten Stats String Lengths #2079

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ice memory reduction - enable ice stats flag #1947

Ice memory reduction - enable ice stats flag #1947

disa6302 commented Mar 8, 2024

codecov-commenter commented Mar 11, 2024 •

edited

Loading

sirknightj Oct 9, 2024

Ice memory reduction - enable ice stats flag #1947

Ice memory reduction - enable ice stats flag #1947

Conversation

disa6302 commented Mar 8, 2024

codecov-commenter commented Mar 11, 2024 • edited Loading

Codecov Report

sirknightj Oct 9, 2024

Choose a reason for hiding this comment

codecov-commenter commented Mar 11, 2024 •

edited

Loading