Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport to 6.1] util/backtrace: Optimize formatter to reduce memory allocation overhead #9

Closed

Conversation

tchaikov
Copy link

@tchaikov tchaikov commented Feb 6, 2025

This commit addresses a critical memory allocation issue in backtrace formatting by directly specializing fmt::formatter for backtrace types, eliminating dependency on iostream-based formatting.

Problem:
When Seastar applications experience high memory pressure, logging backtrace information could fail due to additional memory allocation required by iostream formatting. This resulted in errors like:

ERROR 2024-12-10 01:59:16,905 [shard 0:main] seastar_memory - seastar/src/core/memory.cc:2126 @void seastar::memory::maybe_dump_memory_diagnostics(size_t, bool): failed to log message: fmt='Failed to allocate {} bytes at {}': std::__ios_failure (error iostream:1, basic_ios::clear: iostream error)"

Solution:

  • Implement direct fmt::formatter specialization for backtrace
  • Remove reliance on operator<< for string representation
  • Reduce memory allocation pressure during out-of-memory scenarios
  • Improve reliability of backtrace logging under extreme memory constraints

Compatibility Improvements:

  • Ensure consistent fmt::formatter availability across fmt library versions
  • Deprecate two simple_backtrace constructors previously added in 095a07b, as they are no longer needed with fmt::join
  • Maintain robust formatting approach independent of library version

Impact:
This change enhances system resilience by enabling backtrace logging even under significant memory pressure, providing more reliable post-mortem debugging information.

Signed-off-by: Kefu Chai kefu.chai@scylladb.com
(cherry picked from commit 3133ecd)

Parent PR: scylladb/seastar#2572
Refs scylladb/scylladb#22727

This commit addresses a critical memory allocation issue in backtrace
formatting by directly specializing fmt::formatter for backtrace types,
eliminating dependency on iostream-based formatting.

Problem:
When Seastar applications experience high memory pressure, logging
backtrace information could fail due to additional memory allocation
required by iostream formatting. This resulted in errors like:

```
ERROR 2024-12-10 01:59:16,905 [shard 0:main] seastar_memory - seastar/src/core/memory.cc:2126 @void seastar::memory::maybe_dump_memory_diagnostics(size_t, bool): failed to log message: fmt='Failed to allocate {} bytes at {}': std::__ios_failure (error iostream:1, basic_ios::clear: iostream error)"
```

Solution:
- Implement direct fmt::formatter specialization for backtrace
- Remove reliance on operator<< for string representation
- Reduce memory allocation pressure during out-of-memory scenarios
- Improve reliability of backtrace logging under extreme memory constraints

Compatibility Improvements:
- Ensure consistent `fmt::formatter` availability across fmt library versions
- Deprecate two `simple_backtrace` constructors previously added in
  095a07b, as they are no longer needed with fmt::join
- Maintain robust formatting approach independent of library version

Impact:
This change enhances system resilience by enabling backtrace logging even
under significant memory pressure, providing more reliable post-mortem
debugging information.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
(cherry picked from commit 3133ecd)

Parent PR: scylladb/seastar#2572
Refs scylladb/scylladb#22727
@yaronkaikov yaronkaikov requested a review from denesb February 11, 2025 08:19
@denesb
Copy link

denesb commented Feb 11, 2025

Compilation failed with g++, but we probably don't care about that in private release branches.

denesb pushed a commit that referenced this pull request Feb 11, 2025
This commit addresses a critical memory allocation issue in backtrace
formatting by directly specializing fmt::formatter for backtrace types,
eliminating dependency on iostream-based formatting.

Problem:
When Seastar applications experience high memory pressure, logging
backtrace information could fail due to additional memory allocation
required by iostream formatting. This resulted in errors like:

```
ERROR 2024-12-10 01:59:16,905 [shard 0:main] seastar_memory - seastar/src/core/memory.cc:2126 @void seastar::memory::maybe_dump_memory_diagnostics(size_t, bool): failed to log message: fmt='Failed to allocate {} bytes at {}': std::__ios_failure (error iostream:1, basic_ios::clear: iostream error)"
```

Solution:
- Implement direct fmt::formatter specialization for backtrace
- Remove reliance on operator<< for string representation
- Reduce memory allocation pressure during out-of-memory scenarios
- Improve reliability of backtrace logging under extreme memory constraints

Compatibility Improvements:
- Ensure consistent `fmt::formatter` availability across fmt library versions
- Deprecate two `simple_backtrace` constructors previously added in
  095a07b, as they are no longer needed with fmt::join
- Maintain robust formatting approach independent of library version

Impact:
This change enhances system resilience by enabling backtrace logging even
under significant memory pressure, providing more reliable post-mortem
debugging information.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
(cherry picked from commit 3133ecd)

Parent PR: scylladb/seastar#2572
Refs scylladb/scylladb#22727
Close #9
@denesb
Copy link

denesb commented Feb 11, 2025

Closed by af8ae07.
Submodule update: scylladb/scylladb@b567d60.

@denesb denesb closed this Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants