Install systemd-coredump in SONiC to be used as the core dump utility

[rajendra-dendukuri]

Signed-off-by: Rajendra Dendukuri rajendra.dendukuri@broadcom.com

- Why I did it
Added new way to collect and manage application core files.

- How I did it
Core files generated by kernel are created by the host o/s and also stored
on host o/s. This applies for processes running inside container as well.
Containers do not have access to journal as well as core files. Containers
are supposed to have limited access to host o/s and core files and journal
may contain sensitive information.

To improve debugging of crashes inside a container following changes are made:

Install systemd-coredump in base o/s

Remove existing simple coredump facility

Enable persistent journald to store coredump history

Minimal default coredump configuration

Added a coredump-config service to generate coredump configuration

when INSTALL_DEBUG_TOOLS=y is set in the build. systemd-coredump tool
is installed in all containers beside gdb

When SONIC_DEBUGGING_ON=y is set in the build,
/var/log/journal and /var/lib/systemd/coredump are mapped inside container

To inspect a core file, from a container shell, issue below commands

docker exec -ti coredumpctl

- How to verify it
kill -ABRT
coredumpctl list
coredumpctl info

- Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006

- Description for the changelog

Use systemd-coredump for core file management in SONiC

- A picture of a cute animal (not mandatory but encouraged)

[lgtm-com]

This pull request introduces 4 alerts and fixes 1 when merging 4c09bd6 into 6399258 - view on LGTM.com

new alerts:

• 2 for Unused import
• 1 for Except block handles 'BaseException'
• 1 for Module is imported with 'import' and 'import from'

fixed alerts:

• 1 for Except block handles 'BaseException'

[lgtm-com]

This pull request fixes 1 alert when merging a4edf30 into 6399258 - view on LGTM.com

fixed alerts:

• 1 for Except block handles 'BaseException'

[lguohan]

@rajendra-dendukuri , have you check out this pr? sonic-net/sonic-utilities#1091

the coredump-compress support multi-asic scenario, does this pr support multi-asic scenario?

[rajendra-dendukuri]

@rajendra-dendukuri , have you check out this pr? Azure/sonic-utilities#1091

the coredump-compress support multi-asic scenario, does this pr support multi-asic scenario?

coredump-compress is not required and not used in systemd-coredump. All core files are 7z compressed during creating itself.
The PID is also included in the created core filename. Below is an example core file where the PID is 11314.

/var/lib/systemd/coredump/core.hello.0.32e257577c074240b52748c8787164ab.11314.1607033811000000.lz4

[lguohan]

exactly, what missing now is the namespace id. as you noticed in https://github.com/Azure/sonic-utilities/pull/1091/files#diff-c39e4bc8fd7d7b317f057771197ecaeff71913e200097b208235e43fccbdbc74R11, the compress core analyze the environ and get the namespace id of the crashing process. this is need for multi-asic platform since we need to know which namespace the core dump is generated since all orchagent in different namespace have same names and we would like to differentiate them.

[lguohan]

need to support namespace id for multi-asic.