Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rasdaemon crash during bootup on AMD CPU #19023

Merged
merged 2 commits into from
May 22, 2024

Conversation

vivekrnv
Copy link
Contributor

@vivekrnv vivekrnv commented May 21, 2024

Why I did it

Booting SONiC on a AMD EPYC 16-Core CPU is causing rasdaemon to crash. This is not a major blocker because rasdaemon eventually restarts and is stable after a point.

Coredump stack trace:

[Thread debugging using libthread_db enabled]                                                                                                                                                                                                                               
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".                                                                                                                                                                                                  
Core was generated by `/usr/sbin/rasdaemon -f -r'.                                                                                                                                                                                                                          
Program terminated with signal SIGBUS, Bus error.                                                                                                                                                                                                                           
#0  0x00007f74f62af7f4 in sqlite3_finalize () from /lib/x86_64-linux-gnu/libsqlite3.so.0                                                                                                                                                                                    
[Current thread is 1 (Thread 0x7f73c8ff96c0 (LWP 17416))]    

Known issue for rasdaemon: mchehab/rasdaemon#77

Fixed here:
mchehab/rasdaemon@f1ea763

Unfortunately this fix is not present in the default bookworm version. So, backported the fix and compiled rasdaemon from source

Here is the patch: https://sources.debian.org/patches/rasdaemon/0.8.0-2/0001-Check-CPUs-online-not-configured.patch/

Work item tracking
  • Microsoft ADO (number only):

How I did it

How to verify it

Booted the image built with these changes and no issue in observed

May 16 22:26:37.813343 sonic INFO systemd[1]: Starting rasdaemon.service - RAS daemon to log the RAS events...
May 16 22:26:40.149164 sonic INFO rasdaemon[12059]: rasdaemon: Listening to events for cpus 0 to 31
May 16 22:26:40.265987 sonic INFO systemd[1]: Started rasdaemon.service - RAS daemon to log the RAS events.

Before this change:

May 13 15:42:42.584655 sonic INFO rasdaemon[11724]: rasdaemon: Listening to events for cpus 0 to 127
May 13 15:42:42.593049 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 32
May 13 15:42:42.593120 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 33
May 13 15:42:42.593188 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 34
May 13 15:42:42.593264 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 35
May 13 15:42:42.593333 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 36
May 13 15:42:42.593400 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 37
May 13 15:42:42.593480 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 38
May 13 15:42:42.593549 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 39
May 13 15:42:42.593615 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 40
May 13 15:42:42.593684 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 41
May 13 15:42:42.593749 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 42
May 13 15:42:42.593819 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 43
May 13 15:42:42.593884 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 44
May 13 15:42:42.593952 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 45
May 13 15:42:42.594026 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 46
May 13 15:42:42.594565 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 47
May 13 15:42:42.594635 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 48
May 13 15:42:42.594704 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 49
May 13 15:42:42.594769 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 50
May 13 15:42:42.594836 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 51
May 13 15:42:42.594901 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 52
May 13 15:42:42.594968 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 53
May 13 15:42:42.595034 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 54
May 13 15:42:42.595099 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 55
May 13 15:42:42.595164 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 56
May 13 15:42:42.595230 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 57
May 13 15:42:42.595296 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 58
May 13 15:42:42.595371 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 59
May 13 15:42:42.595437 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 60
May 13 15:42:42.595504 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 61
May 13 15:42:42.595569 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 62
May 13 15:42:42.595635 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 63
May 13 15:42:42.595702 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 64
May 13 15:42:42.595767 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 65
May 13 15:42:42.595834 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 66
May 13 15:42:42.595899 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 67
May 13 15:42:42.595965 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 68
May 13 15:42:42.596031 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 69
May 13 15:42:42.596099 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 70
May 13 15:42:42.596165 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 71
May 13 15:42:42.596230 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 72
May 13 15:42:42.596304 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 73
May 13 15:42:42.596375 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 74
May 13 15:42:42.596440 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 75
May 13 15:42:42.596506 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 76
May 13 15:42:42.596580 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 77
May 13 15:42:42.596647 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 78
May 13 15:42:42.596713 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 79
May 13 15:42:42.596778 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 80
May 13 15:42:42.596874 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 81
May 13 15:42:42.596949 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 82
May 13 15:42:42.597021 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 83
May 13 15:42:42.597090 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 84
May 13 15:42:42.597161 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 85
May 13 15:42:42.597233 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 86
May 13 15:42:42.597303 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 87
May 13 15:42:42.597374 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 88
May 13 15:42:42.597457 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 89
May 13 15:42:42.597528 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 90
May 13 15:42:42.597599 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 91
May 13 15:42:42.597669 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 92
May 13 15:42:42.597740 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 93
May 13 15:42:42.597809 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 94
May 13 15:42:42.597881 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 95
May 13 15:42:42.597953 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 96
May 13 15:42:42.598025 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 97
May 13 15:42:42.598096 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 98
May 13 15:42:42.598174 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 99
May 13 15:42:42.598254 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 100
May 13 15:42:42.598323 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 101
May 13 15:42:42.598390 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 102
May 13 15:42:42.598458 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 103
May 13 15:42:42.598534 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 104
May 13 15:42:42.598600 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 105
May 13 15:42:42.598665 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 106
May 13 15:42:42.598733 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 107
May 13 15:42:42.598798 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 108
May 13 15:42:42.598862 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 109
May 13 15:42:42.598929 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 110
May 13 15:42:42.598996 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 111
May 13 15:42:42.599062 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 112
May 13 15:42:42.599136 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 113
May 13 15:42:42.599203 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 114
May 13 15:42:42.599268 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 115
May 13 15:42:42.599335 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 116
May 13 15:42:42.599400 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 117
May 13 15:42:42.599498 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 118
May 13 15:42:42.599610 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 119
May 13 15:42:42.599708 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 120
May 13 15:42:42.599815 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 121
May 13 15:42:42.600479 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 122
May 13 15:42:42.600573 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 123
May 13 15:42:42.600660 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 124
May 13 15:42:42.600755 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 125
May 13 15:42:42.600858 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 126
May 13 15:42:42.600966 sonic INFO rasdaemon[11724]: rasdaemon: Error on CPU 127

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

vivekrnv and others added 2 commits May 17, 2024 20:10
Signed-off-by: Vivek Reddy <vkarri@nvidia.com>
@vivekrnv vivekrnv requested review from saiarcot895 and dgsudharsan and removed request for lguohan, qiluo-msft and xumia May 21, 2024 00:14
@vivekrnv
Copy link
Contributor Author

/azpw run Azure.sonic-buildimage

@mssonicbld
Copy link
Collaborator

/AzurePipelines run Azure.sonic-buildimage

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@liat-grozovik liat-grozovik merged commit 3a8ba71 into sonic-net:master May 22, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants