feat: Improved FPE monitoring #2157
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andiwand
reviewed
May 26, 2023
andiwand
reviewed
May 28, 2023
Co-authored-by: Andreas Stefl <stefl.andreas@gmail.com>
Co-authored-by: Andreas Stefl <stefl.andreas@gmail.com>
Codecov Report
@@ Coverage Diff @@
## main #2157 +/- ##
==========================================
- Coverage 49.37% 49.36% -0.02%
==========================================
Files 446 445 -1
Lines 25290 25259 -31
Branches 11657 11646 -11
==========================================
- Hits 12488 12468 -20
+ Misses 4515 4511 -4
+ Partials 8287 8280 -7 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
📊 Physics performance monitoring for 8298167Summary VertexingVertexing vs. muIVF seededIVF truth_smearedIVF truth_estimatedIVF orthogonalAMVF seededAMVF truth_smearedAMVF truth_estimatedAMVF orthogonalSeedingSeeding seededSeeding truth_estimatedSeeding orthogonalCKFCKF seededCKF truth_smearedCKF truth_estimatedCKF orthogonalAmbiguity resolutionseededTruth tracking (Kalman Filter)Truth trackingTruth tracking (GSF)Truth tracking |
|
It's green. Let's quickly merge it before it breaks again @andiwand 😅 |
andiwand
approved these changes
Jun 17, 2023
|
Need to debug the compiler segfault on Monday. |
paulgessinger
pushed a commit
to paulgessinger/acts
that referenced
this pull request
Jul 24, 2023
…roject#2086) Our full chain pulls are in a bad state. Looks like the reconstruction and simulation energy loss did not match up. This PR switches the Fatras interactions on which should bring our pulls back to standard normal distribution. Fixes - acts-project#1643 Blocked by - acts-project#2157 - acts-project#2239 - acts-project#2295 - acts-project#2293 - acts-project#2294
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overall the goal is to not fail a job when an FPE occurs, but to mask that FPE type in the signal handler, take a stack trace, resume execution. The sequencer can then demask the type again for the next algorithm. Overall I implemented the resuming based on discussion with @stephenswat and only for x86_64 for now. It keeps stack traces, accumulates them across algorithms / events / threads, deduplicates stack traces, and can print a summary at the end, looking something like this:
Currently, this doesn't fail the job, and the plan is to implement a masking mechanism based on the top level stack frame source file and line, as well as summation by algorithm / reader / writer, rather than just one global one.