Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Improved FPE monitoring #2157

Merged
merged 98 commits into from
Jun 19, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
f7ce657
basic improvement in!
paulgessinger May 25, 2023
0bebb8b
fix boost backtrace setup
paulgessinger May 25, 2023
31dfd0b
fpemonitor bundles up stack traces!
paulgessinger May 25, 2023
2808e54
add count merging
paulgessinger May 26, 2023
5b4e304
stack trace accumulation
paulgessinger May 26, 2023
e464475
refactor, add opaque wrapper around boost stack trace
paulgessinger May 26, 2023
fdbc71d
stack trace deduplication based on top frame
paulgessinger May 26, 2023
3ec89ae
python level test harness
paulgessinger May 26, 2023
1d46873
prototype for sequencer level monitoring, no failure or masking yet
paulgessinger May 26, 2023
2cbb82a
add python fpe test alg
paulgessinger May 26, 2023
9e61b9d
Update Tests/UnitTests/Core/Utilities/CMakeLists.txt
paulgessinger May 31, 2023
a666e09
Update Core/include/Acts/Utilities/FpeMonitor.hpp
paulgessinger May 31, 2023
be85c04
Merge remote-tracking branch 'origin/main' into feat/fpe-imprv
paulgessinger Jun 5, 2023
d210bf4
compile fixes
paulgessinger Jun 5, 2023
26e4c84
Merge branch 'feat/fpe-imprv' of github.com:paulgessinger/acts into f…
paulgessinger Jun 5, 2023
580a336
fpe monitor and stack trace collection uses preallocated buffer
paulgessinger Jun 5, 2023
2bbd024
progress
paulgessinger Jun 6, 2023
99972be
masking mechanism!
paulgessinger Jun 6, 2023
f39ceab
format
paulgessinger Jun 6, 2023
9e7b636
move to plugin
paulgessinger Jun 7, 2023
95ac652
adding the plugin
paulgessinger Jun 7, 2023
584a99f
compile fixes
paulgessinger Jun 7, 2023
986fcec
x86_64
paulgessinger Jun 7, 2023
5be424f
switch to using manual buffer
paulgessinger Jun 7, 2023
d105aeb
format
paulgessinger Jun 7, 2023
1c0cf4e
fixes for non-linux / x86_64
paulgessinger Jun 7, 2023
13eae7f
use forward decl for boost stack trace
paulgessinger Jun 8, 2023
921517f
format fix
paulgessinger Jun 8, 2023
a1ca744
try building with deb info on ubuntu
paulgessinger Jun 8, 2023
d2bc4c1
add FPEMON plugin cmake option
paulgessinger Jun 8, 2023
b019e07
restrict symbol use to where we link
paulgessinger Jun 8, 2023
b8ec04e
link fix
paulgessinger Jun 8, 2023
3d7c524
determine masks automatically
paulgessinger Jun 8, 2023
256227f
mask from yaml, context, format
paulgessinger Jun 8, 2023
99f1c27
gitlab ci relwithdebinfo
paulgessinger Jun 8, 2023
214b504
check mask against any frame from the top, report which location
paulgessinger Jun 8, 2023
fedaf5d
remove offset init
paulgessinger Jun 8, 2023
bb0f48d
fix: FPE in FATRAS hit direction
paulgessinger Jun 9, 2023
b25f92b
improve scoping and suppression
paulgessinger Jun 9, 2023
03fcd5e
configurable stack trace length
paulgessinger Jun 9, 2023
1f1a370
ability to init fpe masks from list of tuples
paulgessinger Jun 9, 2023
e3cb379
stack length config to python
paulgessinger Jun 9, 2023
a3c5ef4
remove manual FPE setup from full chain (sequencer does this now)
paulgessinger Jun 9, 2023
e2d9c3d
format
paulgessinger Jun 9, 2023
47061b3
again format
paulgessinger Jun 9, 2023
eaa953c
test fixes
paulgessinger Jun 9, 2023
d0d590e
fix possible SEGFAULT
paulgessinger Jun 9, 2023
80960bd
mask 1 FPE in FATRAS and one in GSF
paulgessinger Jun 9, 2023
24e9d04
one more FPE ignore due to different CI inlining
paulgessinger Jun 9, 2023
88f0afd
ci: Don't package .o files
paulgessinger Jun 9, 2023
7b3fe42
fix tar command
paulgessinger Jun 9, 2023
bf9dee9
Merge remote-tracking branch 'origin/main' into feat/fpe-imprv
paulgessinger Jun 12, 2023
fa0f35b
fix / suppress two more FPEs
paulgessinger Jun 12, 2023
b9fc858
feat: Geant4 Atlas physics list (#2188)
andiwand Jun 9, 2023
dc02fcd
fix mask, format
paulgessinger Jun 12, 2023
71729f5
and another ignore update
paulgessinger Jun 12, 2023
cb01839
and another mask
paulgessinger Jun 12, 2023
baf7ee3
another mask
paulgessinger Jun 13, 2023
565f6c6
Merge remote-tracking branch 'origin/main' into feat/fpe-imprv
paulgessinger Jun 13, 2023
d1e5fc0
another mask
paulgessinger Jun 13, 2023
c566a86
reduce gitlab ccache size
paulgessinger Jun 13, 2023
e9aef59
another mask
paulgessinger Jun 13, 2023
616bd48
move to yml
paulgessinger Jun 14, 2023
334a6d4
format fix
paulgessinger Jun 12, 2023
9403001
physmon FPE masks in yml (central), improve sequencer failure behavio…
paulgessinger Jun 14, 2023
154089f
only print first occurence of FPE, fix tests
paulgessinger Jun 14, 2023
da422ff
bump revision
paulgessinger Jun 14, 2023
9f32867
when end??
paulgessinger Jun 14, 2023
354394f
use central compare function
paulgessinger Jun 15, 2023
983a7ca
back to ccache r1
paulgessinger Jun 15, 2023
0c81ed2
switch stacktrace pimpl to shared_ptr
paulgessinger Jun 15, 2023
d840309
factorize fpe reporting into separate method in Sequencer
paulgessinger Jun 15, 2023
6ad6c4e
remove .h imports
paulgessinger Jun 15, 2023
498a02b
remove FpeMask operator==, readd in python
paulgessinger Jun 15, 2023
6616483
revert ResPlotTool changes
paulgessinger Jun 15, 2023
e36f2e5
remove unused bits
paulgessinger Jun 15, 2023
dbc5680
change copyright year
paulgessinger Jun 15, 2023
f23ed53
remove unneeded include
paulgessinger Jun 15, 2023
dc63d88
format fixes
paulgessinger Jun 15, 2023
d2c9a19
Merge remote-tracking branch 'origin/main' into feat/fpe-imprv
paulgessinger Jun 15, 2023
7e6ffe6
remove merge markers
paulgessinger Jun 15, 2023
175fb48
cleanup
paulgessinger Jun 15, 2023
093704f
switch back to Release on GH, add physmon job on GL
paulgessinger Jun 15, 2023
961baee
fix gitlab
paulgessinger Jun 15, 2023
0c3adaa
attempt to suppress fpe tracking in github where we don't have debug
paulgessinger Jun 15, 2023
f531711
||true
paulgessinger Jun 15, 2023
d2e9a74
add physmon artifact
paulgessinger Jun 15, 2023
a696eee
bump ccache size to 2G, always upload physmon output
paulgessinger Jun 15, 2023
ee38214
physmon path
paulgessinger Jun 15, 2023
ca89f33
another mask
paulgessinger Jun 15, 2023
5932c22
skip fpe tests if disabled
paulgessinger Jun 15, 2023
58944ba
re-fix gsf tests
paulgessinger Jun 15, 2023
cb7f79d
move envvar based override into C++
paulgessinger Jun 16, 2023
3b3ba3d
Merge branch 'main' into feat/fpe-imprv
kodiakhq[bot] Jun 17, 2023
3c33ea7
reduce number of build threads
andiwand Jun 17, 2023
a624841
Merge branch 'main' into feat/fpe-imprv
andiwand Jun 18, 2023
d76dcce
compilation fix
paulgessinger Jun 19, 2023
8298167
Merge branch 'main' into feat/fpe-imprv
paulgessinger Jun 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 69 additions & 12 deletions Core/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -34,25 +34,82 @@ find_program(addr2line_EXECUTABLE addr2line)
if(APPLE)
list(APPEND _fpe_options -D_GNU_SOURCE)
else()

if(dl_LIBRARY)
target_link_libraries(ActsCore PUBLIC ${dl_LIBRARY})
if(addr2line_EXECUTABLE)
list(APPEND _fpe_options -DBOOST_STACKTRACE_USE_ADDR2LINE)
list(APPEND _fpe_options -DBOOST_STACKTRACE_ADDR2LINE_LOCATION=${addr2line_EXECUTABLE})
elseif(Backtrace_FOUND)
list(APPEND _fpe_options -DBOOST_STACKTRACE_USE_BACKTRACE)
target_link_libraries(ActsCore PUBLIC ${Backtrace_LIBRARY})
else()
list(APPEND _fpe_options -BOOST_STACKTRACE_USE_NOOP)

set(_backtrace_setup_complete FALSE)

if(Backtrace_FOUND)

# check if we need to link against bracktrace or not

file(WRITE
"${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/backtrace.cpp"
"#include <boost/stacktrace.hpp>\n"
"#include <iostream>\n"
"int main() { std::cout << boost::stacktrace::stacktrace(); }\n" )

message(CHECK_START "Does backtrace work without linker flag")
try_compile(_backtrace_nolink "${CMAKE_BINARY_DIR}"
"${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/backtrace.cpp"
LINK_LIBRARIES ${dl_LIBRARY}
COMPILE_DEFINITIONS -DBOOST_STACKTRACE_USE_BACKTRACE
OUTPUT_VARIABLE __OUTPUT)

if(_backtrace_nolink)
message(CHECK_PASS "yes")
set(_backtrace_setup_complete TRUE)
else()
message(CHECK_FAIL "no")

file(WRITE
"${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/backtrace.cpp"
"#include <boost/stacktrace.hpp>\n"
"#include <iostream>\n"
"int main() { std::cout << boost::stacktrace::stacktrace(); }\n" )

message(CHECK_START "Does backtrace work with linker flag")
try_compile(_backtrace_link "${CMAKE_BINARY_DIR}"
"${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/backtrace.cpp"
LINK_LIBRARIES backtrace ${dl_LIBRARY}
COMPILE_DEFINITIONS -DBOOST_STACKTRACE_USE_BACKTRACE
OUTPUT_VARIABLE __OUTPUT)

if(_backtrace_link)
message(CHECK_PASS "yes")
list(APPEND _fpe_options -DBOOST_STACKTRACE_USE_BACKTRACE)
target_link_libraries(ActsCore PUBLIC backtrace)

set(_backtrace_setup_complete TRUE)
else()
message(CHECK_FAIL "no")
endif()

endif()
endif()
else()

if(NOT _backtrace_setup_complete)
message(CHECK_START "Is addr2line available")
if(addr2line_EXECUTABLE)
list(APPEND _fpe_options -DBOOST_STACKTRACE_USE_ADDR2LINE)
list(APPEND _fpe_options -DBOOST_STACKTRACE_ADDR2LINE_LOCATION=${addr2line_EXECUTABLE})
message(CHECK_PASS "yes")

set(_backtrace_setup_complete TRUE)
else()
message(CHECK_FAIL "no")
endif()
endif()

if(NOT _backtrace_setup_complete)
message(STATUS "Unable to set up stacktrace setup: use noop")
list(APPEND _fpe_options -BOOST_STACKTRACE_USE_NOOP)
endif()
endif()
endif()

message(STATUS ${_fpe_options})

set_source_files_properties(src/Utilities/FpeMonitor.cpp PROPERTIES COMPILE_OPTIONS "${_fpe_options}")
set_source_files_properties(src/Utilities/StackTrace.cpp PROPERTIES COMPILE_OPTIONS "${_fpe_options}")

if(ACTS_PARAMETER_DEFINITIONS_HEADER)
target_compile_definitions(
Expand Down
73 changes: 71 additions & 2 deletions Core/include/Acts/Utilities/FpeMonitor.hpp
andiwand marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,89 @@
#pragma once

#include "Acts/Utilities/Helpers.hpp"
#include "Acts/Utilities/StackTrace.hpp"

#include <atomic>
#include <csignal>
#include <memory>
#include <mutex>
#include <stack>

namespace Acts {

enum class FpeType : uint32_t {
INTDIV = FPE_INTDIV,
INTOVF = FPE_INTOVF,
FLTDIV = FPE_FLTDIV,
FLTOVF = FPE_FLTOVF,
FLTUND = FPE_FLTUND,
FLTRES = FPE_FLTRES,
FLTINV = FPE_FLTINV,
FLTSUB = FPE_FLTSUB,
};

std::ostream &operator<<(std::ostream &os, FpeType type);

class FpeMonitor {
public:
struct Result {
struct FpeInfo {
std::size_t count;
FpeType type;
StackTrace st;
};

Result merge(const Result &with) const;
bool encountered(FpeType type) const;
unsigned int count(FpeType type) const;

const std::vector<FpeInfo> &stackTraces() const;
unsigned int numStackTraces() const;

void deduplicate();

void summary(std::ostream &os) const;

Result();

private:
std::vector<FpeInfo> m_stracktraces;
std::array<unsigned int, 32> m_counts{};

friend FpeMonitor;
};

FpeMonitor();
explicit FpeMonitor(int excepts);
~FpeMonitor();

static void enable(int excepts);
static void disable(int excepts);
const Result &result() const;
Result &result();

void rearm() const;

std::size_t stackLimit() const { return m_stackLimit; }
void setStackLimit(std::size_t limit) { m_stackLimit = limit; }

private:
void enable();
void disable();

static void ensureSignalHandlerInstalled();
static void signalHandler(int signal, siginfo_t *si, void *ctx);

struct GlobalState {
std::atomic_bool isSignalHandlerInstalled{false};
std::mutex mutex{};
};

static std::stack<FpeMonitor *> &stack();
static GlobalState &globalState();

int m_excepts;
paulgessinger marked this conversation as resolved.
Show resolved Hide resolved
std::size_t m_stackLimit = 1000;

Result m_result;
};

} // namespace Acts
36 changes: 36 additions & 0 deletions Core/include/Acts/Utilities/StackTrace.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// This file is part of the Acts project.
//
// Copyright (C) 2022 CERN for the benefit of the Acts project
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.

#pragma once

#include <cstddef>
#include <memory>
#include <string>

namespace Acts {
struct StackTrace {
StackTrace(std::size_t skip, std::size_t maxDepth);
StackTrace(StackTrace &&other);
StackTrace(const StackTrace &other);
StackTrace &operator=(StackTrace &&other);
StackTrace &operator=(const StackTrace &other);

std::pair<std::string, std::size_t> topSourceLocation() const;

~StackTrace();

friend std::ostream &operator<<(std::ostream &os, const StackTrace &st);

friend bool operator==(const StackTrace &lhs, const StackTrace &rhs);

private:
struct Impl;
std::unique_ptr<Impl> m_impl;
};

} // namespace Acts
1 change: 1 addition & 0 deletions Core/src/Utilities/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ target_sources(
Logger.cpp
SpacePointUtility.cpp
FpeMonitor.cpp
StackTrace.cpp
)
Loading