-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libssh2 issues with Ruby profilers -- would a workaround PR be acceptable? #959
Comments
Well this is annoying. TIL there's a specific signal for profiling. Ruby profilers aren't the only ones doing this though, right? This feels like a libgit2 thing rather than a rugged thing as it wouldn't be working around an issue that happens in Ruby. |
I think so, too. I'm not entirely sure what that API would look like - presumably a runtime opt of some sort - but it makes sense to do this in libgit2. |
Right! In libssh2/libssh2#955 I list a few other profilers that use this (dart, Java, etc) and I know of more (Google's profiler for Python -- for instance). I suggested the workaround because while definitely being a bit ugly, it's not that hard to implement -- e.g. I'd be happy to open a PR with an actual implementation and a testcase showing how it fixes things for stackprof/the Datadog profiler -- let me know if there's interest :) |
Edit: To clarify, I don't mind taking a stab at opening a PR to do this in libgit2 directly either :) |
Messing with signals should be the app's responsibility but as a bug work-around, maybe libgit2 can get an option to compile with sigmasks in the libssh2 backend so it's limited to that instead of any network function. Ed would really be the one to have the final say, but I guess a define you can set wouldn't be the worst. |
As long as there's a way to query "is this version of rugged safe or not" I'll take it, as I'd be able to tweak the Datadog profiler to use signals on the safe version 😄 For other profilers, one part of this issue is that it's not very obvious what's happening -- e.g. if you have rugged and then you added a profiler it's immediately obvious (it's the profiler/something that the profiler triggered!); but if you had a profiler and then you added rugged/libgit2 then it may be really unclear why network requests are failing. |
Yeah, I wonder if the API should be "give me a callback that runs at the start and end of network functions" and the usage here is signal handling. Or if we should just give you a signal handler. I lead toward the first one. |
OK, revisiting this thought -- should libgit2 really have a callback for "network function about to start" and "network function about to end"... ? What's the granularity? Is it Is it around every |
Yeah, I'm not sure it's worth expanding the libgit2 API in a permanent way for this versus having a more temporary solution (with the intention of removing the temporary solution at some point). |
Hello there :)
I work on Datadog's Ruby profiler and I arrived here while investigating a customer issue where they were using the rugged gem and it breaks in combination with our profiler due to a libssh2 bug.
The TL;DR is that libssh2 doesn't yet correctly handle the interruption of system calls caused by profilers that use unix signals, and thus having such a profiler running breaks network calls using rugged.
This issue is not specific to the Datadog Ruby profiler 😭; the also great stackprof profiler gem is unfortunately affected by this issue:
gets this output:
...but removing
StackProf
makes the clone work fine -- it's not actually an authentication error, it's the system call interruption at work.Ok so why am I double-reporting this issue when I've already reported it to the libssh2 developers as well?
Since it's common to use a system libssh2 with rugged, even if the fix to libssh2 was released today, it'll take months/years to arrive on Linux distros.
Furthermore, it's hard to detect from Ruby code if rugged is linked with a broken libssh2, because while rugged provides a
Rugged.libgit2_version
, there's no corresponding API to probe the libssh2 version (at least I didn't find one in libgit2 directly or rugged).Currently, when rugged is detected, the Datadog profiler needs to fall back to an alternative code path that yields lower-quality data.
I would love for this to not be the case!
Thus my question is: Would you consider accepting a pull request modifying rugged calls that can trigger network operations (I think it's only clone/fetch/pull/push/submodule stuff?) with a method that temporarily disables signal handling (for
SIGPROF
andSIGALARM
) during that call?Something similar to:
This would make rugged work great under stackprof / the Datadog profiler and as a bonus we could detect the fixed version and avoid the fall back to an alternative code path that yields lower-quality data that we currently have.
Thoughts? I'm shamelessly tagging @tenderlove here since you're also a maintainer of stackprof ;)
The text was updated successfully, but these errors were encountered: