Ability to set profiling sample interval

[md384]

I would like the ability to set the sample interval to something greater that 100 per second (probably 1000 per second) to get better visibility into performance bottlenecks (at least temporarily or in a test environment where throughput is much lower). Is that something that would be a possibility?

I see that the V8 CpuProfiler has a function for setting the interval (which presumably sentry already uses to set to 10ms given the default is 1000us).

[AbhiPrasad]

There are no plans to expose this at the current time but maybe we can think about it as an experimental feature in the integration?

cc @JonasBa

[md384]

I run sentry self-hosted. I imagine there could be some additional consequences to this is in the hosted version but can be mitigated more in self-hosted (i.e. additional load and storage requirements for more precise profiles).

[JonasBa]

@md384 You are correct. We set a sampling rate of ~10ms or 100Hz as the sampling interval in Sentry, mostly because we want our users to focus on the aggregate data that we collect instead of individual profiles.

From a product point of view, this is the behavior that we would like to keep, so if we do expose it, we need to be very careful and do it in a way that does not encourage non self hosted users to set it, else we risk a 10x factor on the data we ingest from profiles (which we currently have no guardrails for).

I would be curious to hear if you tried using the aggregate flamegraphs to solve your issue? The main reason is that after enough data is collected, the resolution should be even greater than what you would get from a single profile sampled at 1kH.

[md384]

I have only tried profiling in a test environment with very low traffic (~30-50 samples) and I didn't find the aggregate flamegraphs useful because of the low resolution (and samples), but I will try generating more data and see how they work out.

[md384]

I created an aggregate flamegraph with ~2,500 requests to the same page type (same template, different data) which you can see below alongside the flamegraph of (a single example) of the same page type.

You can pick out some of the larger function calls but lose a lot of the detail. For reference the function calls on the right hand side of the devtools flamegraph (t.a function call) are around 2.5-3ms and in our case are ones we could end up removing. These were not apparent in the aggregate flamegraph but were immediately suspicious in the devtools flamegraph. I spot checked a few individual flamegraphs in devtools and they all have a similar pattern (as expected) and aren't particularly close to the sentry aggregated flamegraph.

I am trying to optimize react SSR rendering here so there are many calls to fast function calls but the concerning part is the right hand side which is pretty much entirely missed in the aggregated flamegraph.