Don't have transactions go through post_process

[JoshFerge]

given that for post_process we use rabbitmq/celery, and its quite high throughput, we should consider that for transactions, this may not be necessary.

• We don't need anything in clickhouse for transactions processing
• the only thing we do for transactions in post_process is record sampling and send a receive signal.

  sentry/src/sentry/ingest/transaction_clusterer/datasource/redis.py

  Line 103 in 5d39254

  def record_transaction_name(project: Project, event_data: Mapping[str, Any], **kwargs: Any) -> None:

We can cut the number of rabbitmq tasks we send by ~40% if we simply don't use post_process for transactions.

We can also cut out the usage of rc-processing for transactions for this.

We'd move this transaction sampling over to save_event_transaction.

[JoshFerge]

Roll out strategy:

Part 1:

• Merge ref(transactions): add functionality to sample transactions in save_event_transaction #81077
• in S4S, turn on flag ("transactions.do_post_process_in_save") to 1.0. confirm looks good
• in DE, turn on flag to 1%, 10%, 25%, 50%, 100%
• in US, turn on flag 1%, 10%, 25%, 50%, 100%

what to watch for during rollout:

• transaction clusterer metrics (should stay the same)
• save_event_transaction on queue status
• s4s sentry perf page for save_event_transaction
• should also ensure traces for save_event_transaction are reporting the new spans
• no new Sentry errors

Part 2:

Stop producing post_process for transactions

• see that post_process enqueues decline
• see logging messages for "not found" decline

[mwarkentin]

send a receive signal.

@JoshFerge during SRE office hours it was raised that sending signals is not an asynchronous event - basically all of the existing (and future) receivers of those signals are still processed synchronously. So we need to understand what is being triggered by that signal and understand how that impacts things. Are we going to be doing those signal handlers inside of save_event_transaction now too?

[JoshFerge]

send a receive signal.

@JoshFerge during SRE office hours it was raised that sending signals is not an asynchronous event - basically all of the existing (and future) receivers of those signals are still processed synchronously. So we need to understand what is being triggered by that signal and understand how that impacts things. Are we going to be doing those signal handlers inside of save_event_transaction now too?

i've changed the implementation to not use signals and call what the signal did previously directly. we also confirmed that these calls are very fast https://sentry.sentry.io/traces/?project=1&query=span.op%3Atasks.post_process_group.transaction_processed_signal&statsPeriod=24h&visualize=%7B%22chartType%22%3A1%2C%22yAxes%22%3A%5B%22avg%28span.duration%29%22%5D%7D

[JoshFerge]

TODO as a follow up, we can remove this update to rc-processing in save_event if its a transaction https://github.com/getsentry/sentry/blob/master/src/sentry/ingest/consumer/processors.py#L81-L88

there may be other places where we're doing unecessary work as well.