Sentry events do not file github issues any more

[knz]

Describe the problem

Sentry is not filing github issues for new events since June approximately.

Yet the sentry-zapier interface seems configured properly: a manual check at https://sentry.io/settings/cockroach-labs/projects/cockroachdb/plugins/webhooks/ will correctly cause a github issue.

It's possible that the webhook API is not used by sentry any more?

To Reproduce

• search sentry for issues with "firstseen" less than 1 month: observe that there are many recent issues; none of them have an accompanying github issue
• search github for issues with label O-sentry. No issue was auto-filed since June.

Expected behavior

New events in Sentry should file github issues.

[knz]

cc @andy-kimball I think this needs to be escalated to SIG - it impairs our ability to improve product stability.

[kenliu]

possibly related to #40414

[bobvawter]

The sentry configuration is set to trigger alerts only if the incoming environment value is set to the exact value of release. It looks like we're currently sending the git tag as the environment value, so the integration alert isn't actually firing for any new code. I've temporarily set the alert to match on "issue first reported" in "all environments" to verify that the plumbing is still working. Will dig into crdb code to see why we're not getting the expected string value.

#40514 just came in from development, so the webhook wiring is still ok.

[bobvawter]

I've tweaked the alert configuration to include all environments that start with "v", to match the released binaries.

[knz]

Do we have a way to get confidence your new setting fixed the problem entirely?

[kenliu]

@bobvawter and I discussed, this is working again