[Cancelled] Detect Bad Releases

[Dhrumil-Sentry]

Developers often ship their code by releasing it through various environments- local, dev, staging, canary, and finally into production. After each release, a developer actively investigates the health of the release in which their code changes were shipped.

We see several users actively come to our Release Details page to investigate the health of their release:

One common piece of feedback we received was about enabling automated release monitoring. Sentry already knows what’s happening with a release and the logical next step is to help developers easily detect bad releases

We see the following gaps in this workflow today:

• Sentry does not actively monitor releases
  • You cannot specify the thresholds to detect and notify for “bad” releases
• Lack of APIs to build automation to enable continuous deployments

We want to build a workflow that bridges these gaps and enables the following:

• Developers can be informed in the correct channels when a bad release is detected in their staging or production environments
• Developers can leverage APIs to poll release health and automatically rollback bad releases

Sentry detects your bad releases

Here’s what we’re planning to build:

1. For a given project and/or environment, developers can specify the thresholds for categorizing a release as “bad”

   1. Thresholds are based on the release health metrics that Sentry already shows in our release details page:
      1. Crash free session and user rates
      2. Failure rates
      3. Error counts
      4. Count of new/regressed/unhandled issues seen in the release
   2. A threshold monitoring window that determines the duration for which Sentry should monitor each release actively

2. Developers can specify how they’d like to be notified for bad releases- Slack, Pagerduty etc

   1. For e.g. you can be notified on Pagerduty for bad releases in production vs Slack for bad releases in your staging environments

3. Once a new release is made for your project, Sentry will actively track your release

   1. Sentry will track if any of the specified thresholds were breached during the active release monitoring window
   2. If a threshold is breached in the release- Sentry will send the notification or fire a webhook based on your preference

4. Developers can also poll our new APIs to fetch this information in order to use this for continuous deployments and canary release management

We want your feedback

We’d love to understand your challenges with release monitoring and your thoughts about how Sentry can simplify your workflow

1. What does your release process look like?
   1. Do you merge your code changes into different environments and search for issues?
2. How do you detect your release is bad?
   1. Do you look for new issues or do you care about your error counts spiking?
3. How can Sentry improve release monitoring for you?

[Dhrumil-Sentry]

Update

We’re rolling out an alpha version of this feature to help you determine whether your release is unhealthy or not!

If you’d like to be a part of this cohort please drop us a note at feedback-releases@sentry.io or comment below 👇

Here’s what you need to do to get started:

Go to Releases→Thresholds and add thresholds for your desired Sentry project and environment. Currently, we only support error counts, but we’ll be adding support for other threshold types like new issues soon

Once you set these thresholds - you can use our new API in your CI to directly poll the health of this release:

Endpoint: api/0/organizations/{org_slug}/release-threshold-statuses/

Required Params
NOTE: start/end define the window to query for release health.
:param start: ISO-8601 formatted datetime of the beginning of the specified date range
:param end: ISO-8601 formatted datetime of the end of the specified date range

Optional Params

:param environment: filter given a list of environments
:param project: filter given a list of project slugs
:param release: filter given a list of release versions

Sample Response:

{
    {proj_slug}-{env}-{release_version}: [
        {
					"id": "7",
					"threshold_type": "total_error_count",
					"trigger_type": "over",
					"value": 500,
					"window_in_seconds": 300,
					"project": {
						"id": "1",
						"slug": {proj_slug},
						"platform": "python",
						...
					},
					"environment": {
						"id": "1234",
						"name": {env}
					},
					"date_added": "2023-11-07T22:34:10.329353Z",
					"key": {proj_slug}-{env}-{release_version},
					"start": "2023-11-21T18:04:42.304135Z",
					"end": "2023-11-21T18:09:42.304135Z",
					"release": {release_version},
					"project_slug": {proj_slug},
					"project_id": 1,
					"is_healthy": true
        },
        {...},
        {...}
    ],
    {proj}-{env}-{release}: [...],
}

The is_healthy response field will tell you if the release has breached the thresholds you set in the Sentry app.

We are actively building this feature and here’s what coming next:

• More threshold types based on new/regressed issue counts and crash-free session/user rates
• Notifications for bad releases

Don’t think this meets your requirements? Join the discussion 👇 or drop us a note at feedback-releases@sentry.io so that we can help you detect bad releases easily

[nhsiehgit]

Update 2

Hey all,

Thank you for your patience as we continue to iterate on our Detect Bad Releases feature.

We’ve heard your feedback and have taken some time to reconsider what a solution to this might look like.

Notably, we’ve recognized that monitoring release metrics very closely mirrors our existing Alert Rules feature, and rather than building out a separate identical vertical, we want to enable the existing Alert Rules to conditionally monitor releases for a period of time.

This next iteration allows you to select a monitor type during Alert Rule creation (either continuously monitor, or conditionally monitor given an activation trigger)

Conditional alert rules will be available for all Metric Alerts such as error counts or crash free rates.

So for example, if you wanted to monitor errors per release, you could set up an Alert Rule with a conditional monitor type triggered via New Release, and set it to monitor each new release for an hour after it’s creation. After creating a new release tag, Sentry will watch for error events tagged with the release version, and if errors rise past your set threshold, we will trigger the set actions - such as sending you a notification in your slack channel.

Notification actions will be treated exactly the same as they do today for existing Metric Alert Rules and you’ll be able to direct these alerts through your existing notification channels such as Email, Slack, etc.

We’re still actively building out this feature and want to make sure that this is useful and valuable to you!

If you’d like access to this early feature and would like to help us shape improvements, please drop us a note at feedback-releases@sentry.io .

[houserx-jmcc]

We'd be interested in alpha testing ✋🏻

[Dhrumil-Sentry]

@houserx-jmcc Great to hear that! We'll enable this for your organization and I will comment here once it's ready

[philwolstenholme]

We'd be interested in helping test this in the co-op-food-ecom org

[rachrwang]

Hi everyone - thank you for your interest and feedback on Detect Bad Releases. After further evaluation, we’ve decided not to move forward with this specific implementation. Our priorities have shifted to focus on a more comprehensive approach with "group by" in metric alerts, which will eventually cover this use case. We really appreciate all the feedback shared during this beta period.