Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Production - [Alerting] Build Analysis: Message Abandoned Alert #1808

Closed
1 of 3 tasks
dotnet-eng-status bot opened this issue Jan 12, 2024 · 27 comments
Closed
1 of 3 tasks

Production - [Alerting] Build Analysis: Message Abandoned Alert #1808

dotnet-eng-status bot opened this issue Jan 12, 2024 · 27 comments
Assignees
@AlitzelMendez
Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)

Comments

@dotnet-eng-status
Copy link

dotnet-eng-status bot commented Jan 12, 2024 •
edited by AlitzelMendez
Loading

💔 Metric state changed to alerting

Description and instructions for this alert

The Build Analysis Service has abandoned a build.

  • count 1

Metric Graph

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-d67c04dd4b26487db22334164bb03b63

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

No release note needed
@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Critical Grafana Alert Issues opened by Grafana Ops - First Responder Production Tied to the Production environment (as opposed to Staging) labels Jan 12, 2024
@andriipatsula
Copy link
Member

andriipatsula commented Jan 12, 2024 •
edited
Loading

BuildID: 521880

Unhandled exception System.ArgumentException: An item with the same key has already been added. Key: 62a9e233-9779-42ce-9765-3972a545148f
...
Message body: {"eventType":"knownissue.validate","organization":"dotnet","repository":"runtime","issueId":96798,"repositoryWithOwner":"dotnet/runtime"}
...
Abandoning message because DeliveryCount of 10 >= MaxRetries 10

The issue is not widespread. I am dealing with other FR issues and will try to investigate more later today.

Per wiki:

If it's a onetime hit and it's coming from a pull request, you can close the issue.

-- it's coming from the dotnet/runtime#96798 issue

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 13, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Description and instructions for this alert

The Build Analysis Service has abandoned a build.

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 14, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

The Build Analysis Service has abandoned a build.

  • count 2

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Jan 16, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Description and instructions for this alert

The Build Analysis Service has abandoned a build.

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added the Inactive Alert Issues from Grafana alerts that are now "OK" label Jan 16, 2024
@AlitzelMendez AlitzelMendez self-assigned this Jan 16, 2024
@AlitzelMendez
Copy link
Member

Hi @andriipatsula,
I am taking a look at this, I think I know from where the issue is coming :)

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 17, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

The Build Analysis Service has abandoned a build.

  • count 5

Metric Graph

Go to rule

@AlitzelMendez
Copy link
Member

PR: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/36575

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 19, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Inactive Alert Issues from Grafana alerts that are now "OK" label Jan 20, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 3

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 20, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 1

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 31, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Feb 5, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 5

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Feb 6, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Feb 6, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 5

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Feb 8, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Inactive Alert Issues from Grafana alerts that are now "OK" label Feb 9, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 5

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Active Alert Issues from Grafana alerts that are now active labels Feb 9, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Feb 10, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

  • count 1

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Active Alert Issues from Grafana alerts that are now active labels Feb 10, 2024
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

The Build Analysis Service has abandoned a build.

Description and instructions for this alert

Metric Graph

Go to rule

@dotnet-eng-status dotnet-eng-status bot added the Inactive Alert Issues from Grafana alerts that are now "OK" label Feb 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlitzelMendez AlitzelMendez

Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AlitzelMendez @andriipatsula