[azure-eventhub] Update input v1 status on start, failure, and stop #41469

zmoog · 2024-10-28T22:23:38Z

Proposed commit message

Update the Elastic Agent status by calling inputContext.UpdateStatus(status.Failed, err.Error()) during the main input lifecycle phases (set up and run). If any of the setup, startup, and run steps fail, the input reports the fatal issue before shutting down.

Without reporting the fatal error, the input logs the error and stops, but users continue to see it as "healthy" in Fleet, causing confusion and making troubleshooting much harder.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Test with wrong event hub credentials (for example, use a connection string from a different namespace)
Test with wrong storage account credentials (for example, a key from a different storage account or a random string)

How to test this PR locally

Build a custom agent
Install the Azure Logs integration
- set an invalid connection string to test setup() failures
- set an invalid storage account key to test run() failures

Related issues

Closes [azure-eventhub] Agent Reports Healthy Status After Input Error Post-Startup integrations#9659

Screenshots

Fatal error during setup() caused by an invalid event hub connection string:

Fatal error during run() caused by an invalid storage account key:

mergify · 2024-10-28T22:24:16Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-10-28T22:24:16Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

zmoog · 2024-10-28T22:32:55Z

x-pack/filebeat/input/azureeventhub/v1_input.go

@@ -105,9 +109,11 @@ func (in *eventHubInputV1) Run(
 	err = in.run(ctx)
 	if err != nil {
 		in.log.Errorw("error running input", "error", err)
+		inputContext.UpdateStatus(status.Failed, err.Error())


Hey @faec, I missed the opportunity to report the input status to the Elastic Agent when I updated the input to the Input API v2.

I am trying to address scenarios (like elastic/integrations#9659) where the input starts, and after a while, it encounters a fatal error, and the SDK worker shuts down.

What are your recommendations for reporting input status to the agent? Which input should I use as a reference?

🙇

When I update the input to the Input API v2, I missed the opportunity to report the input status back to the Elastic Agent.

Now also cover the following phases: - pipeline creation - sanitizers creation - input setup

elasticmachine · 2024-11-06T22:24:17Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

zmoog · 2024-11-06T22:44:45Z

Tested on a custom build of the Elastic Agent. We set an invalid event hub connection string and an invalid storage account key to trigger fatal errors during the input setup and run phases.

…41469) Update the Elastic Agent status by calling `inputContext.UpdateStatus(status.Failed, err.Error())` during the main input lifecycle phases (set up and run). If any setup, startup, and run steps fail, the input reports the fatal issue before shutting down. Without reporting the fatal error, the input logs the error and stops, but users continue to see it as "healthy" in Fleet, causing confusion and making troubleshooting much harder. (cherry picked from commit 882c854)

…41469) (#41547) Update the Elastic Agent status by calling `inputContext.UpdateStatus(status.Failed, err.Error())` during the main input lifecycle phases (set up and run). If any setup, startup, and run steps fail, the input reports the fatal issue before shutting down. Without reporting the fatal error, the input logs the error and stops, but users continue to see it as "healthy" in Fleet, causing confusion and making troubleshooting much harder. (cherry picked from commit 882c854) Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

…41469) (#41546) Update the Elastic Agent status by calling `inputContext.UpdateStatus(status.Failed, err.Error())` during the main input lifecycle phases (set up and run). If any setup, startup, and run steps fail, the input reports the fatal issue before shutting down. Without reporting the fatal error, the input logs the error and stops, but users continue to see it as "healthy" in Fleet, causing confusion and making troubleshooting much harder. (cherry picked from commit 882c854) Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

…art, failure, and stop (#41545) * [azure-eventhub] Update input v1 status on start, failure, and stop (#41469) Update the Elastic Agent status by calling `inputContext.UpdateStatus(status.Failed, err.Error())` during the main input lifecycle phases (set up and run). If any setup, startup, and run steps fail, the input reports the fatal issue before shutting down. Without reporting the fatal error, the input logs the error and stops, but users continue to see it as "healthy" in Fleet, causing confusion and making troubleshooting much harder. (cherry picked from commit 882c854) * Drop unintentional changelog entries --------- Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 28, 2024

mergify bot assigned zmoog Oct 28, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 28, 2024

zmoog commented Oct 28, 2024

View reviewed changes

zmoog added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 28, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 28, 2024

zmoog added needs_team Indicates that the issue/PR needs a Team:* label bugfix labels Oct 28, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 28, 2024

zmoog added needs_team Indicates that the issue/PR needs a Team:* label input:azure-eventhub labels Oct 28, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 28, 2024

zmoog force-pushed the zmoog/azure-eventhub-input-status branch from f112711 to 9412dd2 Compare November 6, 2024 10:29

zmoog added 2 commits November 6, 2024 23:23

Update input v1 status on start, failure, and stop

5212494

When I update the input to the Input API v2, I missed the opportunity to report the input status back to the Elastic Agent.

Update status in all input lifecycle phases

c631779

Now also cover the following phases: - pipeline creation - sanitizers creation - input setup

zmoog force-pushed the zmoog/azure-eventhub-input-status branch from 9412dd2 to c631779 Compare November 6, 2024 22:23

zmoog marked this pull request as ready for review November 6, 2024 22:24

zmoog requested a review from a team as a code owner November 6, 2024 22:24

kaiyan-sheng approved these changes Nov 6, 2024

View reviewed changes

Update CHANGELOG

3fa45d2

zmoog added backport-8.15 Automated backport to the 8.15 branch with mergify backport-8.16 Automated backport with mergify labels Nov 6, 2024

Kavindu-Dodan approved these changes Nov 6, 2024

View reviewed changes

zmoog enabled auto-merge (squash) November 6, 2024 23:44

zmoog merged commit 882c854 into main Nov 7, 2024
22 checks passed

zmoog deleted the zmoog/azure-eventhub-input-status branch November 7, 2024 00:36

mergify bot mentioned this pull request Nov 7, 2024

[8.15](backport #41469) [azure-eventhub] Update input v1 status on start, failure, and stop #41545

Merged

8 tasks

This was referenced Nov 7, 2024

[8.16](backport #41469) [azure-eventhub] Update input v1 status on start, failure, and stop #41546

Merged

[8.x](backport #41469) [azure-eventhub] Update input v1 status on start, failure, and stop #41547

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[azure-eventhub] Update input v1 status on start, failure, and stop #41469

[azure-eventhub] Update input v1 status on start, failure, and stop #41469

zmoog commented Oct 28, 2024 •

edited

Loading

mergify bot commented Oct 28, 2024

mergify bot commented Oct 28, 2024

zmoog Oct 28, 2024

elasticmachine commented Nov 6, 2024

zmoog commented Nov 6, 2024

[azure-eventhub] Update input v1 status on start, failure, and stop #41469

[azure-eventhub] Update input v1 status on start, failure, and stop #41469

Conversation

zmoog commented Oct 28, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

mergify bot commented Oct 28, 2024

mergify bot commented Oct 28, 2024

zmoog Oct 28, 2024

Choose a reason for hiding this comment

elasticmachine commented Nov 6, 2024

zmoog commented Nov 6, 2024

zmoog commented Oct 28, 2024 •

edited

Loading