Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High io consumption after sudden filebeat stop #35893

Closed
Hitych opened this issue Jun 23, 2023 · 4 comments · Fixed by #39392
Closed

High io consumption after sudden filebeat stop #35893

Hitych opened this issue Jun 23, 2023 · 4 comments · Fixed by #39392
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@Hitych
Copy link

Hitych commented Jun 23, 2023 •
edited
Loading

Hi! I tried to ask on discuss.elastic.co but no answer.

The problem is very high io, after sudden termination of a filebeat. The reason is a checkpoint action on each log operation. It is because of log_invalid flag set to true, after failed initial log read operation. After abnormal termination of a filebeat, log may be in a inconsistent state and read of log like this can cause error Incomplete or corrupted log file in /usr/share/filebeat/data/registry/filebeat. Continue with last known complete and consistent state. Reason: invalid character '\\x00' looking for beginning of value
After that, filebeat clears log file, but still not trying to write, and just make checkpoint by checkpoint.

  1. Start filebeat
  2. Shutdown machine suddenly
  3. Start machine again
  4. Start filebeat
  5. Check the log for an errors
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 23, 2023
@emmanueltouzery
Copy link
Contributor

emmanueltouzery commented Feb 12, 2024 •
edited
Loading

We are seeing the same issue:
https://discuss.elastic.co/t/filebeat-causing-a-very-large-iowait-and-lagging-after-uncontrolled-reboot/351981

@Hitych
Copy link
Author

Hitych commented Mar 15, 2024

@elastic/obs-dc can anyone help here?

emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 3, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
911ed90 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
@emmanueltouzery emmanueltouzery mentioned this issue May 3, 2024
Merged
6 tasks
@belimawr belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label May 3, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 3, 2024
@belimawr
Copy link
Contributor

belimawr commented May 3, 2024

Hey folks, thanks for finding this bug and proposing a fix! Looking at the code I can see it indeed is a bug. Restarting Filebeat should bring it back into a consistent state. While not perfect, it is at least a workaround.

emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 19, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
0363887 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 19, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
55f74ed 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 19, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
36eda84 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 19, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
78d364c 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 20, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
243f75f 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 22, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
4d91da9 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
emmanueltouzery added a commit to emmanueltouzery/beats that referenced this issue May 22, 2024
@emmanueltouzery
fix high IO after sudden filebeat stop (elastic#35893)
19725aa 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.
belimawr added a commit that referenced this issue Jun 4, 2024
@emmanueltouzery @belimawr
Fix high IO after sudden filebeat stop (#35893) (#39392)
217f5a6 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes Filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing Filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
mergify bot pushed a commit that referenced this issue Jun 4, 2024
@emmanueltouzery @mergify
Fix high IO after sudden filebeat stop (#35893) (#39392)
0fdbb22 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes Filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing Filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
(cherry picked from commit 217f5a6)

# Conflicts:
#	libbeat/statestore/backend/memlog/diskstore.go
@mergify mergify bot mentioned this issue Jun 4, 2024
Merged
6 tasks
mergify bot pushed a commit that referenced this issue Jun 10, 2024
@emmanueltouzery @mergify
Fix high IO after sudden filebeat stop (#35893) (#39392)
c2cd690 
In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes Filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing Filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
(cherry picked from commit 217f5a6)
@mergify mergify bot mentioned this issue Jun 10, 2024
Merged
6 tasks
pierrehilbert pushed a commit that referenced this issue Jun 13, 2024
@mergify @emmanueltouzery @belimawr
[8.14](backport #39392) [Bug] fix high IO after sudden filebeat stop (#…
20fea92 
…35893) (#39842)

* Fix high IO after sudden filebeat stop (#35893) (#39392)

In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes Filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing Filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
(cherry picked from commit 217f5a6)

* Update CHANGELOG.next.asciidoc

---------

Co-authored-by: emmanueltouzery <etouzery@gmail.com>
Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
belimawr added a commit that referenced this issue Jun 24, 2024
@mergify @emmanueltouzery
@belimawr @pierrehilbert
[7.17](backport #39392) [Bug] fix high IO after sudden filebeat stop (#…
4cc14a1 
…35893) (#39795)

In case of corrupted log file (which has good chances to happen in case
of sudden unclean system shutdown), we set a flag which causes us to
checkpoint immediately, but never do anything else besides that. This
causes Filebeat to just checkpoint on each log operation (therefore
causing a high IO load on the server and also causing Filebeat to fall
behind).

This change resets the logInvalid flag after a successful checkpointing.

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
(cherry picked from commit 217f5a6)

# Conflicts:
#	libbeat/statestore/backend/memlog/diskstore.go

---------

Co-authored-by: emmanueltouzery <etouzery@gmail.com>
Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
Co-authored-by: Pierre HILBERT <pierre.hilbert@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Bug] fix high IO after sudden filebeat stop (#35893) emmanueltouzery/beats
4 participants
@belimawr @emmanueltouzery @elasticmachine @Hitych