Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable writes sync in persistent cache #21754

Merged
merged 1 commit into from
Oct 14, 2020
Merged

Disable writes sync in persistent cache #21754

merged 1 commit into from
Oct 14, 2020

Conversation

jsoriano
Copy link
Member

What does this PR do?

Disable sync on writes in badger-based persistent caches.

With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.

Why is it important?

Reduce overhead caused by persistent caches and increase their performance.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works (should be covered by existing tests)
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc. (feature not released yet, and not use-facing impact).

Related issues

@jsoriano
Disable writes sync in persistent cache
9001c73 
With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.
@jsoriano jsoriano added enhancement needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team v7.10.0 v7.11.0 labels Oct 13, 2020
@jsoriano jsoriano self-assigned this Oct 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 13, 2020
@jsoriano jsoriano mentioned this pull request Oct 13, 2020
Merged
10 tasks
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21754 opened]

  • Start Time: 2020-10-13T15:09:16.496+0000

  • Duration: 92 min 25 sec

Test stats 🧪

Test Results
Failed 0
Passed 4989
Skipped 390
Total 5379

@jsoriano jsoriano merged commit 85b95ee into elastic:master Oct 14, 2020
@jsoriano jsoriano deleted the disable-badger-sync-writes branch October 14, 2020 11:15
jsoriano added a commit to jsoriano/beats that referenced this pull request Oct 14, 2020
@jsoriano
Disable writes sync in persistent cache (elastic#21754)
7833d9e 
With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.

(cherry picked from commit 85b95ee)
@jsoriano jsoriano mentioned this pull request Oct 14, 2020
Merged
6 tasks
@jsoriano jsoriano removed the needs_backport PR is waiting to be backported to other branches. label Oct 14, 2020
jsoriano added a commit to jsoriano/beats that referenced this pull request Oct 14, 2020
@jsoriano
Disable writes sync in persistent cache (elastic#21754)
425198a 
With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.

(cherry picked from commit 85b95ee)
@jsoriano jsoriano mentioned this pull request Oct 14, 2020
Merged
6 tasks
v1v added a commit to v1v/beats that referenced this pull request Oct 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/download-…
2bc6d54 
…dependencies-goals

* upstream/master: (46 commits)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  typofix for dns timeout configuration option (elastic#21069)
  chore: create CI artifacts for DEV usage (elastic#21645)
  [Ingest Manager] Atomic installed forgotten changelog elastic#21780
  [Ingest Manager] Agent atomic installer (elastic#21745)
  Add missing configuration annotations (elastic#21736)
  [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732)
  Remove kafka partition ISR from Metricbeat docs (elastic#21709)
  Skip flaky test with oauth2 config in httpjson input (elastic#21749)
  Fix for azure retrieve resource by ids (elastic#21711)
  Update obs app links (elastic#21682)
  fix: update fleet test suite name (elastic#21738)
  Remove dot from file.extension value in Auditbeat FIM (elastic#21644)
  Fix leaks with metadata processors (elastic#16349)
  Add istiod metricset (elastic#21519)
  [Ingest Manager] Change Sync/Close call order (elastic#21735)
  ...
jsoriano added a commit that referenced this pull request Oct 14, 2020
@jsoriano
Disable writes sync in persistent cache (#21754) (#21788)
7512e8c 
With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.

(cherry picked from commit 85b95ee)
jsoriano added a commit that referenced this pull request Oct 14, 2020
@jsoriano
Disable writes sync in persistent cache (#21754) (#21789)
3577134 
With default configuration, every write is synced to disk. For the
persistent cache, Beats not need to guarantee that everything is on disk,
as the data can be recovered from the source endpoints, but the database
was not being properly closed, so there could happen that most of the
recent data was lost if the beat is stopped.
Now, the processors can close their resources, so the database is
properly closed in an normal shutdown, so we can go on without syncs on
writes, what gives a much better performance.

(cherry picked from commit 85b95ee)
v1v added a commit to v1v/beats that referenced this pull request Oct 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
e60b7d6 
…ne-2.0-arm

* upstream/master: (93 commits)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  typofix for dns timeout configuration option (elastic#21069)
  chore: create CI artifacts for DEV usage (elastic#21645)
  [Ingest Manager] Atomic installed forgotten changelog elastic#21780
  [Ingest Manager] Agent atomic installer (elastic#21745)
  Add missing configuration annotations (elastic#21736)
  [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732)
  Remove kafka partition ISR from Metricbeat docs (elastic#21709)
  Skip flaky test with oauth2 config in httpjson input (elastic#21749)
  Fix for azure retrieve resource by ids (elastic#21711)
  Update obs app links (elastic#21682)
  fix: update fleet test suite name (elastic#21738)
  Remove dot from file.extension value in Auditbeat FIM (elastic#21644)
  Fix leaks with metadata processors (elastic#16349)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-10
1997de1 
* upstream/master: (29 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-7-64
662166b 
* upstream/master: (216 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-2008
a21f05c 
* upstream/master: (156 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-7-32
b359f40 
* upstream/master: (156 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@jsoriano jsoriano

Labels
enhancement Team:Platforms Label for the Integrations - Platforms team v7.10.0 v7.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jsoriano @elasticmachine @ChrsMark