Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init package libbeat/statestore #19117

Merged
merged 4 commits into from
Jun 12, 2020
Merged

Init package libbeat/statestore #19117

merged 4 commits into from
Jun 12, 2020

Conversation

urso
Copy link

@urso urso commented Jun 10, 2020

  • Enhancement

What does this PR do?

Initialize support for the statestore package. The addition of the
statestore package is split up into multiple changeset to ease review.
The final version of the package can be found here.

Once finalized, the libbeat/statestore package contains:

  • The statestore frontend and interface for use within Beats
  • Interfaces for the store backend
  • A common set of tests store backends need to support
  • a storetest package for testing new features that require a store. The
    testing helpers use map[string]interface{} that can be initialized or
    queried after the test run for validation purposes.
  • The default memlog backend + tests

This change includes the frontend and backend interfaces only. Once merged we will add the tests and finally the memlog store.

Why is it important?

The statestore introduces a simple key-value store to Beats. The statestore will be used to replace the registry in filebeat in the future.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 10, 2020
@urso urso added libbeat needs_backport PR is waiting to be backported to other branches. review Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0 labels Jun 10, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 10, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 10, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19117 event]

  • Start Time: 2020-06-11T07:43:58.351+0000

  • Duration: 17 min 25 sec

Steps errors

Expand to view the steps failures

  • Name: Make check
    • Description: make check

    • Duration: 10 min 6 sec

    • Start Time: 2020-06-11T07:50:50.469+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-11T08:01:00.474Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-11T08:01:00.474Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-06-11T08:01:00.553Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.554Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.555Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-11T08:01:00.556Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.557Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.558Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.559Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.560Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:00.561Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-11T08:01:01.071Z] Failed in branch Elastic Agent x-pack
[2020-06-11T08:01:01.072Z] Failed in branch Elastic Agent x-pack Windows
[2020-06-11T08:01:01.072Z] Failed in branch Elastic Agent Mac OS X
[2020-06-11T08:01:01.073Z] Failed in branch Filebeat oss
[2020-06-11T08:01:01.074Z] Failed in branch Filebeat x-pack
[2020-06-11T08:01:01.074Z] Failed in branch Filebeat Mac OS X
[2020-06-11T08:01:01.075Z] Failed in branch Filebeat x-pack Mac OS X
[2020-06-11T08:01:01.075Z] Failed in branch Filebeat Windows
[2020-06-11T08:01:01.076Z] Failed in branch Filebeat x-pack Windows
[2020-06-11T08:01:01.076Z] Failed in branch Auditbeat oss Linux
[2020-06-11T08:01:01.077Z] Failed in branch Auditbeat crosscompile
[2020-06-11T08:01:01.078Z] Failed in branch Auditbeat oss Mac OS X
[2020-06-11T08:01:01.078Z] Failed in branch Auditbeat oss Windows
[2020-06-11T08:01:01.079Z] Failed in branch Auditbeat x-pack
[2020-06-11T08:01:01.079Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-06-11T08:01:01.080Z] Failed in branch Auditbeat x-pack Windows
[2020-06-11T08:01:01.080Z] Failed in branch Libbeat x-pack
[2020-06-11T08:01:01.081Z] Failed in branch Metricbeat OSS Unit tests
[2020-06-11T08:01:01.081Z] Failed in branch Metricbeat OSS Integration tests
[2020-06-11T08:01:01.082Z] Failed in branch Metricbeat Python integration tests
[2020-06-11T08:01:01.083Z] Failed in branch Metricbeat crosscompile
[2020-06-11T08:01:01.083Z] Failed in branch Metricbeat Mac OS X
[2020-06-11T08:01:01.084Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-06-11T08:01:01.084Z] Failed in branch Metricbeat Windows
[2020-06-11T08:01:01.085Z] Failed in branch Metricbeat x-pack Windows
[2020-06-11T08:01:01.085Z] Failed in branch Winlogbeat Windows x-pack
[2020-06-11T08:01:01.086Z] Failed in branch Kubernetes
[2020-06-11T08:01:01.340Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.342Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.344Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-11T08:01:01.345Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.346Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.347Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-11T08:01:01.403Z] Failed in branch Packetbeat
[2020-06-11T08:01:01.406Z] Failed in branch dockerlogbeat
[2020-06-11T08:01:01.407Z] Failed in branch Journalbeat
[2020-06-11T08:01:01.592Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.593Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.594Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-11T08:01:01.595Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-11T08:01:01.632Z] Failed in branch Metricbeat x-pack
[2020-06-11T08:01:01.633Z] Failed in branch Winlogbeat
[2020-06-11T08:01:01.806Z] Failed in branch Heartbeat
[2020-06-11T08:01:01.806Z] Failed in branch Libbeat
[2020-06-11T08:01:01.807Z] Failed in branch Functionbeat
[2020-06-11T08:01:01.808Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-11T08:01:01.881Z] Failed in branch Generators
[2020-06-11T08:01:02.006Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19117/src/github.com/elastic/beats
[2020-06-11T08:01:02.317Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-11T08:01:02.330Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19117/src/github.com/elastic/beats/Lint
[2020-06-11T08:01:02.710Z] + cat
[2020-06-11T08:01:02.710Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-11T08:01:02.710Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-11T08:01:09.315Z] runbld>>> runbld started
[2020-06-11T08:01:09.315Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-11T08:01:11.238Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19117' in order of occurrence in the config (last value wins).
[2020-06-11T08:01:12.629Z] runbld>>> Debug logging enabled.
[2020-06-11T08:01:12.629Z] runbld>>> Storing result
[2020-06-11T08:01:12.891Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-11T08:01:12.891Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200611080112-C6B4EC78
[2020-06-11T08:01:12.891Z] runbld>>> Adding system facts.
[2020-06-11T08:01:13.843Z] runbld>>> Adding vcs info for the latest commit:  7bb2a3337f5c3d3802f12814740c9b1676bef9c5
[2020-06-11T08:01:13.843Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-11T08:01:13.843Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-11T08:01:13.843Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-11T08:01:13.843Z] Processing JUnit reports with runbld...
[2020-06-11T08:01:14.417Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-11T08:01:14.417Z] runbld>>> DURATION: 8ms
[2020-06-11T08:01:14.417Z] runbld>>> STDOUT: 40 bytes
[2020-06-11T08:01:14.417Z] runbld>>> STDERR: 49 bytes
[2020-06-11T08:01:14.417Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-11T08:01:14.417Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19117/src/github.com/elastic/beats
[2020-06-11T08:01:15.367Z] runbld>>> Storing build metadata: 
[2020-06-11T08:01:15.367Z] runbld>>> Adding test report.
[2020-06-11T08:01:15.367Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19117/src/github.com/elastic/beats
[2020-06-11T08:01:16.318Z] runbld>>> Found 0 test output files
[2020-06-11T08:01:16.318Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-06-11T08:01:16.580Z] runbld>>> Storing result
[2020-06-11T08:01:16.580Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-11T08:01:16.580Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200611080112-C6B4EC78
[2020-06-11T08:01:16.580Z] runbld>>> Email notification disabled by environment variable.
[2020-06-11T08:01:16.580Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-11T08:01:22.358Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19117
[2020-06-11T08:01:22.603Z] [INFO] getVaultSecret: Getting secrets
[2020-06-11T08:01:22.659Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-11T08:01:23.374Z] + chmod 755 generate-build-data.sh
[2020-06-11T08:01:23.374Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19117/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19117/runs/5 FAILURE 1044762
[2020-06-11T08:01:23.374Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19117/runs/5/steps/?limit=10000 -o steps-info.json
[2020-06-11T08:01:23.624Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19117/runs/5/tests/?status=FAILED -o tests-errors.json

@urso urso mentioned this pull request Jun 10, 2020
27 tasks
urso added 4 commits June 11, 2020 15:52
Initialize support for the statestore package. The addition of the
statestore package is split up into multiple changeset to ease review.
The final version of the package can be found here: https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore

Once finalized, the libbeat/statestore package contains:
- The statestore frontend and interface for use within Beats
- Interfaces for the store backend
- A common set of tests store backends need to support
- a storetest package for testing new features that require a store. The
  testing helpers use map[string]interface{} that can be initialized or
  queried after the test run for validation purposes.
- The default memlog backend + tests

This change include the frontend and backend interfaces only
Copy link
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🎉 🎉

@urso urso merged commit 1fc18c2 into elastic:master Jun 12, 2020
@urso urso deleted the statestore-init branch June 12, 2020 12:38
v1v added a commit to v1v/beats that referenced this pull request Jun 12, 2020
…ngs-archive

* upstream/master: (119 commits)
  Update filebeat input docs (elastic#19110)
  Add ECS fields from log pipeline of PostgreSQL (elastic#19127)
  Init package libbeat/statestore (elastic#19117)
  [Ingest Manager] Retryable downloads of beats (elastic#19102)
  [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (elastic#18965)
  Add compatibility info (elastic#18929)
  Set ecszap version to v0.2.0 (elastic#19106)
  [filebeat][httpjson] Fix unit test function call (elastic#19124)
  [Filebeat][httpjson] Adds oauth2 support for httpjson input (elastic#18892)
  Allow host.* fields to be disabled in Suricata module (elastic#19107)
  Make selector string casing configurable (elastic#18854)
  Switch the GRPC communication where Agent is running the server and the beats are connecting back to Agent (elastic#18973)
  Disable host.* fields by default for netflow module (elastic#19087)
  Automatically fill zube teams on backports if available (elastic#18924)
  Fix crash on vsphere module (elastic#19078)
  [Ingest Manager] Download snapshot artifacts from snapshots repo (elastic#18685)
  [Ingest Manager] Basic Elastic Agent documentation (elastic#19030)
  Make user.id a string in system/users, in line with ECS (elastic#19019)
  [docs] Add 7.8 release highlights placeholder file (elastic#18493)
  Fix translate_sid's empty target field handling (elastic#18991)
  ...
urso pushed a commit to urso/beats that referenced this pull request Jul 3, 2020
Initialize support for the statestore package. The addition of the
statestore package is split up into multiple changeset to ease review.
The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore).

Once finalized, the libbeat/statestore package contains:
- The statestore frontend and interface for use within Beats
- Interfaces for the store backend
- A common set of tests store backends need to support
- a storetest package for testing new features that require a store. The
  testing helpers use map[string]interface{} that can be initialized or
  queried after the test run for validation purposes.
- The default memlog backend + tests

This change includes the frontend and backend interfaces only. Once merged we will add the tests and finally the memlog store.

(cherry picked from commit 1fc18c2)
@urso urso removed the needs_backport PR is waiting to be backported to other branches. label Jul 3, 2020
urso pushed a commit that referenced this pull request Jul 6, 2020
Initialize support for the statestore package. The addition of the
statestore package is split up into multiple changeset to ease review.
The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore).

Once finalized, the libbeat/statestore package contains:
- The statestore frontend and interface for use within Beats
- Interfaces for the store backend
- A common set of tests store backends need to support
- a storetest package for testing new features that require a store. The
  testing helpers use map[string]interface{} that can be initialized or
  queried after the test run for validation purposes.
- The default memlog backend + tests

This change includes the frontend and backend interfaces only. Once merged we will add the tests and finally the memlog store.

(cherry picked from commit 1fc18c2)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
Initialize support for the statestore package. The addition of the
statestore package is split up into multiple changeset to ease review.
The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore).

Once finalized, the libbeat/statestore package contains:
- The statestore frontend and interface for use within Beats
- Interfaces for the store backend
- A common set of tests store backends need to support
- a storetest package for testing new features that require a store. The
  testing helpers use map[string]interface{} that can be initialized or
  queried after the test run for validation purposes.
- The default memlog backend + tests

This change includes the frontend and backend interfaces only. Once merged we will add the tests and finally the memlog store.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libbeat Project:Filebeat-Input-v2 review skip-test-plan Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants