Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "[Agent] Remove heartbeat from bundled beats in agent." #22695

Merged
merged 12 commits into from
Feb 8, 2021

Conversation

shahzad31
Copy link
Contributor

Adding heartbeat into agent, since we are restarting work on fleet integration with uptime/heartbeat in 7.11

Reverts #21602

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Ingest Management labels Nov 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 20, 2020
@blakerouse
Copy link
Contributor

I think this brings back into the question if heartbeat should be bundled. With the change it will be bundled.

I think its best for it to be bundled, but that is up for discussion.

@ph @ruflin ^

@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 20, 2020

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: andrewvc commented: /test

  • Start Time: 2021-02-08T16:37:27.389+0000

  • Duration: 46 min 21 sec

  • Commit: bf4bb2e

Test stats 🧪

Test Results
Failed 1
Passed 45755
Skipped 4762
Total 50518

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Build&Test / winlogbeat-windows-windows-2008-r2 / TestOpenInvalidProvider – github.com/elastic/beats/v7/winlogbeat/eventlog
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     === RUN   TestOpenInvalidProvider
    2021-02-08T16:59:34.478Z	WARN	[cfgwarn]	eventlog/eventlogging.go:278	DEPRECATED: The eventlogging API reader is deprecated. Will be removed in version: 8.0
    2021-02-08T16:59:34.479Z	INFO	eventlog/eventlogging.go:131	EventLogging[nonExistentProvider] contains 53 records. Record number range [4716, 4768]. Starting at 4716 (ignoringFirst=false)
    2021-02-08T16:59:34.480Z	DEBUG	[eventlog]	eventlog/eventlogging.go:161	EventLogging[nonExistentProvider] ReadEventLog returned error Access is denied.
        eventlogging_test.go:279: 
            	Error Trace:	eventlogging_test.go:279
            	Error:      	Received unexpected error:
            	            	Access is denied.
            	Test:       	TestOpenInvalidProvider
    2021-02-08T16:59:34.481Z	DEBUG	[eventlog]	eventlog/eventlogging.go:210	EventLogging[nonExistentProvider] Closing handle
    --- FAIL: TestOpenInvalidProvider (0.00s)
     
    

Steps errors 2

Expand to view the steps failures

winlogbeat-windows-windows-2008-r2 - mage build unitTest
  • Took 1 min 3 sec . View more details on here
  • Description: mage build unitTest
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-02-08T17:23:23.424Z] 16.77s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_056_o365
[2021-02-08T17:23:23.424Z] 15.66s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_260_snort
[2021-02-08T17:23:23.424Z] 14.04s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_065_o365
[2021-02-08T17:23:23.424Z] 9.91s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_090_ibmmq
[2021-02-08T17:23:23.424Z] 9.84s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_196_cisco
[2021-02-08T17:23:23.424Z] 9.62s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_088_ibmmq
[2021-02-08T17:23:23.424Z] 9.41s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_208_cisco
[2021-02-08T17:23:23.424Z] 9.40s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_108_panw
[2021-02-08T17:23:23.424Z] 9.31s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_107_panw
[2021-02-08T17:23:23.424Z] 9.26s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_105_panw
[2021-02-08T17:23:23.424Z] 9.07s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_014_oracle
[2021-02-08T17:23:23.424Z] 9.03s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_006_zscaler
[2021-02-08T17:23:23.424Z] 9.01s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_030_oracle
[2021-02-08T17:23:23.424Z] 9.00s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_005_tomcat
[2021-02-08T17:23:23.424Z] 8.93s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_087_juniper
[2021-02-08T17:23:23.424Z] 8.84s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_037_fortinet
[2021-02-08T17:23:23.425Z] 8.83s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_009_gcp
[2021-02-08T17:23:23.425Z] 8.72s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_100_squid
[2021-02-08T17:23:23.425Z] ======================= 306 passed in 1339.24s (0:22:19) =======================
[2021-02-08T17:23:23.683Z] >> python test: Integration Testing Complete
[2021-02-08T17:23:27.316Z] Client: Docker Engine - Community
[2021-02-08T17:23:27.316Z]  Version:           20.10.3
[2021-02-08T17:23:27.316Z]  API version:       1.41
[2021-02-08T17:23:27.316Z]  Go version:        go1.13.15
[2021-02-08T17:23:27.316Z]  Git commit:        48d30b5
[2021-02-08T17:23:27.316Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-08T17:23:27.316Z]  OS/Arch:           linux/amd64
[2021-02-08T17:23:27.316Z]  Context:           default
[2021-02-08T17:23:27.316Z]  Experimental:      true
[2021-02-08T17:23:27.316Z] 
[2021-02-08T17:23:27.316Z] Server: Docker Engine - Community
[2021-02-08T17:23:27.316Z]  Engine:
[2021-02-08T17:23:27.316Z]   Version:          20.10.3
[2021-02-08T17:23:27.316Z]   API version:      1.41 (minimum version 1.12)
[2021-02-08T17:23:27.316Z]   Go version:       go1.13.15
[2021-02-08T17:23:27.316Z]   Git commit:       46229ca
[2021-02-08T17:23:27.316Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-08T17:23:27.316Z]   OS/Arch:          linux/amd64
[2021-02-08T17:23:27.316Z]   Experimental:     false
[2021-02-08T17:23:27.316Z]  containerd:
[2021-02-08T17:23:27.316Z]   Version:          1.4.3
[2021-02-08T17:23:27.316Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-08T17:23:27.316Z]  runc:
[2021-02-08T17:23:27.316Z]   Version:          1.0.0-rc92
[2021-02-08T17:23:27.316Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-08T17:23:27.316Z]  docker-init:
[2021-02-08T17:23:27.316Z]   Version:          0.19.0
[2021-02-08T17:23:27.316Z]   GitCommit:        de40ad0
[2021-02-08T17:23:27.316Z] Unable to find image 'alpine:3.4' locally
[2021-02-08T17:23:27.882Z] 3.4: Pulling from library/alpine
[2021-02-08T17:23:28.139Z] c1e54eec4b57: Pulling fs layer
[2021-02-08T17:23:28.397Z] c1e54eec4b57: Download complete
[2021-02-08T17:23:28.655Z] c1e54eec4b57: Pull complete
[2021-02-08T17:23:28.655Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-02-08T17:23:28.655Z] Status: Downloaded newer image for alpine:3.4
[2021-02-08T17:23:30.842Z] + python .ci/scripts/pre_archive_test.py
[2021-02-08T17:23:32.871Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-02-08T17:23:32.894Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22695/src/github.com/elastic/beats/build
[2021-02-08T17:23:33.293Z] + rm -rf ve
[2021-02-08T17:23:33.294Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-02-08T17:23:33.312Z] Recording test results
[2021-02-08T17:23:36.518Z] [Checks API] No suitable checks publisher found.
[2021-02-08T17:23:36.875Z] Client: Docker Engine - Community
[2021-02-08T17:23:36.875Z]  Version:           20.10.3
[2021-02-08T17:23:36.875Z]  API version:       1.41
[2021-02-08T17:23:36.875Z]  Go version:        go1.13.15
[2021-02-08T17:23:36.875Z]  Git commit:        48d30b5
[2021-02-08T17:23:36.876Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-08T17:23:36.876Z]  OS/Arch:           linux/amd64
[2021-02-08T17:23:36.876Z]  Context:           default
[2021-02-08T17:23:36.876Z]  Experimental:      true
[2021-02-08T17:23:36.876Z] 
[2021-02-08T17:23:36.876Z] Server: Docker Engine - Community
[2021-02-08T17:23:36.876Z]  Engine:
[2021-02-08T17:23:36.876Z]   Version:          20.10.3
[2021-02-08T17:23:36.876Z]   API version:      1.41 (minimum version 1.12)
[2021-02-08T17:23:36.876Z]   Go version:       go1.13.15
[2021-02-08T17:23:36.876Z]   Git commit:       46229ca
[2021-02-08T17:23:36.876Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-08T17:23:36.876Z]   OS/Arch:          linux/amd64
[2021-02-08T17:23:36.876Z]   Experimental:     false
[2021-02-08T17:23:36.876Z]  containerd:
[2021-02-08T17:23:36.876Z]   Version:          1.4.3
[2021-02-08T17:23:36.876Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-08T17:23:36.876Z]  runc:
[2021-02-08T17:23:36.876Z]   Version:          1.0.0-rc92
[2021-02-08T17:23:36.876Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-08T17:23:36.876Z]  docker-init:
[2021-02-08T17:23:36.876Z]   Version:          0.19.0
[2021-02-08T17:23:36.876Z]   GitCommit:        de40ad0
[2021-02-08T17:23:47.097Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-02-08T17:23:47.155Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22695/src/github.com/elastic/beats
[2021-02-08T17:23:47.598Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-22695
[2021-02-08T17:23:47.824Z] [INFO] getVaultSecret: Getting secrets
[2021-02-08T17:23:47.915Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-08T17:23:49.919Z] + chmod 755 generate-build-data.sh
[2021-02-08T17:23:49.919Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22695/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22695/runs/11 FAILURE 2781148
[2021-02-08T17:23:49.919Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22695/runs/11/steps/?limit=10000 -o steps-info.json
[2021-02-08T17:23:55.092Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22695/runs/11/tests/?status=FAILED -o tests-errors.json
[2021-02-08T17:23:58.262Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22695/runs/11/log/ -o pipeline-log.txt

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 45755
Skipped 4762
Total 50518

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / winlogbeat-windows-windows-2008-r2 / TestOpenInvalidProvider – github.com/elastic/beats/v7/winlogbeat/eventlog

@shahzad31
Copy link
Contributor Author

@elasticmachine merge upstream

@shahzad31 shahzad31 added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Nov 23, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@michalpristas
Copy link
Contributor

michalpristas commented Nov 23, 2020

if this is just to ease up development on apm side and not a product wise decision i think it would be better to focus on this issue instead #19623 or something similar so you can pack in HB on demand

@ruflin
Copy link
Member

ruflin commented Nov 24, 2020

We need to have the bundling discussion again. I'm suggesting to first add support for heartbeat and not bundle it to unblock @shahzad31 and we will have a follow up discussion on bundling it or not as a follow up.

@paulb-elastic
Copy link
Contributor

There is a related discussion issue open regarding the packaging of Synthetics/Heartbeat/Agent

@shahzad31 shahzad31 closed this Nov 30, 2020
@shahzad31 shahzad31 deleted the revert-21602-revert-heartbeat-agent branch November 30, 2020 15:05
@andrewvc andrewvc restored the revert-21602-revert-heartbeat-agent branch January 25, 2021 20:58
@andrewvc andrewvc reopened this Jan 25, 2021
@andresrc andresrc added Team:Elastic-Agent Label for the Agent team and removed Team:Ingest Management labels Jan 25, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@andrewvc
Copy link
Contributor

As a note on testing, in the x-pack/elastic-agent folder, you can run:

env DEV=true PLATFORMS="+all linux/amd64" mage package will build a linux version of this in: x-pack/elastic-agent/build/distributions.

Use darwin/amd64 if you're on a mac.

You can unpack the tar.gz in build/distributions and add a monitor to the elastic-agent.yml file using the heartbeat syntax, making sure to prefix the type config option with synthetics/ so, for http you'd use:

inputs:
  - type: synthetics/http
    id: blahblah
    name: Blah
    urls: "https://www.elastic.co"
    schedule: "@every 15s"

@urso
Copy link

urso commented Feb 4, 2021

I've added data_stream support in my most recent commit 60bfdfe . I still need some sort of testing there, but it could probably use a quick review if @urso you have a minute to give your thoughts on doing it the way I'm doing.

The commit doesn't seem to be included in this PR, but the idea looks good to me.

@shahzad31 shahzad31 requested a review from a team as a code owner February 4, 2021 20:24
@andrewvc
Copy link
Contributor

andrewvc commented Feb 4, 2021

@urso apologies, pushed to the wrong remote, it's now updated.

@@ -52,9 +53,16 @@ type publishSettings struct {
// Output meta data settings
Pipeline string `config:"pipeline"` // ES Ingest pipeline name
Index fmtstr.EventFormatString `config:"index"` // ES output index pattern
DataStream *datastream `config:"data_stream"`
DataSet string `config:"dataset"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like you are missing a go fmt

index := fmt.Sprintf("%s-%s-%s",
settings.DataStream.Type,
settings.DataStream.Dataset,
settings.DataStream.Namespace)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if Type is not set you might want to set it to synthetics. Empty Dataset => generic, or empty Namespace would be default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmm, I actually wonder if we should just hardcode the type is there actually use-case for overriding it given that heartbeat only ever produces the one type?

@urso
Copy link

urso commented Feb 4, 2021

Do we require modifications to the heartbeat.spec? Have you tested the modification with the synthetics package?

@andrewvc
Copy link
Contributor

andrewvc commented Feb 4, 2021

@urso we have not tested it yet, @justinkambic and @dominiqueclarke are working on the package and should be testing it against this branch shortly. I believe the spec should work fine here. Are there specific concerns you have?

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

dataset := settings.DataSet
if dataset == "" {
dataset = "uptime"
if settings.DataStream.Dataset != "" {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataStream is a pointer. Do we need a nil check here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!


index := fmt.Sprintf("%s-%s-%s",
typ,
settings.DataStream.Dataset,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should top-level Dataset take precedence?

how about:

func defaultStr(str, def string) string {
  if str != "" {
    return str
  }
  return def
}

...

index := fmt.Sprintf("%s-%s-%s",
  defaultStr(settings.DataStream.Type, "synthetics"),
  defaultStr(settings.Dataset, defaultStr(settings.DataStream.Dataset, "generic")),
  defaultStr(settings.DataStream.Namespace, "default")
)

}
typ := settings.DataStream.Type
if typ == "" {
typ = "generic"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be synthetics, such that the index name becomes synthetics-... by default?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think I misunderstood a previous discussion of ours, glad to change it

@andrewvc
Copy link
Contributor

andrewvc commented Feb 5, 2021

@urso I believe I've resolved your concerns in edf4173 , mind taking a look?

Version: "8.0.0",
}
tests := []struct {
name string
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, remove name and:

tests := map[string]struct { ... }

@andrewvc
Copy link
Contributor

andrewvc commented Feb 5, 2021

@urso 92ea9ac should resolve your last concern re: test structure using a map

@andrewvc
Copy link
Contributor

andrewvc commented Feb 8, 2021

/test

@urso urso merged commit f87eac2 into master Feb 8, 2021
@urso urso deleted the revert-21602-revert-heartbeat-agent branch February 8, 2021 20:55
andrewvc pushed a commit to andrewvc/beats that referenced this pull request Feb 8, 2021
…ic#22695)

Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
(cherry picked from commit f87eac2)
andrewvc added a commit that referenced this pull request Feb 9, 2021
… (#23914)

Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
(cherry picked from commit f87eac2)

Co-authored-by: Shahzad <shahzad.muhammad@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Heartbeat Team:Elastic-Agent Label for the Agent team Team:obs-ds-hosted-services Label for the Observability Hosted Services team v7.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants