[x-pack/metricbeat] ec2_integration_test

[v1v]

Flaky Test

• Test Name: TestFetch – ec2
• Link:

  beats/x-pack/metricbeat/module/aws/ec2/ec2_integration_test.go

  Line 28 in 8fce110

  assert.NotEmpty(t, events)

• Branch: master
• Artifact Link: build logs build logs backup (if the first link got broke)
• Notes: @kaiyan-sheng dig deeper and found that there is no metric collected by ec2 metricset, which means there is no CloudWatch metrics for EC2 yet.

Stack Trace

    ec2_integration_test.go:28: 
        	Error Trace:	ec2_integration_test.go:28
        	Error:      	Should NOT be empty, but was []
        	Test:       	TestFetch

How to reproduce it

• Go to build
• Select awsCloudTests
• Unselect windowsTest
• Click on Build

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

@jsoriano Could you point to me where is the terraform code? My suspicion is: we have a sleep/check waiting for the new created EC2 instance running but we don't check if this EC2 instance actually pushes metrics into CloudWatch yet before we start Metricbeat. Thank you!

[jsoriano]

@kaiyan-sheng scenario started in CI is the one in ./x-pack/metricbeat/module/aws/terraform.tf. I think the only instance it starts is the one for the database.

[kaiyan-sheng]

@jsoriano Thank you!! Do we have a sleep timer or check to make sure the instance is up and running and reporting metrics to CloudWatch at somewhere else?

[jsoriano]

@kaiyan-sheng I think the terraform scenario waits till the instance is up and running (it takes some time to complete the db instance), but I don't think it explicitly waits to have metrics in CloudWatch. If we need to wait for that maybe we need to do it in the test, with some timeout.

Other thing that could be happening is that ec2 instances created for db instances don't report the same CloudWatch metrics (or no ec2 metrics at all), and the test was passing before because some other ec2 instance existed on this account. In that case we would need to add an ec2 instance to the scenario.

[kaiyan-sheng]

@jsoriano Good point! I think adding a check for CloudWatch metrics will be needed. I will assign this issue to myself and put it on my todo list. Thank you!