Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autodiscover ephemeral containers in kubernetes #22389

Merged
merged 1 commit into from
Nov 4, 2020
Merged

Autodiscover ephemeral containers in kubernetes #22389

merged 1 commit into from
Nov 4, 2020

Conversation

marqc
Copy link
Contributor

@marqc marqc commented Nov 3, 2020 •
edited
Loading

Enhancement: Autodiscover ephemeral containers in kubernetes when "scrape_ephemeral_containers" flag is set to True

What does this PR do?

Adds autodiscovery of kubernetes ephemeral containers. It is disabled by default and needs to set "scrape_ephemeral_containers" flag to "True"

Why is it important?

I'm using filebeat to scrape logs of pods in my organizations kubernetes clusters, and there is currently no support for collecting logs for ephemeral containers.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

  1. Set kubernetes cluster with EphemeralContainers feature gate enabled
  2. Run filebeat on kubernetes cluster with autodiscovery as in official docs.
  3. Add config flag "scrape_ephemeral_containers: True"
  4. Start ephemeral container and check if filebeat starts collecting logs

Related issues

Use cases

Add support for autodiscovery of ephemeral containers in kubernetes cluster

Screenshots

Logs

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 3, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 3, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [jsoriano commented: jenkins run the tests please]

  • Start Time: 2020-11-04T16:23:02.916+0000

  • Duration: 72 min 20 sec

Test stats 🧪

Test Results
Failed 0
Passed 16400
Skipped 1342
Total 17742

@marqc marqc force-pushed the autodiscover_kube_ephemeral_containers branch from 2171279 to fabecb0 Compare November 3, 2020 16:51
@andresrc andresrc added the Team:Platforms Label for the Integrations - Platforms team label Nov 3, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 3, 2020
@marqc marqc force-pushed the autodiscover_kube_ephemeral_containers branch from fabecb0 to 4602132 Compare November 4, 2020 07:39
jsoriano
jsoriano reviewed Nov 4, 2020
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @marqc, thanks for this contribution!

I am ok with emitting events for ephemeral containers, and I would prefer not to add an option for that, let's just discover them always.

@ChrsMark what do you think?

libbeat/autodiscover/providers/kubernetes/pod.go Outdated Show resolved Hide resolved
@jsoriano
Copy link
Member

jsoriano commented Nov 4, 2020

jenkins run the tests

@ChrsMark
Copy link
Member

ChrsMark commented Nov 4, 2020

Hey @marqc, thanks for this contribution!

I am ok with emitting events for ephemeral containers, and I would prefer not to add an option for that, let's just discover them always.

@ChrsMark what do you think?

+1 for handling ephemerals without adding an extra option. I would also suggest adding a test for this similar to

beats/libbeat/autodiscover/providers/kubernetes/pod_test.go

Line 359 in e7ba4d0

Message: "Test common pod start",

@marqc marqc force-pushed the autodiscover_kube_ephemeral_containers branch 3 times, most recently from edbf50c to f85531e Compare November 4, 2020 14:35
@marqc
Copy link
Contributor Author

marqc commented Nov 4, 2020

@ChrsMark

I would also suggest adding a test for this similar to

beats/libbeat/autodiscover/providers/kubernetes/pod_test.go

Line 359 in e7ba4d0

Message: "Test common pod start",

test added

@marqc marqc force-pushed the autodiscover_kube_ephemeral_containers branch from f85531e to b6fdec0 Compare November 4, 2020 14:38
@marqc marqc force-pushed the autodiscover_kube_ephemeral_containers branch from b6fdec0 to 379bba4 Compare November 4, 2020 15:05
@jsoriano jsoriano self-assigned this Nov 4, 2020
@jsoriano
Copy link
Member

jsoriano commented Nov 4, 2020

jenkins run the tests please

jsoriano
jsoriano approved these changes Nov 4, 2020
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, waiting for green.

libbeat/autodiscover/providers/kubernetes/pod.go Show resolved Hide resolved
@jsoriano jsoriano added needs_backport PR is waiting to be backported to other branches. v7.11.0 labels Nov 4, 2020
@elasticmachine
Copy link
Collaborator

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 16400
Skipped 1342
Total 17742

@jsoriano jsoriano merged commit 5a4d407 into elastic:master Nov 4, 2020
@jsoriano jsoriano mentioned this pull request Nov 4, 2020
Merged
6 tasks
@jsoriano jsoriano removed the needs_backport PR is waiting to be backported to other branches. label Nov 4, 2020
jsoriano pushed a commit to jsoriano/beats that referenced this pull request Nov 4, 2020
@marqc @jsoriano
Autodiscover ephemeral containers in kubernetes (elastic#22389)
94d7a28 
(cherry picked from commit 5a4d407)
@jsoriano jsoriano mentioned this pull request Nov 4, 2020
Merged
6 tasks
@jsoriano jsoriano mentioned this pull request Nov 4, 2020
Merged
6 tasks
jsoriano added a commit that referenced this pull request Nov 5, 2020
@jsoriano @marqc
Autodiscover ephemeral containers in kubernetes (#22389) (#22434)
c98e0bb 
(cherry picked from commit 5a4d407)

Co-authored-by: marqc <marqc@users.noreply.github.com>
This was referenced Nov 9, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@jsoriano jsoriano

Labels
Team:Platforms Label for the Integrations - Platforms team v7.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@marqc @elasticmachine @jsoriano @ChrsMark @andresrc