Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add cloudfoundry module to metricbeat #16671

Merged
merged 22 commits into from
Mar 2, 2020
Merged

Add cloudfoundry module to metricbeat #16671

merged 22 commits into from
Mar 2, 2020

Conversation

blakerouse
Copy link
Contributor

@blakerouse blakerouse commented Feb 27, 2020 •
edited by andresrc
Loading

What does this PR do?

Adds a new cloudfoundry module to x-pack/metricbeat. Adds 3 metric sets container, counter, and value.

This module/metricsets are different from standard metricsets as the Cloud Foundry loggregator pushes to metrics to the client. Each metricset shares the same connection from the module so it does not create a connection per metricset.

Why is it important?

Adds support for metrics from cloudfoundry.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • Receives metrics from Cloud Foundry loggregator.
  • Selecting individual metricsets works.

How to test this PR locally

With a running Cloud Foundry run metricbeat with the following configuration:

metricbeat.modules:
  - &cloudfoundry
    module: cloudfoundry
    api_address: https://api.dev.cfdev.sh
    client_id: firehose-to-filebeat
    client_secret: verysecret
    ssl:
      verification_mode: none

processors:
  - add_cloudfoundry_metadata:
      <<: *cloudfoundry

output.console:
  pretty: true

Related issues

Use cases

Screenshots

Logs

@blakerouse blakerouse requested a review from a team as a code owner February 27, 2020 19:08
@blakerouse blakerouse added enhancement review Team:Platforms Label for the Integrations - Platforms team labels Feb 27, 2020
blakerouse and others added 16 commits February 27, 2020 15:22
@blakerouse
Run make update.
8e0fdf0
@blakerouse
Fix documentation add changelog.
b3be6f7
@blakerouse
Fix bad change in azure module.
cbc4c67
@blakerouse
Run make update.
f9a3f21
@blakerouse
Improve reference config.
99be099
@ycombinator @dedemorton @blakerouse
Document Kibana Space ID setting (elastic#16622)
5da73b2 
* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
@ph @blakerouse
Remove the special pins type for ca_sha256 (elastic#16615)
23d7c06 
We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.
@blakerouse
Remove feature.Describer and enhance Details (elastic#16658)
e79af86
@faec @blakerouse
Make memqueue.Broker internal (now memqueue.broker) (elastic#16667)
e4e683b
@dedemorton @blakerouse
[docs] Add skeleton files for 7.7 release highlights and breaking cha…
421fd91 
…nges (elastic#16584)
@kaiyan-sheng @blakerouse
[Metricbeat] Add vpc metricset for aws module (elastic#16111)
6e79791 
* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway
@dedemorton @blakerouse
Update cisco.asciidoc (elastic#16434)
ba49a12 
* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
@dedemorton @blakerouse
Fix wording in dev guide (elastic#16497)
8d5367a
@blakerouse
Added NetBeat (elastic#12612)
60c3df0
@jsoriano @blakerouse
Close files earlier when generating fields files (elastic#16681)
37ab97d 
Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.
@mtojek @blakerouse
[Metricbeat] Add new module for Redis Enterprise (elastic#16482)
06c7585 
Add metricsets: node, proxy
@blakerouse blakerouse requested review from a team as code owners February 28, 2020 14:26
@blakerouse blakerouse removed request for a team February 28, 2020 14:26
sayden
sayden reviewed Mar 2, 2020
View reviewed changes
Copy link
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have taken an overall look and it seems fine.

However it will be nice to have some comments for maintainers to help in the logic flow 😉

@blakerouse
Copy link
Contributor Author

@kaiyan-sheng Added the config fields to the reference yaml if you could give it another look. Thanks!

kaiyan-sheng
kaiyan-sheng approved these changes Mar 2, 2020
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@blakerouse blakerouse merged commit 07b03a7 into elastic:master Mar 2, 2020
@blakerouse blakerouse deleted the metricbeat-cf-module branch March 2, 2020 22:40
@blakerouse blakerouse mentioned this pull request Mar 2, 2020
Merged
5 tasks
blakerouse added a commit to blakerouse/beats that referenced this pull request Mar 3, 2020
@blakerouse
Add cloudfoundry module to metricbeat (elastic#16671)
44e4375 
* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (elastic#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (elastic#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (elastic#16658)

* Make memqueue.Broker internal (now memqueue.broker) (elastic#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (elastic#16584)

* [Metricbeat] Add vpc metricset for aws module (elastic#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (elastic#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (elastic#16497)

* Added NetBeat (elastic#12612)

* Close files earlier when generating fields files (elastic#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (elastic#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
(cherry picked from commit 07b03a7)
blakerouse added a commit that referenced this pull request Mar 3, 2020
@blakerouse
Cherry-pick #16671 to 7.x: Add cloudfoundry module to metricbeat (#16741
5f7c5a6 
)

* Add cloudfoundry module to metricbeat (#16671)

* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (#16658)

* Make memqueue.Broker internal (now memqueue.broker) (#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584)

* [Metricbeat] Add vpc metricset for aws module (#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (#16497)

* Added NetBeat (#12612)

* Close files earlier when generating fields files (#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
(cherry picked from commit 07b03a7)

* Fix changelog.

* Run mage fmt.

* Fix imports to have v7.

* Run mage fmt update.

* Fix changelog.
@jsoriano jsoriano mentioned this pull request Feb 16, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sayden sayden sayden left review comments

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees
No one assigned
Labels
enhancement review Team:Platforms Label for the Integrations - Platforms team v7.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@blakerouse @sayden @kaiyan-sheng @andresrc @ycombinator @ph @faec @dedemorton @jsoriano @mtojek