Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporting prometheus metrics for OVS performance and routes #236

Closed
jayunit100 opened this issue Dec 16, 2019 · 5 comments · Fixed by #321
Closed

Exporting prometheus metrics for OVS performance and routes #236

jayunit100 opened this issue Dec 16, 2019 · 5 comments · Fixed by #321
Assignees
@ksamoray
Labels
area/monitoring/health-performance Issues or PRs related to health and performance monitoring enhancement New feature or request kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone
Antrea v0.6.0 rel...

Comments

@jayunit100
Copy link
Contributor

Describe the problem/challenge you have
It would be nice to be able to visualize and integrate CNIperformance over time with other dashboards in a K8s environment via prometheus

Describe the solution you'd like
Instrumenting the calls made to OVS with prometheus counters and so on.
@jayunit100 jayunit100 added the enhancement New feature or request label Dec 16, 2019
@antoninbas
Copy link
Contributor

This was discussed at the 12/18/2019 Antrea Community Meeting. The decision was to start scoping the work and see if we can have some initial integration with support for non highly dynamic metrics for v0.3.0. @ksamoray volunteered to drive that.

@antoninbas antoninbas assigned antoninbas and ksamoray and unassigned antoninbas Dec 19, 2019
@antoninbas antoninbas added this to the Antrea v0.3.0 release milestone Dec 19, 2019
@antoninbas
Copy link
Contributor

As discussed at the 01/22/2020 community meeting, we are pushing this to 0.4.0. @ksamoray has an open PR (#321) ready for review, which he can present at the next community meeting.

@jayunit100
Copy link
Contributor Author

great thanks !

@ksamoray
Copy link
Contributor

@McCodeman
Enlisted below the capabilities and metrics which are currently covered within Antrea PRs.
Please provide your feedback and recommend any additional required metrics.

** PR #321 **
From the agent:

  • Initialize prometheus exporter, expose config parameters
  • Expose process, golang metrics per the agent process (or exclude them)
  • Expose the following agent metrics:
    • antrea_agent_host: Antrea agent hostname (as a label)
    • antrea_agent_local_pod_count: Number of pods on local node
    • antrea_agent_ovs_flow_table: Per-OVS-table flow count (table id within a label)

From the controller:

  • Initialize prometheus exporter, expose config parameters
  • Expose process, golang metrics per the agent process (or exclude them)
  • Expose the following controller metrics:
  • antrea_controller_host: Antrea agent hostname (as a label)

** PR #325 **

  • Expose the following controller metrics:
    • antrea_controller_applied_to_group_processed: The total number of applied-to-group processed
    • antrea_controller_address_group_processed: The total number of address-group processed
    • antrea_controller_network_policy_processed: The total number of internal-networkpolicy processed
    • antrea_controller_applied_to_group_process_duration_milliseconds: The duration of processing applied-to groups
    • antrea_controller_address_group_process_duration_milliseconds: The duration of processing address-group
    • antrea_controller_network_policy_process_duration_milliseconds: The duration of processing internal-networkpolicy
    • antrea_controller_applied_to_group_sync_duration_milliseconds: The duration of syncing applied-to-group
    • antrea_controller_address_group_sync_duration_milliseconds: The duration of syncing address-group
    • antrea_controller_network_policy_sync_duration_milliseconds: The duration of syncing internal-networkpolicy
    • antrea_controller_length_applied_to_group_queue: The length of AppliedToGroupQueue
    • antrea_controller_length_address_group_queue: The length of AddressGroupQueue
    • antrea_controller_length_network_policy_queue: The length of InternalNetworkPolicyQueue

@salv-orlando salv-orlando removed this from the Antrea v0.4.0 release milestone Feb 12, 2020
@antoninbas antoninbas added this to the Antrea v0.5.0 release milestone Feb 12, 2020
@McCodeman McCodeman added kind/feature Categorizes issue or PR as related to a new feature. area/monitoring/health-performance Issues or PRs related to health and performance monitoring lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Feb 13, 2020
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
59f4ade 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
1d8987c 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
iAntrea Prometheus integration (antrea-io#236)
be259b1 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
7361fd9 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
2844f1e 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 17, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
349d2b4 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 22, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
e15a3e6 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 23, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
e401e3b 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 24, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
a51f9d7 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
@salv-orlando
Copy link
Contributor

@ksamoray completed patch reorganization, but it's too late for Antrea 0.5 inclusion. I would consider deferring this integration to 0.6 (and also enable prometheus in CI - so that we can verify data collection works ok)

ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 25, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
dbe8492 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 26, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
90603cd 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 29, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
1b83e6a 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 30, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
e3c7dcb 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 31, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
abe6c9d 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Mar 31, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
a674dce 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 1, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
bc3697f 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 5, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
f8ad50d 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 21, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
ea126e6 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 21, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
1a95323 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 21, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
f0bda31 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 21, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
9f2a954 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 22, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
4c7ff9d 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 22, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
0ab5d9e 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 23, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
f172a5a 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 23, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
8dbb082 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 23, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
ff273ee 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 23, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
0ab4ff6 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 25, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
6abee53 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 28, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
181d3ed 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
ksamoray added a commit to ksamoray/antrea that referenced this issue Apr 28, 2020
@ksamoray
Antrea Prometheus integration (antrea-io#236)
9845b1b 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.
antoninbas pushed a commit that referenced this issue Apr 29, 2020
@ksamoray
Antrea Prometheus integration (#321)
ec17ec0 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.

See #236
Dyanngg pushed a commit to Dyanngg/antrea that referenced this issue Apr 29, 2020
@ksamoray @Dyanngg
Antrea Prometheus integration (antrea-io#321)
6e3be6c 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.

See antrea-io#236
McCodeman pushed a commit to McCodeman/antrea that referenced this issue Jun 2, 2020
@ksamoray @McCodeman
Antrea Prometheus integration (antrea-io#321)
e0df6b7 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.

See antrea-io#236
McCodeman pushed a commit that referenced this issue Jun 2, 2020
@ksamoray @McCodeman
Antrea Prometheus integration (#321)
ad86396 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.

See #236
GraysonWu pushed a commit to GraysonWu/antrea that referenced this issue Sep 22, 2020
@ksamoray @GraysonWu
Antrea Prometheus integration (antrea-io#321)
d630388 
Integrate with Prometheus monitoring solution.
Integration of the Prometheus client into Antrea controller and agent
allows the exposure of various metrics to Prometheus server.
In addition to Antrea's own set of metrics, Prometheus client will also
expose metrics which are defined by various components which are part of
the Antrea ecosystem, e.g golang, Prometheus itself etc.

See antrea-io#236
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ksamoray ksamoray

Labels
area/monitoring/health-performance Issues or PRs related to health and performance monitoring enhancement New feature or request kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
Antrea v0.6.0 release
Development

Successfully merging a pull request may close this issue.

Antrea Prometheus integration ksamoray/antrea
5 participants
@jayunit100 @salv-orlando @antoninbas @McCodeman @ksamoray