-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NETOBSERV-740: Metrics integration - console plugin frontend #516
Conversation
Skipping CI for Draft Pull Request. |
This introduces prometheus as a new datasource, that can be used either as a replacement or as a complement of Loki. It doesn't require to change the frontend/backend API interface: on every frontend query, backend checks if that query is transposable to prometheus, and if so, runs it on prometheus, else falls back on loki (if it's enabled) There's some refactoring of the config and topology handlers to make place for prometheus.
To this by probing with the existing filter encoder function
Also, merge all "allowXXX" scope props into a single "allowedScopes" prop
Added 2 commits:
|
A few thoughts, IMO no blockers, perhaps that's for the next release, if we don't find time here:
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #516 +/- ##
==========================================
- Coverage 57.27% 56.71% -0.56%
==========================================
Files 169 174 +5
Lines 8487 8990 +503
Branches 1160 1179 +19
==========================================
+ Hits 4861 5099 +238
- Misses 3299 3528 +229
- Partials 327 363 +36
Flags with carried forward coverage won't be shown. Click here to find out more.
|
- Manage FlowDirection: first check if query can be performed using a metric agnostic to direction (Any); else, check both Ingress and Egress metric, and combine them in promQL with OR - Handle "getNamesForPrefix" with prom - Split "getTopologyFlows" in 2 parts for cyclo cplx - Improve error messages when query can't be performed using prometheus - Add tests
unfortunately, no. But I did manage to capture the API Response to prometheus query, it's a datapoint with timestamp: |
@memodi I don't have much idea on what could cause that :( We've seen doubled rates before, when there was a double-configured prometheus (user workload + cluster monitoring) .. any chance it could be that? (like, having set both of them for a short period of time, before removing one) |
ah, there's definitely that chance, because we use this script: https://gitlab.cee.redhat.com/netobserv-qe/netobserv-qe-scripts/-/blob/main/netobserv.sh?ref_type=heads#L309 which enables user workload monitoring but I think I didn't see metrics being populated (probably fast enough) so patched CSV with |
/ok-to-test |
New image: It will expire after two weeks. To deploy this build, run from the operator repo, assuming the operator is running: USER=netobserv VERSION=9794230 make set-plugin-image |
In a previous implementation, metric names came with their "_bucket" prefix for histograms; now this isn't the case anymore but the query generation wasn't updated accordingly
@memodi new build fixes an issue with percentiles (stats on RTT or DNS latency) |
New image: It will expire after two weeks. To deploy this build, run from the operator repo, assuming the operator is running: USER=netobserv VERSION=1b91ab5 make set-plugin-image |
// Unfortunately we don't know a safe and generic way to pre-flight check if the user will be authorized | ||
hlog.Info("Retrying with Loki...") | ||
flows, code, err = h.getTopologyFlows(ctx, clients, params, constants.DataSourceLoki) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not using something like wait.PollUntilContextTimeout
to implement proper retry instead of retrying just once ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not a "Retry until something" kind of thing, but replaying same query while switching to the other datasource
@jotak - it doesn't look like NETOBSERV-1652 is fixed and I see other issue with topology view where filters are not remembered take a look at recording .
|
Oh sure, that's because I didn't implement datasource memorization, I fixed NETOBSERV-1652 the other way around ie. re-playing the query to Loki when I get a 401 or 403 error from Prometheus, which should be good to fix the UX issue. What's the other settings that are not memorized? I'm not sure if every settings have to be memorized I think it's on purpose to keep only the one we consider more important... perhaps that will be something to check with @jpinsonneau when he's back |
/lgtm |
/test plugin-cypress |
it's the filters, in recording at 0:19 seconds, when I come back to the page , the topology view shows all traffic including the infra traffic whereas there's already "app" filter added. I didn't mean to suggest to cache everything, but this seems like a regression bug. I think it only happens when Datasource is Loki or prometheus query returns 401/403 and loki is used as fallback.
|
/retest |
you're right, I confirm issue exists on /label qe-approved |
@jpinsonneau: This pull request references NETOBSERV-740 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jotak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Description
Add data source to query option dropdown
Dependencies
Based on #513
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.