-
Notifications
You must be signed in to change notification settings - Fork 162
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wsgi daemon mode #5
Comments
Hi Thatcher, I think I've seen both thread and process based aggregation in the official Prometheus Python library that's used here, so I think it might work, but I'll have to check. Thanks, |
I can definitely verify i can start my micro-service, run a handful of request, and when i continue to refresh my metrics endpoints i get different responses from each daemon, which makes sense. It looks like promethues_client added support for a multi process collectors as in the examples provided here: I think we could patch this by looking for
It looks like I can test this out just by passing my own registry to PrometheusMetrics. Ill let you know. |
Yup it worked! heres the gist: wsgy.py
in my flask app:
Thats it! You could add this as an introspected feature or just document the recipe for others. Thanks rycus86! Thatcher |
Awesome, thanks a lot! |
I was playing with this library and even with a single instance, when i hit the /metrics endpoint i get 3 different results. I am following the steps outlined in the conversation here.
So why "3" different responses ? And how to resolve the issue ? I am using 0.2.2 version of prometheus_flask_exporter |
@snegivulcan I discovered the same issue and realized the issue is related to some of the code in the prometheus_flask_exporter and I didnt have time to dig into it. I'd like to eventually figure out how to get it to play nicely with prometheus_flask_exporter. I can confirm that the issue does not occur if I use the prometheus client directly. Here is how i am using it for now in production:
|
Hi @snegivulcan and @thatcher , Have you had a look at the small example in https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/wsgi ? Thanks! |
I'm using the same code as @snegivulcan from his Flask example and I noticed from the comment headers that what's being exposed relates to one of the workers and the other belongs to the multiprocess metrics:
|
I found a fix from: https://github.com/jonashaag/prometheus-multiprocessing-example/blob/master/yourapp.py Where I define the following as a
|
Hi @jpds, Thanks for sharing your findings! The project you mentioned also links the multiprocessing section of the official Prometheus Python client library: https://github.com/prometheus/client_python#multiprocess-mode-gunicorn I'll try to have a look if this could be supported somehow better with this library. Thanks! |
Hi, There's a new release with some multiprocessing changes from @elephantum - https://pypi.org/project/prometheus-flask-exporter/0.2.3/ It would be great if you could give it a look if the previous problem still happens using this version. Thanks! |
I've added some more multiprocessing changes to version 0.5.0, mainly targeted for Gunicorn and uWSGI, but should work in a generic way - see the README for more info. |
@rycus86
But I do get them when I don't try to use multiprocess module (albeit with different values for each process of course) |
Hi @Torquerrr Let me know if you still think it's not working, things may have changed since the last release. |
@rycus86 I was able to narrow the problem down to |
Have you tried the multiprocess support classes from this library?
That should take care of setting up the multiprocessing-ready registries and such, you can see it in the example I linked above. |
@rycus86 Yes, now I am running it on the same host as main Flask app (host explicitly specified) and I get
|
@Torquerrr see the example I linked above, the uwsgi metrics object needs an explicit call to enable the metrics endpoint on a port:
|
@rycus86 I've added that call to |
@rycus86 Does this example imply that I also need to set my uwsgi config to |
I tested it like that, not a 100% sure if it's required or not. |
Also, I wanted to say, if you can provide a stripped down example, I'm happy to try and have a look if the library needs changing perhaps? |
@rycus86 Thank you for your help, I will try this approach and return with results :) |
@rycus86 By the way - I can't find examples of how to add custom metrics like Counter/Gauge, are there any? |
You can decorate your functions with the metrics helper, as shown in the readme: https://github.com/rycus86/prometheus_flask_exporter/blob/master/README.md
|
The |
@rycus86 |
Oh, so sorry I forgot to mention that! :/ Glad you managed to work it out! 👍 |
@rycus86 In fact I was using |
I run my stateless flask apps with mod_wsgi/apache using daemon mode like:
Which means when a request gets to the service it could be hitting 1 of 8 daemon processes each of which have their own memory in isolation of the others. Does the metrics endpoint store the prometheus data in a way that is shared across these daemons?
I can create some tests to verify if thats the case or not, just curious if the answer is already known.
Thanks,
Thatcher
The text was updated successfully, but these errors were encountered: