Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLite cache: Use requests_cache.CachedSession for better concurrency #130

Merged
merged 1 commit into from
Mar 31, 2024
Merged

SQLite cache: Use requests_cache.CachedSession for better concurrency #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ in progress
- Add subcommand ``explore permissions``. Thanks, @meyerder.
- Added support for Python 3.12
- Removed support for Python 3.7
- SQLite cache: Use ``requests_cache.CachedSession`` for better concurrency
behaviour. Thanks, @JensRichnow and @JWCook.

2024-03-07 0.18.0
=================
Expand Down
3 changes: 3 additions & 0 deletions grafana_wtf/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@

__appname__ = "grafana-wtf"
__version__ = "0.18.0"

# Amalgamate `requests` to `niquests`.
import grafana_wtf.compat
12 changes: 12 additions & 0 deletions grafana_wtf/compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from sys import modules

import niquests
import urllib3

# Amalgamate the module namespace to make all modules aiming
# to use `requests`, in fact use `niquests` instead.
modules["requests"] = niquests
modules["requests.adapters"] = niquests.adapters
modules["requests.sessions"] = niquests.sessions
modules["requests.exceptions"] = niquests.exceptions
modules["requests.packages.urllib3"] = urllib3
Comment on lines +1 to +12
Copy link
Contributor Author

@amotl amotl Mar 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ousret: Could this monkeypatching / amalgamation code be also provided by Niquests itself, as a convenience shorthand? To be invoked, for example, like:

import niquests
niquests.patch_all()

Can't patch itself while it's already loaded, hm? Can it? We barely remember it, but isn't it the same with Gevent, like that... ?

import gevent
gevent.patch_all()

Edit: Ah, slightly similar, it is:

from gevent import monkey
monkey.patch_all()

-- https://www.gevent.org/api/gevent.monkey.html

wdyt?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this isn't the way to go. We should not tamper with sys.module outside of a development environment.

Try this instead

import requests_cache
import niquests


class CacheSession(requests_cache.session.CacheMixin, niquests.Session):
    ...


if __name__ == "__main__":

    s = CacheSession()

    for i in range(60):
        r = s.get('https://httpbin.org/delay/1')

While its still not ideal, it should work more accordingly.
I am thinking about forking popular extensions and see how we can ease this.

Copy link
Contributor Author

@amotl amotl Mar 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your advise. We will submit a patch to work with requests-cache in the way you are suggesting it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converged that into a patch, thanks again.

18 changes: 13 additions & 5 deletions grafana_wtf/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from grafana_client.api import GrafanaApi
from grafana_client.client import GrafanaClientError, GrafanaUnauthorizedError
from munch import Munch, munchify
from requests_cache import CachedSession
from tqdm import tqdm
from tqdm.contrib.logging import tqdm_logging_redirect
from urllib3.exceptions import InsecureRequestWarning
Expand All @@ -34,6 +35,8 @@


class GrafanaEngine:
session = niquests.Session()

def __init__(self, grafana_url, grafana_token):
self.grafana_url = grafana_url
self.grafana_token = grafana_token
Expand All @@ -56,9 +59,11 @@ def enable_cache(self, expire_after=60, drop_cache=False):
log.info(f"Response cache will expire immediately (expire_after=0)")
else:
log.info(f"Response cache will expire after {expire_after} seconds")
requests_cache.install_cache(cache_name=__appname__, expire_after=expire_after, use_cache_dir=True)
cache_database_file = requests_cache.get_cache().db_path
log.info(f"Response cache database location is {cache_database_file}")

self.session = CachedSession(cache_name=__appname__, expire_after=expire_after, use_cache_dir=True)

cache = self.session.cache
log.info(f"Response cache database location is: {cache.db_path}")
if drop_cache:
log.info("Dropping response cache")
self.clear_cache()
Expand All @@ -72,8 +77,8 @@ def clear_cache(self):
def enable_concurrency(self, concurrency):
self.concurrency = concurrency

@staticmethod
def grafana_client_factory(grafana_url, grafana_token=None):
@classmethod
def grafana_client_factory(cls, grafana_url, grafana_token=None):
url = urlparse(grafana_url)

# Grafana API Key auth
Expand All @@ -97,6 +102,9 @@ def grafana_client_factory(grafana_url, grafana_token=None):
url_path_prefix=url.path.lstrip("/"),
verify=verify,
)
if cls.session:
cls.session.headers["User-Agent"] = grafana.client.user_agent
grafana.client.s = cls.session

return grafana

Expand Down
Empty file added tests/__init__.py
View file
Open in desktop
Empty file.