Requests sent over a proxy aren't cached the same way?

[nk9]

I started a large batch of requests in a terminal where I was using HTTP Toolkit to examine web traffic. After a time, I decided to close HTTP Toolkit; that killed the proxy, and the terminal was no longer able to be used for HTTP fetching. So I exited that terminal, created a new one, and ran my script from the top.

However, in the new terminal, the script no longer found the cached responses. 🤔 It started re-fetching everything. When I turned on the HTTP Toolkit proxy again, the cache immediately worked.

I was surprised by this, because I was expecting only elements of the request that are known before sending to be used to calculate the cache key. But it seems like the network environment has something to do with generating the cache key. I presume this is where the cache key is generated.

So my questions:

1. Is this a bug?
2. Why is the proxy environment relevant?
3. How should I modify my code so that the proxy environment is irrelevant?
4. Is there any way to salvage the thousands of requests I've made already? I COULD keep HTTP Toolkit running forever, but I'd really prefer not to.

[nk9]

Here's some code:

from requests import Session
from requests_cache import CacheMixin
from requests_ratelimiter import LimiterMixin

class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
    pass

rate = RequestRate(1, Duration.SECOND * 4)
limiter = Limiter(rate)

self.session = CachedLimiterSession(
        cache_name="http_cache",
        allowable_codes=[200],
        allowable_methods=["GET", "HEAD", "POST"],
        limiter=limiter,
)

[nk9]

Alright, so I put a print in the cache_key method and there IS a difference between the lists used to generate the two cache keys.

Without the proxy:

['GET', 'https://apple.com', '', True, 'SerializerPipeline(name=pickle, n_stages=2)']

With the proxy:

['GET', 'https://apple.com', '', '/Users/nick/Library/Preferences/httptoolkit/ca.pem', 'SerializerPipeline(name=pickle, n_stages=2)']

It seems this is the verify parameter. And when I add it manually without the proxy running, the cache works great… but new requests fail:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)

So it looks like I'm stuck with the proxy, at least in order to access the requests I've already made. Let me know if I've missed an option.

[nk9]

It turns out, there's a method for that! I was able to get the keys to take account of the new, proxy-less envrionment by just adding this line after creating the session:

self.session.cache.recreate_keys()

Sorry for the spam, but maybe it will help someone in the future. And thank you so much for this incredibly useful tool!

[JWCook]

Good, I'm glad that worked for you.

The verify parameter is included in the cache key because a server can potentially return different response content with or without SSL (#70, #209). But providing an explicit cert path shouldn't return a different response compared to verify=True (as long as the cert is valid), so I could update requests-cache to treat verify=(cert path) the same as verify=True. Any thoughts on that?

[nk9]

That makes sense to me, but I'm not a TLS expert. It sounds like it would be a breaking change, though pretty easy to recover from. I expect relatively few people are running their cache sessions through proxies, so maybe it's not that big a deal?

I think I expected things to be set up this way, so if you can find a way to introduce that behavior in a low-impact way, I think it would be a good change for users in the future.

[JWCook]

In general, the network environment isn't expected to affect cache keys, but I'm not specifically familiar with HTTP Toolkit. Does it rewrite URLs, add/modify request parameters or cache headers, disable SSL, or do anything else to modify requests?

It should be possible to reuse your previously cached requests, either by manually modifying cache data or changing session settings. I could give an example if you can find what exactly changed. Can you show me complete info for a cached response with and without HTTP Toolkit (with any sensitive info redacted)? There's info in the docs on getting request info from the cache.