FileCache: raw field serialization

[farwayer]

Pickle don't serialize raw field of Response for some reason. And after recover Response from cache this field is not valid.

adapter.py

 def build_response(self, request, response):
...
    elif request.method == 'GET':
            if response.status == 304:
                # We must have sent an ETag request. This could mean
                # that we've been expired already or that we simply
                # have an etag. In either case, we want to try and
                # update the cache if that is the case.
                resp = self.controller.update_cached_response(
                    request, response
                )

controller.py

def update_cached_response(self, request, response):
        """On a 304 we will get a new set of headers that we want to
        update our cached value with, assuming we have one.

        This should only ever be called when we've sent an ETag and
        gotten a 304 as the response.
        """
        cache_url = self.cache_url(request.url)

        resp = self.cache.get(cache_url)

Then here we catch exception:
sessions.py from request

def send(self, request, **kwargs):
...
       # Response manipulation hooks
        r = dispatch_hook('response', hooks, r, **kwargs)
...
        extract_cookies_to_jar(self.cookies, request, r.raw)

File "/mnt/data/work/projects/autocompas/parser.py", line 79, in check_car
    r = session.get(image)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 498, in send
    extract_cookies_to_jar(self.cookies, request, r.raw)
AttributeError: 'Response' object has no attribute 'raw'

[ionrock]

@farwayer This is actually a known issue. I realize now that I should have updated the documentation to reflect this. My apologies.

The reason I haven't tried to fix it is because the raw attribute is a file like object of the response that is used when streaming. It comes directly from urllib3 (if I remember correctly) and mainly there as an escape hatch for user iterating over a normal streaming response may not be enough. I fixed this because when a response comes from the cache, the entire response body already exists in full.

With that in mind, I realize now that the pickling support in requests doesn't include a .raw attribute set to None as it is in my monkeypatch for requests within cache control. So this is actually a bug in requests.

If you want to file a bug in the requests issue tracker that would be best. I'm happy to submit a pull request to requests to fix it.

In the meantime, you should be able to patch requests by importing cachecontrol.patch_requests.

Thanks for submitting the issue!

[farwayer]

@ionrock thx for fast reply. Now I understand that problem is with requests. I am using 2.2.1 requests version however I tried to use monkeypatch forcibly but without success.

I got this exception first time

File "/mnt/data/work/projects/autocompas/parser.py", line 110, in check_car
    r = session.get(image)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/adapter.py", line 30, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/adapters.py", line 391, in send
    r = self.build_response(request, resp)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/adapter.py", line 56, in build_response
    request, response
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/controller.py", line 242, in update_cached_response
    self.cache.set(cache_url, resp)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/caches/file_cache.py", line 39, in set
    dump(value, fh)
  File "/usr/lib64/python2.7/pickle.py", line 1370, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib64/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib64/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/copy_reg.py", line 84, in _reduce_ex
    dict = getstate()
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/patch_requests.py", line 21, in response_getstate
    if not self._content_consumed:
AttributeError: 'Response' object has no attribute '_content_consumed'

and next

File "/mnt/data/work/projects/autocompas/parser.py", line 110, in check_car
    r = session.get(image)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/adapter.py", line 21, in send
    request.url, request.headers
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/controller.py", line 83, in cached_request
    in_cache = self.cache.get(cache_url)
  File "/mnt/data/work/projects/autocompas/env/lib/python2.7/site-packages/cachecontrol/caches/file_cache.py", line 32, in get
    return load(codecs.open(name, 'rb'))
  File "/usr/lib64/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib64/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib64/python2.7/pickle.py", line 880, in load_eof
    raise EOFError

[ionrock]

@farwayer Ah, ok. Sorry. I hoped that might work for you.

The issue has been fixed in requests. I'll try to find a fix for this in the meantime.

[farwayer]

Hi! We still have a chance to get exception if 304 response returned from server.

[ionrock]

@farwayer Great catch! Do you mind fixing the spelling error of "fiel" in the comment? Thanks!