Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash for response larger than 4GB #238

Closed
uranusjr opened this issue Feb 4, 2021 · 2 comments
Closed

Crash for response larger than 4GB #238

uranusjr opened this issue Feb 4, 2021 · 2 comments

Comments

@uranusjr
Copy link

uranusjr commented Feb 4, 2021

Originally reported in pypa/pip#9549.

CacheControl crashes with ValueError: bytes is too large when trying to downloading a file larger than 4GB. Since this is carried over from msgpack’s limitation, it may be reasonable to skip trying to cache such a response (maybe with a warning), or provide e.g. a callback for the user code to rescure when this happens.

@RonnyPfannschmidt
Copy link

An alternative could be a cc=5 format which stores the request/response content lengths as integer in the msgpack message and puts the data contents into the data after message

This could also enable using optimization for retrieval of responses data to actual files as it could be passed from a file descriptor

@woodruffw
Copy link
Member

Doing some triage: I believe this has been fixed upstream via pypa/pip#11143, which uses SeparateBodyFileCache to work around msgpack's size limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants