Zip32/64 Issue for Gigantic Packages

[Miserlou]

Reported via Slack:

robinelvin [10:30 AM] 
Hi. I've just started with Zappa and trying to deploy an existing Django app.

[10:30]  
When packaging I get an error: LargeZipFile: Files count would require ZIP64 extensions

[10:31]  
Is this a bug or have I missed something?

mcrowson [10:34 AM] 
try `zappa package`

[10:34]  
how big is the zip it gives you

robinelvin [10:36 AM] 
It falls over with the same message. The zip file it wrote is 233220933 bytes which looks like the 32 bit limit?

mcrowson [10:36 AM] 
so > 233MB?

robinelvin [10:37 AM] 
Looks like it

mcrowson [10:37 AM] 
it might be possible with the slim_handler, but there would need to be a change to the zip like the warning says

[10:37]  
might want to make an issue for it

robinelvin [10:38 AM] 
I have the slim_handler set to true already

mcrowson [10:39 AM] 
right, but the zip change is what needs the issue

[10:39]  
might be a super simple PR if you want to take a stab at it

robinelvin [10:40 AM] 
Ok, will have a look at how to do the PR

[10:41]  
Thanks

mcrowson [10:41 AM] 
my 2 pennies: https://github.com/Miserlou/Zappa/blob/master/zappa/zappa.py#L500 (edited)

[10:41]  
https://docs.python.org/2/library/zipfile.html#zipfile-objects

[10:41]  
could just be allowZip64=True

[10:41]  
as a parameter when instantiating the ZipFile

[10:42]  
not sure what would need to change on the unzip side though, that'd be the place to look for possible issues

[10:46]  
maybe just here? https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L166

[mcrowson]

Not needed with #1037 ?

[antwan]

Experiencing the same issue here with a large project. Please add the allowZip64=True setting when creating the zip file.

[mcrowson]

Slim handler projects don't use the zip compression method. So is this just for <50M projects?

[antwan]

Yes I just did it with slim_handler, it turns out tarball compression works better. Maybe it's worth adding a note "You should consider turning slim_handler on" instead of the nasty exception:

Traceback (most recent call last):
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/cli.py", line 2610, in handle
    sys.exit(cli.handle())
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/cli.py", line 505, in handle
    self.dispatch_command(self.command, stage)
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/cli.py", line 539, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/cli.py", line 709, in deploy
    self.create_package()
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/cli.py", line 2171, in create_package
    disable_progress=self.disable_progress
  File "/Users/antwan/Projects/myproject/venv/lib/python2.7/site-packages/zappa/core.py", line 681, in create_lambda_zip
    archivef.writestr(zipi, f.read(), compression_method)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 1234, in writestr
    self._writecheck(zinfo)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 1114, in _writecheck
    " would require ZIP64 extensions")
LargeZipFile: Files count would require ZIP64 extensions

[mcrowson]

So the total project size is <50M, but you had a few individual files that were very large? Mind giving some stats/details?

[antwan]

To clarify I had too many files for the ZIP (slim_handler = False) but it's working well with a tarball (slim_handler = True).