Support conda environments

[mathieu1]

This is a proposition to support conda, which is a superset of virtualenv but which also deals with the difficulties of binary dependencies. This would provide an alternative to the self-maintained lambda-packages. Zappa would benefit from the large collection of community-contibuted and/or enterprise-backed conda packages with no overhead. AWS lambda-specific packages could also be contributed back to the conda community, should this be necessary.

To see conda in action:

• Get conda (either through pip, or using the bootstrap Miniconda installer), making sure conda is on your path.
• Create an environment.yml file with the following content:

name:zappa_dev_env
channels:
  - conda-forge
  - defaults
dependencies:
  - python=2.7
  - flask
  - pandas
  - numpy
  - nomkl
  - pip
  - pip:
    - zappa

• Create the conda environment with conda env create
• Activate the environment with source activate zappa_dev_env
• zappa deploy dev

Currently, this should only be used by novice conda users in a linux-64bit architecture (as warned by the checks in the code). There are various ways to extend this to all architectures, although there will always be limitations to doing so as dependencies are sometimes platform-specific.

Further improvements could include:

• Improve the warning display
• Add tests
• Document how to use conda with zappa
• Mechanism(s) to ship conda packages that use the AWS lambda architecture when developping on a different architecture (similar to the current way of using lambda-packages).
• conda package zappa and put it on conda-forge
• look into packages with placeholders (/opt/anaconda1anaconda2anaconda3)

and more (to be completed), such as the weirder...

• Test if shipping python 3 or R to AWS lambda using conda is feasible

[coveralls]

Coverage decreased (-0.8%) to 76.226% when pulling 8060140 on mathieu1:conda_support into 1202044 on Miserlou:master.

[Miserlou]

Hi @mathieu1! Thanks for this PR, this is an interesting idea!

There are a few issues here that I'd like to address before I can merge this in.

Firstly - I simply don't quite understand the point of supporting Conda here. I'm sure that I could be convinced if it was explained to me, but I literally don't understand why this is beneficial.

Secondly - should this be accompanied with a similar PR to the lambda-packages project? Do I also need to create an account somewhere and publish Zappa to a Conda repository?

Thirdly - Since this is a new feature, it will need a test in order to be merged, as well as any associated documentation.

(Also, since this will likely have a few more commits and discussion associated with it, please submit this pull request to the 'dev' branch rather than master.)

Thanks!
Rich

[mathieu1]

Thanks for the feedback!

I will open soon an updated pull request targeted at the dev branch. Each of your points requires quite some discussion:

1. Let me motivate a bit more why conda support is useful: lambda-packages and conda basically do the same thing: providing pre-compiled binaries (in this case of C dependencies of python libraries, in order to ship to lambda. Conda actually goes well beyond that). conda is useful in at least two ways:
   • conda supports a very long list of packages (and quickly increasing thanks to a thriving community). For example, with some tweaks to zappa (see upcoming PR), it is possible to successfully deploy an API which uses psycopg2, pandas (hence numpy), shapely (hence geos), pyproj (hence proj.4) without relying on lambda-packages, and without having to figure out how to compile any C library like geos or proj.4. I'm not claiming every conda package will work, but I expect many to work out of the box.
   • conda handles versions of packages and versions of dependencies (e.g. a specific version of pandas may require different binaries depending on which version of numpy it is compiled against). Solving these kind of problems can become very difficult, and sounds to me beyond the scope of lambda-packages.
2. I see conda and lambda-packages as two separate options which may coexist (as long as they are not used simultaneously). People relying only on virtualenv/pip may prefer the lambda-packages option rather than having to learn conda, while conda users will appreciate the advantages mentioned above. conda and lambda-packages may benefit from each other as follows: lambda-packages can copy the content of selected conda packages, and vice-versa, conda could benefit from new conda packages specifically compiled to be small (to fit on lambda). Those packages could be published under a new channel on anaconda.org, e.g. zappa, and could be used by people relying on conda. This can be discussed more in-depth later.
3. Regarding tests, I'm a bit unsure what they would look like, any suggestion is welcome! I see a bootstraping issue: from a test script within zappa, create a conda environment and install zappa in it. Then I'm not sure what to test: actually deploy to aws and see if importing a bunch of packages works there? Let's discuss it in the next PR.

[lsorber]

+1

[mathieu1]

Continued in #108