Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

is it possible to specify order of installation of packages in the requirements.txt? #294

Closed
m-kkk opened this issue Nov 30, 2018 · 10 comments
Closed

is it possible to specify order of installation of packages in the requirements.txt? #294

m-kkk opened this issue Nov 30, 2018 · 10 comments

Comments

@m-kkk
Copy link

m-kkk commented Nov 30, 2018

my requirements.txt includes fastparquet and numpy, and no matter what order I put them in the file, it seems like the plugin always look for fastparquet before numpy. However, to run setup.py of fastparquet, it requires numpy to be installed first.
I wonder is there any way to change the order of the installation, and if it does, will it solve the problem?

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/kmao/git/parquet_to_json/requirements.txt in /Users/kmao/git/parquet_to_json/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/kmao/git/parquet_to_json/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Collecting fastparquet (from -r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/0d/1f/df5c0159dc0f96b362e8f00d9cf912a7343fa1ec8114aec8ae39d4de174b/fastparquet-0.2.0.tar.gz (147kB)
Collecting numpy (from -r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/ff/7f/9d804d2348471c67a7d8b5f84f9bc59fd1cefa148986f2b74552f8573555/numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)
Collecting pandas (from -r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/e1/d8/feeb346d41f181e83fba45224ab14a8d8af019b48af742e047f3845d8cff/pandas-0.23.4-cp36-cp36m-manylinux1_x86_64.whl (8.9MB)
Collecting requests (from -r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/ff/17/5cbb026005115301a8fb2f9b0e3e8d32313142fe8b617070e7baad20554f/requests-2.20.1-py2.py3-none-any.whl (57kB)
Collecting numba>=0.28 (from fastparquet->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/31/55/938f0023a4f37fe24460d46846670aba8170a6b736f1693293e710d4a6d0/numba-0.41.0-cp36-cp36m-manylinux1_x86_64.whl (3.2MB)
Collecting thrift>=0.11.0 (from fastparquet->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)
Collecting six (from fastparquet->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting pytest-runner (from fastparquet->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/72/a4/d7a5738a3096f22a98bec1609e237b250ebff04e5ea2930305d485337263/pytest_runner-4.2-py2.py3-none-any.whl
Collecting pytz>=2011k (from pandas->-r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
Collecting python-dateutil>=2.5.0 (from pandas->-r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting certifi>=2017.4.17 (from requests->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
Collecting idna<2.8,>=2.5 (from requests->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
Collecting urllib3<1.25,>=1.21.1 (from requests->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting llvmlite>=0.26.0dev0 (from numba>=0.28->fastparquet->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/a2/60/d22966c97a47687ac1cc57c2e756380897c264f1ce40780105d7dbcd9564/llvmlite-0.26.0-cp36-cp36m-manylinux1_x86_64.whl (16.1MB)
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, llvmlite, numba, thrift, pytest-runner, fastparquet, chardet, certifi, idna, urllib3, requests
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
  Running setup.py install for fastparquet: started
    Running setup.py install for fastparquet: finished with status 'error'
    Complete output from command /var/lang/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0bw5vusg/fastparquet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-f4krd8kg-record/install-record.txt --single-version-externally-managed --compile --home=/tmp/tmprwoclka4:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/dataframe.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/__init__.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/util.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/core.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/encoding.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/schema.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/converted_types.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/compression.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/thrift_structures.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/writer.py -> build/lib.linux-x86_64-3.6/fastparquet
    copying fastparquet/api.py -> build/lib.linux-x86_64-3.6/fastparquet
    running egg_info
    writing fastparquet.egg-info/PKG-INFO
    writing dependency_links to fastparquet.egg-info/dependency_links.txt
    writing requirements to fastparquet.egg-info/requires.txt
    writing top-level names to fastparquet.egg-info/top_level.txt
    warning: manifest_maker: standard file '-c' not found

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-0bw5vusg/fastparquet/setup.py", line 102, in <module>
        **extra
      File "/var/lang/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/var/lang/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/var/lang/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/var/lang/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/var/lang/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/var/lang/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/var/lang/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/var/lang/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/var/lang/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/build_py.py", line 53, in run
        self.build_package_data()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/build_py.py", line 118, in build_package_data
        for package, src_dir, build_dir, filenames in self.data_files:
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/build_py.py", line 66, in __getattr__
        self.data_files = self._get_data_files()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/build_py.py", line 82, in _get_data_files
        self.analyze_manifest()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/build_py.py", line 138, in analyze_manifest
        self.run_command('egg_info')
      File "/var/lang/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/var/lang/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 279, in run
        self.find_sources()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
        mm.run()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 533, in run
        self.add_defaults()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 562, in add_defaults
        sdist.add_defaults(self)
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/var/lang/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/var/lang/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/var/lang/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-build-0bw5vusg/fastparquet/setup.py", line 22, in finalize_options
        import numpy
    ModuleNotFoundError: No module named 'numpy'
@m-kkk m-kkk changed the title is it possible to specify order of packing installation in requirements.txt? is it possible to specify order of installation of packages in the requirements.txt? Nov 30, 2018
@AndrewFarley
Copy link
Contributor

There is no way that I know of currently but I think we could adopt a feature similar to nodeploy to hint to keep them in a specific order. Eg:

numpy #no-sort
fastparquet

Thoughts? Could also be named #first or something.

@dschep
Copy link
Contributor

dschep commented Nov 30, 2018

fastparquet specifies numpy as a dependency. Have you tried leaving numpy out of your requirements.txt?

@bsamuel-ui
Copy link
Contributor

bsamuel-ui commented Nov 30, 2018 •
edited
Loading

I tried these steps:

virtualenv v
. v/bin/activate
pip install numpy fastparquet

And that failed. But (blowing away pip's cache and) replacing the last step with pip install numpy; pip install fastparquet works.

That's supposed to be fixed on their end, maybe it's a regression?

Otherwise the solution for the plugin is to run a sequence of pip commands, which is starting to get pretty klugey. Another solution is #225, was to be able to run arbitrary commands pre and post commands docker. That's still klugey, but it would be far more flexible for one-off cases like this.

@m-kkk
Copy link
Author

m-kkk commented Nov 30, 2018

I solve the installation problem by adding pip install numpy in the Dockimage.

Installing collected packages: idna, certifi, urllib3, chardet, requests, pytz, numpy, six, python-dateutil, pandas, llvmlite, numba, thrift, pytest-runner, fastparquet
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
  Running setup.py install for fastparquet: started
    Running setup.py install for fastparquet: finished with status 'done'
Successfully installed certifi-2018.11.29 chardet-3.0.4 fastparquet-0.2.0 idna-2.7 llvmlite-0.26.0 numba-0.41.0 numpy-1.15.4 pandas-0.23.4 pytest-runner-4.2 python-dateutil-2.7.5 pytz-2018.7 requests-2.20.1 six-1.11.0 thrift-0.11.0 urllib3-1.24.1

But the lambda function still has this error when I run it

module initialization error
[Errno 38] Function not implemented

I wonder if fastparquet can't work in lambda.

@bsamuel-ui
Copy link
Contributor

That sounds like it's trying to do IPC, and Lambda greatly limits what IPC you can do. The multiprocessing module, for instance, relies on semaphores and shared memory and generally won't work. This will fail with the same error:

from multiprocessing import Semaphore

def lambda_handler(event, context):
    s = Semaphore(6)
    return {}

@m-kkk
Copy link
Author

m-kkk commented Nov 30, 2018

Thanks, close the issue now!

@m-kkk m-kkk closed this as completed Nov 30, 2018
@bsamuel-ui
Copy link
Contributor

If you're looking for next steps, probably ECS and Fargate would be the route to go, possibly via containerless. (Which I need to try some time, myself...)

@dschep
Copy link
Contributor

dschep commented Nov 30, 2018

Ooh. Neat plugin @bsamuel-ui. BTW, keep your eyes peeled for sls v2 pre-release early next year. one of the things it'll feature is near-frictionless ability to move a function from running on lamba to running on fargate 🙂

@bertoneone
Copy link

I solve the installation problem by adding pip install numpy in the Dockimage.

Installing collected packages: idna, certifi, urllib3, chardet, requests, pytz, numpy, six, python-dateutil, pandas, llvmlite, numba, thrift, pytest-runner, fastparquet
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
  Running setup.py install for fastparquet: started
    Running setup.py install for fastparquet: finished with status 'done'
Successfully installed certifi-2018.11.29 chardet-3.0.4 fastparquet-0.2.0 idna-2.7 llvmlite-0.26.0 numba-0.41.0 numpy-1.15.4 pandas-0.23.4 pytest-runner-4.2 python-dateutil-2.7.5 pytz-2018.7 requests-2.20.1 six-1.11.0 thrift-0.11.0 urllib3-1.24.1

But the lambda function still has this error when I run it

module initialization error
[Errno 38] Function not implemented

I wonder if fastparquet can't work in lambda.

I am experiencing the same issue with numpy. Whatever order I use in the requirements.txt, the order gets changed when the requirements.txt is generated within the .serverless folder.

Could you detail how you fixed it? You added that part in the DockImage? (might be a naive question but I just started experimenting with serverless framework)

@m-kkk
Copy link
Author

m-kkk commented Jan 4, 2019 •
edited
Loading

I solve the installation problem by adding pip install numpy in the Dockimage.

Installing collected packages: idna, certifi, urllib3, chardet, requests, pytz, numpy, six, python-dateutil, pandas, llvmlite, numba, thrift, pytest-runner, fastparquet
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
  Running setup.py install for fastparquet: started
    Running setup.py install for fastparquet: finished with status 'done'
Successfully installed certifi-2018.11.29 chardet-3.0.4 fastparquet-0.2.0 idna-2.7 llvmlite-0.26.0 numba-0.41.0 numpy-1.15.4 pandas-0.23.4 pytest-runner-4.2 python-dateutil-2.7.5 pytz-2018.7 requests-2.20.1 six-1.11.0 thrift-0.11.0 urllib3-1.24.1

But the lambda function still has this error when I run it

module initialization error
[Errno 38] Function not implemented

I wonder if fastparquet can't work in lambda.

I am experiencing the same issue with numpy. Whatever order I use in the requirements.txt, the order gets changed when the requirements.txt is generated within the .serverless folder.

Could you detail how you fixed it? You added that part in the DockImage? (might be a naive question but I just started experimenting with serverless framework)

you can create your Dockerfile copy the source Dockerfile from https://github.com/lambci/docker-lambda/blob/master/python3.6/build/Dockerfile
and add pip install -U numpy --no-cache-dir in it, then add the Dockerfile in the serverless.yml like this.

  pythonRequirements:
    dockerizePip: true
    dockerFile: /home/Dockerfile

or if you just want to reorder the package in the requirements.txt, you can find this linefilteredRequirements.sort(); // Sort remaining alphabetically that does the sort at your_serverless_path/node_modules/serverless-python-requirements/lib/pip.js and just comment that out, it works fine for me without sorting it alphabetically.

@AndrewFarley AndrewFarley mentioned this issue Feb 28, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@AndrewFarley @dschep @bertoneone @bsamuel-ui @m-kkk