Error in psycopg2 installation during pip install -r requirements-dev.txt

[prutskov]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): lnx
• Modin version (modin.__version__): 95d71b2
• Python version: 3.8.12
• Code we can use to reproduce:

$ pip install -r requirements-dev.txt

Describe the problem

Developer environment couldn't be installed via pip. Errors during installation of package psycopg2 is occured:

Collecting psycopg2
  Using cached psycopg2-2.9.3.tar.gz (380 kB)
    ERROR: Command errored out with exit status 1:
     command: /localdisk/aprutsko/miniconda3/envs/modin-unidist/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-brz5no5r/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-brz5no5r/psycopg2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vmc0b6zu
         cwd: /tmp/pip-install-brz5no5r/psycopg2/
    Complete output (23 lines):
    running egg_info
    creating /tmp/pip-pip-egg-info-vmc0b6zu/psycopg2.egg-info
    writing /tmp/pip-pip-egg-info-vmc0b6zu/psycopg2.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-vmc0b6zu/psycopg2.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-vmc0b6zu/psycopg2.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-vmc0b6zu/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Temporary solution: install the package via conda: conda install psycopg2

[YarShev]

There is no a build of psycopg2 for linux on PyPI so the error is occurred. Where does psycopg2 come from?

[YarShev]

I see, we have it in requirements-dev.txt file. We should install psycopg2-binary, but not psycopg2. Could you create a PR resolving this?

[YarShev]

Although, no. psycopg advises to use the source distribution for production use.

[YarShev]

I think, in case of using requirements-dev.txt we should install psycopg2 either with conda or with pip as psycopg suggests in this section https://www.psycopg.org/docs/install.html.

[YarShev]

@mvashishtha, IIRC you added this dependency. Could you investigate the issue and find the better solution?

[Garra1980]

Probably, could be linked with #3615

[mvashishtha]

@YarShev After looking around online, it looks like there are three solutions:

1. pip install psycopg2-binary as @YarShev suggested.
2. conda install psycopg2 as @prutskov suggests
3. Install postgresql-devel outside of pip via pip install libpq-dev on Debian/Ubuntu, or via pip install libpq-devel on Centos/Fedora/Cygwin/Babun. This worked for me just now on Ubuntu 20.04.4 LTS.

Because Modin is only using this module in testing and not in production, I think 1) is the best solution.

[YarShev]

Modin itself can be (it is already) used in production. A company that uses Modin in production may install psycopg2 and perform an operation, which depends on the package, but we will be testing Modin only with psycopg2-binary as you are suggesting. There might be inconsistency. The tests may pass with psycopg2-binary but may fail with psycopg2. So I am not sure the 1) is the best solution.

cc @devin-petersohn

[vnlitvinov]

I guess the error happens if a user does not have postgreSQL itself installed, so we should add a note on it being required I think.

[anmyachev]

psycopg2 is no longer used in requirements-dev.txt file. Can we close this issue? Or, on the contrary, does it mean adding this library to the testing environment?