Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Django)(MAC Big Sur 11.0.1) ImportError: dlopen #1216

Closed
awildjosh opened this issue Jan 4, 2021 · 89 comments
Closed

(Django)(MAC Big Sur 11.0.1) ImportError: dlopen #1216

awildjosh opened this issue Jan 4, 2021 · 89 comments

Comments

@awildjosh
Copy link

awildjosh commented Jan 4, 2021

Apologies in advance -- this is probably doubling up on #1200 and #1208, but I'm curious as to whether there's any workaround whatsoever (as I really would like to use postgresql with my project, but I'm too much of a newb to comprehend most of the issues I run into). I've tried pretty much every suggestion to be found through googling to no avail. The issue is as below:

  • Mac Big Sur (OS 11.0.1)
  • Python 3.9 (venv)

Installed pyscopg2 to the venv without any issues using:
pip install psycopg2

Meanwhile, Postgresql was successfully installed and the db I'd like to connect to is active (using pgadmin4). Django settings are configured accordingly.

However, when trying to access psycopg2 through the terminal, whether it be through:
python -c "import psycopg2" or python manage.py migrate, etc.

The below error occurs:
(venv) ...-MacBook-Air project % python -c "import psycopg2" Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): no suitable image found. Did find: /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture

Any help/insight would be greatly appreciated. Or are all of us who jumped ship to M1 doomed until Apple stop breaking things? :)

@bschollnick
Copy link

Second this. Unable to use natively under M1. I am forced to use under Intel Translation / Rosetta 2.

Psycopg3 also does not compile natively...

@dvarrazzo
Copy link
Member

Is there an M1 to test with?

@bschollnick
Copy link

I'm more than happy to volunteer my own time to test... I can either run whatever tests you need, or I can volunteer to allow you to remote in, etc.

@dvarrazzo
Copy link
Member

I would appreciate that, thank you.

Most likely rebuilding packages using the newer pypa infrastructure would just solve the issue.

Could you please checkout https://github.com/psycopg/psycopg2-wheels/ and check if the scripts/build_macos.sh produces working packages?

Thank you very much.

Ref. pypa/pip#9138

@bschollnick
Copy link

bschollnick commented Jan 11, 2021

As far as I can tell, with Rosetta 2, everything was fine (which seems correct, since I can install via PIP under Rosetta 2).

README.rst	appveyor.yml	data		psycopg2	scripts
benjamin@Nerv psycopg2-wheels-master % cd scripts
benjamin@Nerv scripts % build_libpq_macos.sh
configure: error: library 'crypto' is required for OpenSSL
benjamin@Nerv scripts % pip install crypto
Defaulting to user installation because normal site-packages is not writeable
Collecting crypto
  Downloading crypto-1.4.1-py2.py3-none-any.whl (18 kB)
Collecting Naked
  Downloading Naked-0.1.31-py2.py3-none-any.whl (590 kB)
     |████████████████████████████████| 590 kB 2.9 MB/s 
Requirement already satisfied: requests in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from Naked->crypto) (2.25.1)
Collecting pyyaml
  Using cached PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Requirement already satisfied: idna<3,>=2.5 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2020.12.5)
Collecting shellescape
  Downloading shellescape-3.8.1-py2.py3-none-any.whl (3.1 kB)
Installing collected packages: pyyaml, shellescape, Naked, crypto
  WARNING: The script naked is installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts crypto and decrypto are installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Naked-0.1.31 crypto-1.4.1 pyyaml-5.3.1 shellescape-3.8.1
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
benjamin@Nerv scripts % ls      
build-manylinux.sh		build_manylinux1_x86_64.sh
build_libpq.sh			build_manylinux2014_aarch64.sh
build_libpq_macos.sh		build_sdist.sh
build_macos.sh			configure_postgres.sh
build_manylinux1_i686.sh	upload_packages.sh
benjamin@Nerv scripts % build_macos.sh
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
	  ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
	  for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
	    /usr/bin/install -c -m 644 ./$file \
			    /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	  for file in Makevars; do \
	    rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	else \
	  : ; \
	fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2
benjamin@Nerv scripts % 

But at the end, it dies looking for Cellar? I don't know what that's about?

Trying next under Native M1.

@bschollnick
Copy link

First attempt:

(testing) benjamin@Nerv scripts % build_macos.sh      
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(threadlib.o) has no symbols
make[2]: *** [lib/libsed.a] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

But that may be due to previously running the build under Rosetta 2 in that folder structure.
I nuked it, and re-unarchived it, which resulted in:

+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
	  ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
	  for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
	    /usr/bin/install -c -m 644 ./$file \
			    /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	  for file in Makevars; do \
	    rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	else \
	  : ; \
	fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2

But a significantly faster built (Rosetta took about 2-3 minutes?, M1 about 20-30 seconds).

And ended with the seller / sed messages above.

@dvarrazzo
Copy link
Member

make install should run as sudo I think. This script is designed to run in Travis with a superuser account.

@bschollnick
Copy link

I'm trying to make sense out of the build_macos script?
And I'm failing hard. There are a lot of moving parts here, and I'm not making headway on why any of it is there...

And the more I look through the build script, it's heavily personalized, and is requiring virtual environments that I don't have configured, and home-brew which is not yet working reliably under the native M1.

I couldn't get the scripts as supplied to work under either Intel Mode, or M1 Native mode.

Instead I just grabbed the psycopg2 dist from GitHub, and ran:

psycopg2 % python3 setup.py sdist bdist_wheel

Which created the Intel and m1 native based wheels. I'm unclear on what all the futzing around in the other scripts is suppose to accomplish?

I haven't had a chance to test these yet...

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl.zip

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_x86_64.whl.zip

@dvarrazzo
Copy link
Member

The futzing is to create self-contained pacakges, containing the libpq. Without it, the wheels need some system libraries to run, and because on mac os there is no standard for installation, people complain that it's too difficult.

I'll try and re-run the packages build, maybe you can help me testing if the packages work alright.

@dvarrazzo
Copy link
Member

@bschollnick Do packages available here work? https://upload.psycopg.org/psycopg2-2.8.7.dev0/ Thank you!

@bschollnick
Copy link

Works fine under Intel Translation, but fails under M1 Native.

(testing) Nerv ~ % pip install /Volumes/4TB_Drive/sorted_downloads/2021-01-11/psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
ERROR: psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl is not a supported wheel on this platform.

python3 -c"from packaging import tags; print('\n'.join([str(t) for t in tags.sys_tags()]))" |head -5
Requirement already satisfied: packaging in ./venvs/testing/lib/python3.8/site-packages (20.8)
Requirement already satisfied: pyparsing>=2.0.2 in ./venvs/testing/lib/python3.8/site-packages (from packaging) (2.4.7)
cp38-cp38-macosx_11_0_arm64
cp38-cp38-macosx_11_0_universal2
cp38-abi3-macosx_11_0_arm64
cp38-abi3-macosx_11_0_universal2
cp38-none-macosx_11_0_arm64

@bschollnick
Copy link

Is there anything else that I can do to help?

@bschollnick
Copy link

bschollnick commented Jan 14, 2021

FYI, I meant to post this here. But I ended up reposting it back to the same Pillow Thread (too many windows open at once).

Okay, obviously I'm missing something here.

_

(testing) dist % pip install psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl
ERROR: psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl is not a supported wheel on this platform.
From a comment from Pillow:

I build Pillow and jpegsrc-6b from source before filing this issue and that didn't need any changes other than specifying "-arch arm64 -arch x86_64" when building the latter. That said, I haven't tested the intel part of the wheel yet (but I don't expect problems there, given past experience with the ppc to intel transition).

_

So I probably need to change the target build architecture, but I don't see where to do that... ?

@dvarrazzo
Copy link
Member

Hello @bschollnick

I think you should hear from the people who contributed to pypa/pip#9138

I have no details about the M1 architecture and the specific details to address to build packages there.

@awildjosh
Copy link
Author

awildjosh commented Jan 14, 2021

As a quick aside, thank you both very much for your efforts :^]

If anyone with more know-how would like to remote in on my machine to further troubleshoot, let me know

@hishamkaram
Copy link

version 2.8.1 works fine for me
Screen Shot 2021-01-15 at 10 13 16 PM

@Andrey-Yakovtsev
Copy link

version 2.8.1 works fine for me
Good for you, but I tried that and failed as well (((
As far as I understood from this thread the issue isn't resolved yet?

@kevbradwick
Copy link

kevbradwick commented Jan 15, 2021

I can confirm 2.8.6 (including psycopg2-binary) works for me.

➜  psycopg2-test pip install psycopg2
Collecting psycopg2
  Downloading psycopg2-2.8.6.tar.gz (383 kB)
     |████████████████████████████████| 383 kB 3.8 MB/s
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.8.6-cp39-cp39-macosx_10_9_universal2.whl size=226356 sha256=bf4bdc2b8d73783423eec6a5f094d5d18f8b178f83a67017db821fb1e7353e6a
  Stored in directory: /Users/kevinbradwick/Library/Caches/pip/wheels/a2/07/10/a9a82e72d50feb8d646acde6a88000bbf2ca0f82e41aea438a
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6

This is on a Macbook Air M1 running arm64.

@dvarrazzo
Copy link
Member

@kevbradwick yes, it works for you, but you are a good soul who has a compiler and libraries on your machine. Which I appreciate, were everyone like you...

This problem here is with people who expect a binary package to work. Every few years, OSX people come out with a different machine with all the cheese moved around. or missing keys. Why people buy this stuff is beyond my belief.

So, this ticket is about making the binary package work. psycopg2 code works, but "Downloading psycopg2-2.8.6.tar.gz" says that you have installed the package from source, not from binary (although you indeed requested the psycopg2-binary package - that package has a source fallback for reason that now I can't remember but it was necessary).

Solving this problem requires the OSX build script to be fixed and the package to be generated by Travis. Happy to receive a MR for it.

@bschollnick
Copy link

Okay, it's now installing via PIP... Which is great, but when Django attempts to access the database..

I am seeing an exception for Symbol not found (Symbol not found: _PQbackendPID)

I have rebuilt the virtual environment, just in case there had been some left-overs from the trials... But no change.

- Benjamin
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
    import psycopg2 as Database
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so

@bschollnick
Copy link

Wait a sec. Why is it calling something in python 3.9? I'm running stock MOSX 3.8. I'll have to check that home-brew hasn't messed my paths..

@bschollnick
Copy link

Okay, rebuild the virtual environment again, and now it's showing the right python version... But same issue.

Successfully built psycopg2-binary
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.8.6
(quickbbs) benjamin@Nerv quickbbs % server4
Running on nerv.local
Debug is  True
Running on nerv.local
Debug is  True
Watching for file changes with StatReloader
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
    import psycopg2 as Database
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so

@kevbradwick
Copy link

how did you install postgres? did you compile it manually?

@dvarrazzo
Copy link
Member

Note that you don't need to compile postgres in order to build psycopg2, but you may need to compile the libpq, the postgres client-side library. psycopg2 does it in the scripts/build_libpq_macos.sh (in the psycopg2-wheels project).

@bschollnick
Copy link

bschollnick commented Jan 15, 2021 via email

@bschollnick
Copy link

bschollnick commented Jan 15, 2021 via email

@djch
Copy link

djch commented Jan 16, 2021

@bschollnick Exact same problem here per #1208

I'm using python@3.9 from homebrew, though

@dvarrazzo
Copy link
Member

@bschollnick

That’s not binary? That’s the source. Isn’t psycopg2-binary is the binary version?

For example:

(quickbbs) benjamin@Nerv quickbbs % pip install psycopg2
Collecting psycopg2
Using cached psycopg2-2.8.6.tar.gz (383 kB)

This ^ says that the package installed is the source one. The packages are found from this list:

https://pypi.org/project/psycopg2-binary/#files

Your pip run identifies which package is the most suitable for your system. For some reasons, which might be incompatibility with your system but also an out-of-date pip, which doesn't know enough of it, the macos package is not chosen. Because there is a source package too in the list (the .tar.gz one), the latter is selected.

It sounds silly that there is a source choice too in the binary package... Initially there wasn't, but that created more problems. I can't remember exactly, but they were of the tone that if someone has psycopg2-binary in a requirements.txt file, and tries to install the system e.g. in a python-alpine docker (which is not compatible with wheel packages), it would have resulted impossible, whereas now it is possible as long as they have the build tools too.

However, if you pip install psycopg2-binary and you get the .tar.gz package, even if your installation works alrigh I would say that the binary package is not doing its job.

@kevbradwick
Copy link

kevbradwick commented Jan 16, 2021

Postgres is working fine. I’m using PostgresApp… And Psycopg2 is working fine in Intel mode, as well. - Benjamin

could it be that the binary needs some shared dynamic library (also arm64) but the one you have installed is via Postgress.app which is x86?

@dvarrazzo
Copy link
Member

Yes: It needs the libraries built by scripts/build_libpq_macos.sh

@FedericoGregori
Copy link

FedericoGregori commented Aug 8, 2021

I fixed the same problem in Mac BigSur 11.5.1 M1 following the tips of this thread in the next order:

$ pip3 uninstall psycopg2
$ brew install libpq --build-from-source
export LDFLAGS="-L/opt/homebrew/opt/libpq/lib" # Add to .zshrc
$ pip3 install psycopg2 --no-cache-dir

Thanks to everyone who helped.

@ghost
Copy link

ghost commented Aug 18, 2021

I'm still having this issue:
>>> import psycopg2 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/bartek/testenv2/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/bartek/testenv2/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID Referenced from: /Users/bartek/testenv2/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so Expected in: flat namespace in /Users/bartek/testenv2/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so

My setup:
macOS 11.5.1, M1
python 3.9.6 [pyenv]
using virtualenv

What I have tried:
creating new virtualenv with 3.9.2 and 3.9.6 to do following:
pip install psycopg2-binary --no-cache-dir
python
import psycopg2

also:
brew install libpq --build-from-source
export LDFLAGS="-L/opt/homebrew/opt/libpq/lib" # Add to .zshrc
$ pip3 install psycopg2 --no-cache-dir

not sure if this is related since it's an import issue:
I have Postgress.app installed, but I have not used in long time and it's not running. I use Docker for Postgress.

UPDATE:
What solved the issue for me is:
brew install postgresql
create new virtualenv
pip install psycopg2-binary --no-cache-dir

@psycopg psycopg deleted a comment from presentyle Sep 3, 2021
@abhishekjain-fin
Copy link

pyscopg2-binary needs to be installed in mac instead of psycopg2

pip uninstall psycopg2
pip install psycopg2-binary

@samrxth
Copy link

samrxth commented Sep 28, 2021

If anyone is struggling to make it work, make sure your terminal application supports M1 natively(the inbuilt terminal or iterm2 support m1 without rosetta)

@peter-feeney
Copy link

pip install psycopg2-binary solved the issue for me.

@nicksergeant
Copy link

@samrathchadha thank you so much for this:

#1216 (comment)

I'm using Alacritty and its ARM support is busted, so I had to install via x86 🤦.

@samrxth
Copy link

samrxth commented Dec 7, 2021

No problem @nicksergeant
I was using alacritty too before I tried installing it in iterm2

@lyntree
Copy link

lyntree commented Jan 3, 2022

I fixed the same problem in Mac BigSur 11.5.1 M1 following the tips of this thread in the next order:

$ pip3 uninstall psycopg2 $ brew install libpq --build-from-source export LDFLAGS="-L/opt/homebrew/opt/libpq/lib" # Add to .zshrc $ pip3 install psycopg2 --no-cache-dir

Thanks to everyone who helped.

Thank you for this, I just discovered this issue yesterday with Django, I gave it ago and and restarted my terminal and sure enough, at last started working :)

@EtienneLeconte-Stuart
Copy link

Hi there, I face the same issue as some of you guys. And after trying a lot of different solutions you proposed, I'm still faced with this error when trying to create an engine:

ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): no suitable image found. Did find: /opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture /opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture

I'm on Big Sur 11.0 M1 and using Pycharm as my IDE.

Any tip would be greatly appreciated!

@lyntree
Copy link

lyntree commented Jan 7, 2022

@EtienneLeconte-Stuart try following @bartfto post below as that did work for me and I was using Big Sur 11.0 M1 and got this fixed just 4 days ago :) / Also might I also add, What I also found with this is - There is another version of Python3 which is supported for 'universal' which is made for M1 (which i didn't know either)

''

My setup:
macOS 11.5.1, M1
python 3.9.6 [pyenv]
using virtualenv

What I have tried:
creating new virtualenv with 3.9.2 and 3.9.6 to do following:
pip install psycopg2-binary --no-cache-dir
python
import psycopg2

also:
brew install libpq --build-from-source
export LDFLAGS="-L/opt/homebrew/opt/libpq/lib" # Add to .zshrc
$ pip3 install psycopg2 --no-cache-dir

not sure if this is related since it's an import issue:
I have Postgress.app installed, but I have not used in long time and it's not running. I use Docker for Postgress.

UPDATE:
What solved the issue for me is:
brew install postgresql
create new virtualenv
pip install psycopg2-binary --no-cache-dir''

@EtienneLeconte-Stuart
Copy link

@lyntree thanks a lot for your answer! I tried the proposed solution and unfortunately that did nothing for me... I'm still at the same point.

Something must definitely be wrong with my configuration, but I struggle to get what.

@lyndseyjw
Copy link

lyndseyjw commented Jan 17, 2022

thank you @lyntree & everyone else above! after a longgggg time attempting most everything above, the key for me was "create new virtualenv"

I am running MacOS Big Sur Version 11.3.1, M1

I deleted the old venv I was working with & did the following:
in root folder ran:
pip3 uninstall psycopg2
pip3 install psycopg2 --no-cache-dir
brew install postgresql
created new virtualenv & in project folder ran:
pip3 install psycopg2-binary --no-cache-dir

the issue was magically solved

good luck everyone else, this is definitely a frustrating error!

@lyntree
Copy link

lyntree commented Jan 17, 2022 via email

@winny1212
Copy link

@nicksergeant @Jfeng3 @dvarrazzo @kevbradwick @leohakim
Sorry, I am totally new here, a bit confused about to make my texts following looks good.

I am having a similar issue connecting to a Postgres server through psycopg2 , please please help me have a look if anyone is free, thank you very much. I have tried to uninstall and reinstall, change the env installation method like the above solution but it doesn't work at all , the error is the same

I'm running:

Mac Big Sur 11.2
Python 3.10

when I running "python manage.py makemigrations", the errors accur as follows:

Traceback (most recent call last):
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 25, in
import psycopg2 as Database
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/init.py", line 51, in
from psycopg2._psycopg import ( # noqa
ImportError: dlopen(/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/_psycopg.cpython-310-darwin.so, 2): Symbol not found: _PQbackendPID
Referenced from: /Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/_psycopg.cpython-310-darwin.so
Expected in: flat namespace
in /Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/_psycopg.cpython-310-darwin.so

During handling of the above exception, another exception occurred:

_Traceback (most recent call last):
File "/Users/yujingchen/Desktop/real-estate/backend/manage.py", line 22, in
main()
File "/Users/yujingchen/Desktop/real-estate/backend/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/core/management/init.py", line 425, in execute_from_command_line
utility.execute()
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/core/management/init.py", line 401, in execute
django.setup()
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/init.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/apps/config.py", line 300, in import_models
self.models_module = import_module(models_module_name)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/contrib/auth/models.py", line 3, in
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/contrib/auth/base_user.py", line 47, in
class AbstractBaseUser(models.Model):
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/models/base.py", line 122, in new
new_class.add_to_class('_meta', Options(meta, app_label))
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/models/base.py", line 326, in add_to_class
value.contribute_to_class(cls, name)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/models/options.py", line 207, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/utils/connection.py", line 15, in getattr
return getattr(self._connections[self._alias], item)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/utils/connection.py", line 62, in getitem
conn = self.create_connection(alias)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/utils.py", line 204, in create_connection
backend = load_backend(db['ENGINE'])
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/utils.py", line 111, in load_backend
return import_module('%s.base' % backend_name)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 29, in
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/_psycopg.cpython-310-darwin.so, 2): Symbol not found: _PQbackendPID
Referenced from: /Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/_psycopg.cpython-310-darwin.so
Expected in: flat namespace
in /Users/yujingchen/Desktop/real-estate/backend/venv/lib/python3.10/site-packages/psycopg2/psycopg.cpython-310-darwin.so

@Danitocode
Copy link

This worked for me If you need to change your python path, please do it using `source ///bin/activate

  1. make sure that the current architecture is arm64, to do this you can set
    env /usr/bin/arch -arm64 /bin/zsh --login
    brew install libpq --build-from-source
    export LDFLAGS="-L/opt/homebrew/opt/libpq/lib"
    pip install psycopg2-binary
    Make sure to uninstall psycopg2 and psycopg2-binary to do a new install

You are the best man, you saved my life.

@m-hollow
Copy link

@winny1212 -- were you able to resolve the issue you posted? I got the same error today, and there are so many different proposed 'solutions' I'm not sure where to start; my error output is identical to yours, and occurs for the same reason -- when I run python manage.py makemigrations.
thanks!

@Danitocode
Copy link

@winny1212 -- were you able to resolve the issue you posted? I got the same error today, and there are so many different proposed 'solutions' I'm not sure where to start; my error output is identical to yours, and occurs for the same reason -- when I run python manage.py makemigrations.
thanks!

Maybe you can try with this as is solving the architecture issue with Macs M1 but maybe you already have tried. You have to run this commands with Rosseta terminal, just that's it, almost for me:

env /usr/bin/arch -arm64 /bin/zsh --login
brew install libpq --build-from-source
export LDFLAGS="-L/opt/homebrew/opt/libpq/lib"
pip install psycopg2-binary

@benthorner
Copy link

benthorner commented Mar 15, 2022

A slightly shorter version worked for me:

brew install libpq
export LDFLAGS="-L/opt/homebrew/opt/libpq/lib"
pip install --force psycopg2-binary

I think this issue is related to #1286 and may possibly be fixed by #1348.

@ArhanChaudhary
Copy link

None of the above solutions worked for me, so I tinkered around and got this to work:

env /usr/bin/arch -arm64 pip uninstall psycopg2-binary
env /usr/bin/arch -arm64 pip install psycopg2-binary --no-cache-dir
env /usr/bin/arch -arm64 pip uninstall psycopg2
env /usr/bin/arch -arm64 pip install psycopg2 --no-cache-dir

Cheers!

@holdenweb
Copy link

holdenweb commented Jul 8, 2022

An additional data point: I found that my m1 Mac was installing an x86_64 version of any extension packages (not pure Python), in my case apsw-wheels,lxml and psycopg2-binary. A typical error (reformatted with additional whitespace) was

ImportError: dlopen(/private/tmp/tmpTamato/lib/python3.9/site-packages/apsw.cpython-39-darwin.so, 0x0002):
    tried: '/private/tmp/tmpTamato/lib/python3.9/site-packages/apsw.cpython-39-darwin.so' 
    (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')),
    '/usr/local/lib/apsw.cpython-39-darwin.so' (no such file),
    '/usr/lib/apsw.cpython-39-darwin.so' (no such file)

I eventually discovered by trial and error how to overcome this, which was to uninstall with pip uninstall and then reinstall with (for example)

ARCHFLAGS="-arch arm64" pip install lxml --compile --no-cache-dir --no-binary :all:

This tells pip not to use any wheels with binary content (i.e., compile from the source distribution), and not to trust any code downloaded earlier but always seek fresh copies. Adding ARCHFLAGS to the process's environment forces compilation of arm binaries. After that everything loaded and ran without complaint.

Having discovered this I investigated why the standard process was producing x86 binaries. Examining the value of ARCHFLAGS in a brand new shell to my horror showed me this:

sholden@overhead ~ % echo $ARCHFLAGS
-arch x86_64

This was a setting I tracked down to my .zshrc, but I have no memory of that line so I'm assuming it was an Apple precautionary measure I didn't notice when I migrated from bash to zsh back before the m1 days, then pulled across thoughtlessly to the m1.

Mea culpa for not checking more carefully when I migrated (usual story, did it in a rush while busy with other things).
I've removed the line from .zshrc, and a standard build in a clean shell now runs happily. So the moral of this story is:

"Check your ARCHFLAGS setting."

@denangeles
Copy link

Thanks for the info above, was encountering this after upgrading from python3.8 to python3.10 and installing psycopg2-binary. My machine is OSX Big Sur 12.3.1, M1 Pro.

My project uses pipenv so just needed to convert these pip args:
--compile --no-cache-dir --no-binary :all:

into the following environment variables:
PIP_COMPILE=True
PIP_NO_CACHE_DIR=True
PIP_NO_BINARY=:all

Reinstalled psycopg2-binary and everything was working again.

@HardyNapses
Copy link

None of the above solutions worked for me, so I tinkered around and got this to work:

env /usr/bin/arch -arm64 pip uninstall psycopg2-binary
env /usr/bin/arch -arm64 pip install psycopg2-binary --no-cache-dir
env /usr/bin/arch -arm64 pip uninstall psycopg2
env /usr/bin/arch -arm64 pip install psycopg2 --no-cache-dir

Cheers!

Worked fine for me and I guess this is proper solution !!!

@io-ma
Copy link

io-ma commented Sep 20, 2022

@HardyNapses thank you, this fixed it for me!

@dvarrazzo
Copy link
Member

Assuming this was fixed with the release of M1 binary packages and can be fixed.

@PABourdais
Copy link

None of the above solutions worked for me, so I tinkered around and got this to work:

env /usr/bin/arch -arm64 pip uninstall psycopg2-binary
env /usr/bin/arch -arm64 pip install psycopg2-binary --no-cache-dir
env /usr/bin/arch -arm64 pip uninstall psycopg2
env /usr/bin/arch -arm64 pip install psycopg2 --no-cache-dir

Cheers!

Worked for me, thank you ♥️

@dvarrazzo
Copy link
Member

Closing the conversation because the ticket is solve and only attracting metoo replies.

(on a nonsense answer, but hey, whatever floats your macOS).

@psycopg psycopg locked as resolved and limited conversation to collaborators Feb 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests