[ERROR] Runtime.ImportModuleError: Unable to import module 'ueba': libpq.so.5: cannot open shared object file: No such file or directory

[nitishsaboo]

Hi,

I am trying to use PostgreSQL in my AWS Lambda service.As part of that I have a build script that installs postgres using pip and sets LD_LIBRARY_PATH

requirements.txt

boto3==1.10.5
botocore==1.13.5
psycopg2==2.8.4

---

build.sh

#!/bin/sh
export PYTHONUNBUFFERED=1

if [ ! -d "venv" ]; then
virtualenv venv
fi

if [ -z "$VIRTUAL_ENV" ]; then
source venv/bin/activate
fi

if [ -n "$VIRTUAL_ENV" ]; then
venv/bin/pip install -U pip
venv/bin/pip install -U -r ./asset_db/requirements/requirements.txt
venv/bin/pip freeze --local
fi
export LD_LIBRARY_PATH="/usr/lib64/:$LD_LIBRARY_PATH"

---

I have a python file called ueba.py that has invocation function for lambda service.

---

ueba.py

import psycopg2

def database():
psycopg2.connect(user=,
password=,
host=,
port=,
database=)

def lambda_handler(event, context):
print("Lambda started")
print(event)
database()
print("Lambda Completed")

---

As part of build process I am creating a zip of the lambda service and uploading it on AWS and testing the lambda function.But when the Lambda function gets triggered I get the following error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'ueba': libpq.so.5: cannot open shared object file: No such file or directory

I understand it is a shared object that is needed at runtime.Since I am installing the psycopg2 using pip where will I find this shared object file and how to set the path to locate it.

Thanks,
Nitish

[nitishsaboo]

So i changed the build.sh to install psycopg2-binary instead of psycopg2 as per the github link:
#892

build.sh

#!/bin/sh
export PYTHONUNBUFFERED=1

if [ ! -d "venv" ]; then
virtualenv venv
fi

if [ -z "$VIRTUAL_ENV" ]; then
source venv/bin/activate
fi

if [ -n "$VIRTUAL_ENV" ]; then
venv/bin/pip install -U pip
venv/bin/pip install -U psycopg2-binary
venv/bin/pip install -U -r ./asset_db/requirements/requirements.txt
venv/bin/pip freeze --local
fi

When I tried running the lambda I am getting the following error now:

[ERROR] Runtime.ImportModuleError: Unable to import module 'ueba': /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

Can you please let me know what am I missing here ?

Thanks,
Nitish

[dvarrazzo]

This is pretty explicit.

[ERROR] Runtime.ImportModuleError: Unable to import module 'ueba': libpq.so.5: cannot open shared object file: No such file or directory

In the second case probably wrong python library version.

If you install psycopg by taking bits off one computer and copying to another computer you should know yourself what you are doing. Whatever happens is not our fault and I can't help with it.

[nitishsaboo]

@dvarrazzo
I fixed the problem by installing psycopg2-binary instead of psycopg2.
Yes, there was python version issue which I resolved.

1. Is installing psycopg2-binary the right approach for solving this problem ?Just need your small guidance on that. Please correct me if I am wrong.
   I referred to the following posts
   psycopy2-2.8.1 libpq.so.5: cannot open shared object file: No such file or directory #892
   and
   https://forums.aws.amazon.com/profile.jspa?userID=494804.

2. I am sorry but couldn't exactly get what you meant 'by taking bits off one computer and copying to another computer' in your earlier comment.Do you mean copying binaries of psycopg from one computer to another ?

3. Yes, I understand the following error is pretty explicit.But my question is how can we solve this library issue when running pscopg2 on AWS Lambda ? Do I have to link this library statically in my packaging zip rather than dynamic ?

[ERROR] Runtime.ImportModuleError: Unable to import module 'ueba': libpq.so.5: cannot open shared object file: No such file or directory

I see you have closed this issue.
Can you please clarify these points and guide me in the right direction ?

Thanks,
Nitish

[reubenfirmin]

For anybody who doesn't like this developer's hostile attitude (because of his architectural design choices that require native code), there are various pure python postgres clients, some of which claim to be faster than psycopy2-2.8.1. Switching to ps8000 for lambdas is a solid solution. See the full list here: https://wiki.postgresql.org/wiki/Python