Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Bug: Fetching models from_dir #1532

Open
arnaucoma24 opened this issue Feb 4, 2025 · 8 comments
Open

🐛 Bug: Fetching models from_dir #1532

arnaucoma24 opened this issue Feb 4, 2025 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@arnaucoma24
Copy link

arnaucoma24 commented Feb 4, 2025

Describe the bug.

Hi all!

After cloning a model repository to a local directory (eos4wt0 to /aloy/home/acomajuncosa/Ersilia/models/eos4wt0), I am trying to fetch it via ersilia -v fetch eos4wt0 --from_dir /aloy/home/acomajuncosa/Ersilia/models/eos4wt0 (ersilia, version 0.1.41, python 3.10) and I encounter this issue (see below). Is there something I'm doing wrong? The file /home/acomajuncosa/eos/current.log seems to be empty.

Additionally, I tried to clone & fetched other models -all of them seem to give a similar (but not identical) JSONDecodeError. Let me know if you want to check more logs.


(ersilia) acomajuncosa@dante:/aloy/home/acomajuncosa/Ersilia$ ersilia -v fetch eos4wt0 --from_dir /aloy/home/acomajuncosa/Ersilia/models/eos4wt0
10:16:03 | DEBUG    | Repo path specified: /aloy/home/acomajuncosa/Ersilia/models/eos4wt0
10:16:03 | DEBUG    | Absolute path: /aloy/home/acomajuncosa/Ersilia/models/eos4wt0
⬇️  Fetching model eos4wt0: morgan-fps
10:16:03 | DEBUG    | Initialized with URL: None
10:16:03 | DEBUG    | Getting model source
10:16:03 | DEBUG    | Model getting fetched from Local Repository
10:16:03 | INFO     | Model doesn't exist on your system, fetching it now.
10:16:03 | DEBUG    | Starting fetching procedure
10:16:03 | DEBUG    | Fetching in your system, not from DockerHub
10:16:03 | DEBUG    | Deciding fetcher (BentoML or FastAPI)
10:16:03 | DEBUG    | Fetching using BentoML
Installing bentoml (the ersilia version)
Collecting git+https://github.com/ersilia-os/bentoml-ersilia.git
  Cloning https://github.com/ersilia-os/bentoml-ersilia.git to /tmp/pip-req-build-hfovysb1
  Running command git clone --filter=blob:none --quiet https://github.com/ersilia-os/bentoml-ersilia.git /tmp/pip-req-build-hfovysb1
  Resolved https://github.com/ersilia-os/bentoml-ersilia.git to commit d9d9f342294da3f1a69c9e81adb17f77bc010aac
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: docker in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (7.1.0)
Requirement already satisfied: alembic in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.14.1)
Requirement already satisfied: packaging in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (24.2)
Requirement already satisfied: boto3 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (1.36.12)
Requirement already satisfied: psutil in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.1)
Requirement already satisfied: cerberus in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.3.7)
Requirement already satisfied: sqlalchemy in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (2.0.37)
Requirement already satisfied: sqlalchemy-utils in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.41.2)
Requirement already satisfied: tabulate in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.9.0)
Requirement already satisfied: humanfriendly in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (10.0)
Requirement already satisfied: flask in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.0)
Requirement already satisfied: chardet in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (5.2.0)
Requirement already satisfied: numpy in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (1.26.4)
Requirement already satisfied: multidict in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.0)
Requirement already satisfied: requests in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from bentoml==0.11.0) (2.32.3)
Requirement already satisfied: python-json-logger in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.2.1)
Requirement already satisfied: ruamel.yaml in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.18.10)
Requirement already satisfied: werkzeug in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.3)
Collecting protobuf<3.19,>=3.8.0 (from bentoml==0.11.0)
  Using cached protobuf-3.18.3-py2.py3-none-any.whl.metadata (777 bytes)
Requirement already satisfied: prometheus_client in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.21.1)
Requirement already satisfied: Mako in /home/acomajuncosa/.local/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (1.3.8)
Requirement already satisfied: typing-extensions>=4 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (4.12.2)
Requirement already satisfied: greenlet!=0.4.17 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from sqlalchemy->bentoml==0.11.0) (3.1.1)
Requirement already satisfied: botocore<1.37.0,>=1.36.12 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.36.12)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.0.1)
Requirement already satisfied: s3transfer<0.12.0,>=0.11.0 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (0.11.2)
Requirement already satisfied: urllib3>=1.26.0 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from docker->bentoml==0.11.0) (2.3.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (2025.1.31)
Requirement already satisfied: Jinja2>=3.1.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (3.1.5)
Requirement already satisfied: itsdangerous>=2.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (2.2.0)
Requirement already satisfied: click>=8.1.3 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (8.1.8)
Requirement already satisfied: blinker>=1.9 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (1.9.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from werkzeug->bentoml==0.11.0) (3.0.2)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from ruamel.yaml->bentoml==0.11.0) (0.2.12)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from botocore<1.37.0,>=1.36.12->boto3->bentoml==0.11.0) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.37.0,>=1.36.12->boto3->bentoml==0.11.0) (1.17.0)
Using cached protobuf-3.18.3-py2.py3-none-any.whl (174 kB)
Installing collected packages: protobuf
Successfully installed protobuf-3.18.3
10:16:08 | DEBUG    | Checking if the model is installable with BentoML
10:16:08 | DEBUG    | Starting fetching procedure
10:16:08 | INFO     | GitHub CLI is not installed. Ersilia can work without it, but we highly recommend that you install this tool.
10:16:08 | DEBUG    | Git LFS is installed
Error: Failed to call git rev-parse --git-dir: exit status 128 
Git LFS initialized.
10:16:08 | DEBUG    | Git LFS has been activated
10:16:08 | DEBUG    | Connected to the internet
10:16:08 | DEBUG    | Conda is installed
10:16:08 | DEBUG    | EOS Home path exists
10:16:08 | INFO     | Starting delete of model eos4wt0
10:16:08 | DEBUG    | Attempting Bento delete
10:16:09 | DEBUG    | No BentoML Catalog available
10:16:09 | INFO     | Deleting conda environment eos4wt0
10:16:09 | DEBUG    | Attempting temporary folder delete
10:16:09 | DEBUG    | Attempting lake delete (local)
10:16:09 | DEBUG    | Deleting /home/acomajuncosa/eos/isaura/lake/eos4wt0_local.h5
10:16:09 | DEBUG    | Attempting lake delete (public)
10:16:09 | DEBUG    | Deleting /home/acomajuncosa/eos/isaura/lake/eos4wt0_public.h5
10:16:09 | INFO     | Removing docker images and stopping containers related to eos4wt0
10:16:09 | DEBUG    | Model entry eos4wt0 was not available in the fetched models registry
10:16:09 | SUCCESS  | Model eos4wt0 deleted successfully
10:16:09 | DEBUG    | Getting repository
10:16:10 | DEBUG    | Preparing inner template if necessary
10:16:10 | DEBUG    | The pack.py file already exists
10:16:10 | DEBUG    | The Dockerfile file already exists
10:16:10 | DEBUG    | The src folder already exists
10:16:10 | DEBUG    | Changing python version if necessary
10:16:10 | DEBUG    | {'version': '0.11.0', 'slim': False, 'python': 'py310'}
10:16:10 | DEBUG    | User is not root
10:16:10 | DEBUG    | Copying example file if available
10:16:10 | DEBUG    | Example file exists
10:16:10 | DEBUG    | Done getting model
10:16:10 | DEBUG    | Check if model can be run with vanilla (system) code (i.e. dockerfile has no installs)
10:16:10 | DEBUG    | Check bentoml and python version
10:16:10 | INFO     | BentoML version {'version': '0.11.0', 'slim': False, 'python': 'py310'}
10:16:10 | DEBUG    | Custom Ersilia BentoML is used, no need for modifying protobuf version
10:16:10 | DEBUG    | Model needs some installs
10:16:10 | DEBUG    | Checking if only python/conda install will be sufficient
10:16:10 | DEBUG    | Mode: conda
10:16:10 | DEBUG    | BentoML path exists! Removing it: /home/acomajuncosa/bentoml/repository/eos4wt0
10:16:10 | DEBUG    | Attempting Bento delete
10:16:10 | DEBUG    | No BentoML Catalog available
10:16:10 | DEBUG    | Trying to remove path: /home/acomajuncosa/bentoml/repository/eos4wt0
10:16:10 | DEBUG    | ...successfully
10:16:10 | DEBUG    | Initializing conda packer
10:16:10 | DEBUG    | Packing model with Conda
10:16:10 | DEBUG    | Writing install commands
10:16:10 | DEBUG    | Run commands: ['pip install rdkit==2023.9.2']
10:16:10 | DEBUG    | Writing install commands in /home/acomajuncosa/eos/dest/eos4wt0/model_install_commands.sh
10:16:10 | DEBUG    | Setting up
10:16:10 | DEBUG    | Installs file /home/acomajuncosa/eos/dest/eos4wt0/model_install_commands.sh
10:16:10 | DEBUG    | Conda environment eos4wt0
10:16:11 | DEBUG    | Environment eos4wt0 does not exist
10:16:11 | INFO     | Cloning base conda environment and adding model dependencies
Source:      /home/acomajuncosa/programs/anaconda3/envs/eosbase-bentoml-0.11.0-py310
Destination: /home/acomajuncosa/programs/anaconda3/envs/eos4wt0
Packages: 24
Files: 6154
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate eos4wt0
#
# To deactivate an active environment, use
#
#     $ conda deactivate

10:16:23 | DEBUG    | Run commandlines on eos4wt0
10:16:23 | DEBUG    | python -m pip --disable-pip-version-check install rdkit==2023.9.2
python -m pip --disable-pip-version-check install git+https://github.com/ersilia-os/bentoml-ersilia.git

10:16:23 | DEBUG    | Activating base environment
10:16:23 | DEBUG    | Current working directory: /home/acomajuncosa/eos/dest/eos4wt0
10:16:23 | DEBUG    | Running bash /tmp/ersilia-69xcq9h7/script.sh 2>&1 | tee -a /tmp/ersilia-_a1g7j_m/command_outputs.log
Collecting rdkit==2023.9.2
  Using cached rdkit-2023.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Requirement already satisfied: numpy in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from rdkit==2023.9.2) (2.2.2)
Collecting Pillow (from rdkit==2023.9.2)
  Using cached pillow-11.1.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
Using cached rdkit-2023.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30.5 MB)
Using cached pillow-11.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
Installing collected packages: Pillow, rdkit
Successfully installed Pillow-11.1.0 rdkit-2023.9.2
Collecting git+https://github.com/ersilia-os/bentoml-ersilia.git
  Cloning https://github.com/ersilia-os/bentoml-ersilia.git to /tmp/pip-req-build-oc8f_h8w
  Running command git clone --filter=blob:none --quiet https://github.com/ersilia-os/bentoml-ersilia.git /tmp/pip-req-build-oc8f_h8w
  Resolved https://github.com/ersilia-os/bentoml-ersilia.git to commit d9d9f342294da3f1a69c9e81adb17f77bc010aac
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: docker in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (7.1.0)
Requirement already satisfied: alembic in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.14.1)
Requirement already satisfied: packaging in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (24.2)
Requirement already satisfied: boto3 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (1.36.11)
Requirement already satisfied: psutil in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.1)
Requirement already satisfied: cerberus in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.3.7)
Requirement already satisfied: sqlalchemy in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (2.0.37)
Requirement already satisfied: sqlalchemy-utils in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.41.2)
Requirement already satisfied: tabulate in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.9.0)
Requirement already satisfied: humanfriendly in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (10.0)
Requirement already satisfied: flask in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.0)
Requirement already satisfied: chardet in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (5.2.0)
Requirement already satisfied: numpy in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (2.2.2)
Requirement already satisfied: multidict in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.0)
Requirement already satisfied: requests in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (2.32.3)
Requirement already satisfied: python-json-logger in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.2.1)
Requirement already satisfied: ruamel.yaml in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.18.10)
Requirement already satisfied: werkzeug in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.3)
Requirement already satisfied: protobuf<3.19,>=3.8.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (3.18.3)
Requirement already satisfied: prometheus_client in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.21.1)
Requirement already satisfied: Mako in /home/acomajuncosa/.local/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (1.3.8)
Requirement already satisfied: typing-extensions>=4 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (4.12.2)
Requirement already satisfied: greenlet!=0.4.17 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from sqlalchemy->bentoml==0.11.0) (3.1.1)
Requirement already satisfied: botocore<1.37.0,>=1.36.11 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.36.11)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.0.1)
Requirement already satisfied: s3transfer<0.12.0,>=0.11.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (0.11.2)
Requirement already satisfied: urllib3>=1.26.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from docker->bentoml==0.11.0) (2.3.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (2025.1.31)
Requirement already satisfied: Jinja2>=3.1.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (3.1.5)
Requirement already satisfied: itsdangerous>=2.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (2.2.0)
Requirement already satisfied: click>=8.1.3 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (8.1.8)
Requirement already satisfied: blinker>=1.9 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (1.9.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from werkzeug->bentoml==0.11.0) (3.0.2)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from ruamel.yaml->bentoml==0.11.0) (0.2.12)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from botocore<1.37.0,>=1.36.11->boto3->bentoml==0.11.0) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.37.0,>=1.36.11->boto3->bentoml==0.11.0) (1.17.0)
10:16:30 | DEBUG    | Collecting rdkit==2023.9.2
  Using cached rdkit-2023.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Requirement already satisfied: numpy in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from rdkit==2023.9.2) (2.2.2)
Collecting Pillow (from rdkit==2023.9.2)
  Using cached pillow-11.1.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
Using cached rdkit-2023.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30.5 MB)
Using cached pillow-11.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
Installing collected packages: Pillow, rdkit
Successfully installed Pillow-11.1.0 rdkit-2023.9.2
Collecting git+https://github.com/ersilia-os/bentoml-ersilia.git
  Cloning https://github.com/ersilia-os/bentoml-ersilia.git to /tmp/pip-req-build-oc8f_h8w
  Running command git clone --filter=blob:none --quiet https://github.com/ersilia-os/bentoml-ersilia.git /tmp/pip-req-build-oc8f_h8w
  Resolved https://github.com/ersilia-os/bentoml-ersilia.git to commit d9d9f342294da3f1a69c9e81adb17f77bc010aac
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: docker in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (7.1.0)
Requirement already satisfied: alembic in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.14.1)
Requirement already satisfied: packaging in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (24.2)
Requirement already satisfied: boto3 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (1.36.11)
Requirement already satisfied: psutil in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.1)
Requirement already satisfied: cerberus in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (1.3.7)
Requirement already satisfied: sqlalchemy in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (2.0.37)
Requirement already satisfied: sqlalchemy-utils in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.41.2)
Requirement already satisfied: tabulate in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.9.0)
Requirement already satisfied: humanfriendly in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (10.0)
Requirement already satisfied: flask in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.0)
Requirement already satisfied: chardet in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (5.2.0)
Requirement already satisfied: numpy in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (2.2.2)
Requirement already satisfied: multidict in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (6.1.0)
Requirement already satisfied: requests in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (2.32.3)
Requirement already satisfied: python-json-logger in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.2.1)
Requirement already satisfied: ruamel.yaml in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.18.10)
Requirement already satisfied: werkzeug in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (3.1.3)
Requirement already satisfied: protobuf<3.19,>=3.8.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from bentoml==0.11.0) (3.18.3)
Requirement already satisfied: prometheus_client in /home/acomajuncosa/.local/lib/python3.10/site-packages (from bentoml==0.11.0) (0.21.1)
Requirement already satisfied: Mako in /home/acomajuncosa/.local/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (1.3.8)
Requirement already satisfied: typing-extensions>=4 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from alembic->bentoml==0.11.0) (4.12.2)
Requirement already satisfied: greenlet!=0.4.17 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from sqlalchemy->bentoml==0.11.0) (3.1.1)
Requirement already satisfied: botocore<1.37.0,>=1.36.11 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.36.11)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (1.0.1)
Requirement already satisfied: s3transfer<0.12.0,>=0.11.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from boto3->bentoml==0.11.0) (0.11.2)
Requirement already satisfied: urllib3>=1.26.0 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from docker->bentoml==0.11.0) (2.3.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from requests->bentoml==0.11.0) (2025.1.31)
Requirement already satisfied: Jinja2>=3.1.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (3.1.5)
Requirement already satisfied: itsdangerous>=2.2 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (2.2.0)
Requirement already satisfied: click>=8.1.3 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (8.1.8)
Requirement already satisfied: blinker>=1.9 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from flask->bentoml==0.11.0) (1.9.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from werkzeug->bentoml==0.11.0) (3.0.2)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /home/acomajuncosa/.local/lib/python3.10/site-packages (from ruamel.yaml->bentoml==0.11.0) (0.2.12)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from botocore<1.37.0,>=1.36.11->boto3->bentoml==0.11.0) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.37.0,>=1.36.11->boto3->bentoml==0.11.0) (1.17.0)

10:16:30 | DEBUG    | Activation done
10:16:30 | DEBUG    | Creating environment YAML file
10:16:32 | DEBUG    | Storing Conda environment in the local environment database
10:16:32 | DEBUG    | Done with the Conda setup
10:16:33 | DEBUG    | Using environment eos4wt0
10:16:33 | DEBUG    | Running command: python pack.py
10:16:33 | DEBUG    | Run commandlines on eos4wt0
10:16:33 | DEBUG    | 
        python pack.py
        
10:16:33 | DEBUG    | Activating base environment
10:16:33 | DEBUG    | Current working directory: /home/acomajuncosa/eos/dest/eos4wt0
10:16:33 | DEBUG    | Running bash /tmp/ersilia-7ut32yo5/script.sh 2>&1 | tee -a /tmp/ersilia-s8n1n912/command_outputs.log
[2025-02-04 10:16:34,340] WARNING - Python 3.10.16 found in current environment is not officially supported by BentoML. The docker base image used is'bentoml/model-server:0.11.0' which will use conda to install Python 3.10.16 in the build process. Supported Python versions are: f3.6, 3.7, 3.8
[2025-02-04 10:16:35,484] INFO - BentoService bundle 'eos4wt0:20250204101634_76410F' saved to: /home/acomajuncosa/bentoml/repository/eos4wt0/20250204101634_76410F
10:16:35 | DEBUG    | [2025-02-04 10:16:34,340] WARNING - Python 3.10.16 found in current environment is not officially supported by BentoML. The docker base image used is'bentoml/model-server:0.11.0' which will use conda to install Python 3.10.16 in the build process. Supported Python versions are: f3.6, 3.7, 3.8
[2025-02-04 10:16:35,484] INFO - BentoService bundle 'eos4wt0:20250204101634_76410F' saved to: /home/acomajuncosa/bentoml/repository/eos4wt0/20250204101634_76410F

10:16:35 | DEBUG    | Activation done
10:16:35 | DEBUG    | Previous command successfully run inside eos4wt0 conda environment
10:16:35 | DEBUG    | Now trying to establish symlinks
10:16:35 | DEBUG    | BentoML location is /home/acomajuncosa/bentoml/repository/eos4wt0/20250204101634_76410F
10:16:35 | DEBUG    | Ersilia Bento location is /home/acomajuncosa/eos/repository/eos4wt0/20250204101634_76410F
10:16:35 | DEBUG    | Building symlinks between /home/acomajuncosa/eos/repository/eos4wt0/20250204101634_76410F and /home/acomajuncosa/bentoml/repository/eos4wt0/20250204101634_76410F
10:16:35 | DEBUG    | Creating model symlink bundle artifacts > dest
10:16:35 | DEBUG    | Creating model_install_commands.sh symlink dest <> bundle
10:16:35 | INFO     | Could not create symbolic link from /home/acomajuncosa/eos/dest/eos4wt0/data.h5 to /home/acomajuncosa/eos/isaura/lake/eos4wt0_public.h5
10:16:35 | DEBUG    | Run file found in framework: /home/acomajuncosa/eos/repository/eos4wt0/20250204101634_76410F/eos4wt0/artifacts/framework/run.sh
10:16:35 | DEBUG    | Run commandlines on eos4wt0
10:16:35 | DEBUG    | which python > /tmp/ersilia-wq7cnn_o/tmp.txt
10:16:36 | DEBUG    | Activating base environment
10:16:36 | DEBUG    | Current working directory: /aloy/home/acomajuncosa/Ersilia
10:16:36 | DEBUG    | Running bash /tmp/ersilia-v3nxtj7v/script.sh 2>&1 | tee -a /tmp/ersilia-yev3qr5e/command_outputs.log
10:16:36 | DEBUG    | 
10:16:36 | DEBUG    | Activation done
10:16:36 | DEBUG    | Python executable: /home/acomajuncosa/programs/anaconda3/envs/eos4wt0/bin/python
10:16:36 | DEBUG    | Conda is needed
10:16:36 | DEBUG    | Checking if model needs to be integrated to a tool
10:16:36 | DEBUG    | Getting model card of eos4wt0
10:16:36 | DEBUG    | Trying to get metadata from: /home/acomajuncosa/eos/dest/eos4wt0
10:16:36 | DEBUG    | Card saved at /home/acomajuncosa/eos/dest/eos4wt0/card.json
10:16:36 | DEBUG    | Saving slug morgan-fps
10:16:36 | DEBUG    | Checking that autoservice works
10:16:36 | DEBUG    | Setting BentoML AutoService for eos4wt0
10:16:36 | DEBUG    | No service class provided, deciding automatically
10:16:36 | DEBUG    | No service class file exists in /home/acomajuncosa/eos/repository/eos4wt0/20250204101634_76410F/service_class.txt
10:16:36 | DEBUG    | Pack method is: bentoml
10:16:36 | DEBUG    | Pack method is: bentoml
10:16:36 | DEBUG    | Setting virtual environment at /home/acomajuncosa/eos/dest/eos4wt0
10:16:36 | DEBUG    | Pack method is: bentoml
10:16:36 | DEBUG    | Pack method is: bentoml
10:16:36 | DEBUG    | Service class: conda
10:16:36 | DEBUG    | Getting APIs from list file
10:16:36 | DEBUG    | Getting APIs from BentoML
10:16:36 | DEBUG    | Getting APIs from Bento
10:16:36 | DEBUG    | Getting info from BentoML and storing in /tmp/ersilia-lub1k86a/information.json
🚨🚨🚨 Something went wrong with Ersilia 🚨🚨🚨

Error message:

Expecting value: line 1 column 1 (char 0)
If this error message is not helpful, open an issue at:
 - https://github.com/ersilia-os/ersilia
Or feel free to reach out to us at:
 - hello[at]ersilia.io

If you haven't, try to run your command in verbose mode (-v in the CLI)
 - You will find the console log file in: /home/acomajuncosa/eos/current.log
Task exception was never retrieved
future: <Task finished name='Task-1' coro=<ModelFetcher.fetch() done, defined at /aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch.py:345> exception=SystemExit(1)>
Traceback (most recent call last):
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/utils/exceptions_utils/throw_ersilia_exception.py", line 25, in inner_function
    return func(*args, **kwargs)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch.py", line 203, in _fetch_from_bentoml
    mf.fetch(model_id=self.model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch_bentoml.py", line 213, in fetch
    self._fetch(model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch_bentoml.py", line 171, in _fetch
    self._fetch_not_from_dockerhub(model_id=model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch_bentoml.py", line 154, in _fetch_not_from_dockerhub
    self._check()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch_bentoml.py", line 127, in _check
    mc.check()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/actions/check.py", line 27, in check
    AutoService(self.model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/serve/autoservice.py", line 216, in __init__
    self._set_apis()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/serve/autoservice.py", line 257, in _set_apis
    for api_name in self.service._get_apis_from_where_available():
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/serve/services.py", line 124, in _get_apis_from_where_available
    apis_list = self._get_apis_from_bento()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/serve/services.py", line 98, in _get_apis_from_bento
    info = self._get_info_from_bento()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/serve/services.py", line 92, in _get_info_from_bento
    info = json.load(f)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/cli/commands/__init__.py", line 31, in wrapper
    return func(*args, **kwargs)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/cli/commands/fetch.py", line 142, in fetch
    fetch_result = _fetch(mf, model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/cli/commands/fetch.py", line 37, in _fetch
    res = asyncio.run(mf.fetch(model_id))
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/nest_asyncio.py", line 92, in run_until_complete
    self._run_once()
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/site-packages/nest_asyncio.py", line 133, in _run_once
    handle._run()
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/acomajuncosa/programs/anaconda3/envs/ersilia/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch.py", line 366, in fetch
    fr = await self._fetch(model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch.py", line 334, in _fetch
    self._fetch_not_from_dockerhub(model_id=model_id)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/utils/exceptions_utils/throw_ersilia_exception.py", line 25, in inner_function
    return func(*args, **kwargs)
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/hub/fetch/fetch.py", line 228, in _fetch_not_from_dockerhub
    self._fetch_from_bentoml()
  File "/aloy/home/acomajuncosa/Ersilia/code/ersilia/ersilia/utils/exceptions_utils/throw_ersilia_exception.py", line 41, in inner_function
    sys.exit(DEFAULT_ERSILIA_ERROR_EXIT_CODE)
SystemExit: 1


Describe the steps to reproduce the behavior

No response

Operating environment

Ubuntu 20.04 3 LTS

@arnaucoma24 arnaucoma24 added the bug Something isn't working label Feb 4, 2025
@Abellegese
Copy link
Contributor

Hi @arnaucoma24 thanks for creating issue. This problem is related to bentoml dependency. We are creating a fix for this currently. To quickly solve this you can do the following on ersilia venv activated terminal.

pip uninstall bentoml

The just continue the fetching again.

@GemmaTuron
Copy link
Member

Thanks @Abellegese for being on top of this.
I am also interested in this issue as I think we have run into it other times. Does it have its specific debugging issue? or can you share more about the problem? I thought BentoML was not installed with Ersilia but I see this line in the pyproject.toml: bentoml = "ersilia.setup.requirements:check_bentoml"

Is that the source of the problem? And is it an issue that only happens when installing Ersilia in a new system?

@Abellegese
Copy link
Contributor

Hi @GemmaTuron so I highly suspect that this problem is, what we call it module shadow. Let me explain. We have a file called in ersilia bentoml.py located in ersilia/setup/requirements/. So when you install ersilia for the first time, this module (bentoml.py) will be packed as ersilia subpackage. Then you start fetching from github, local dir or s3 which might require bentoml installation, and when bentoml gets installed at this time, it override the first module and it works perfectly. But this installed bentoml package will be broken for the following possible reasons:

  1. Ersilia subpackage (bentoml.py) creates shadow over it and breaks dependency
  2. it might be due to a corrupted installation or conflicts in the environment
  3. So as we know ersilia, usually gets installed in editable mode. Editable installations often use symlinks to link ersilia package's source code to the Python environment. If these symlinks are broken or if there are leftover artifacts from a previous installation, it could cause issues.

So @OlawumiSalaam I think has this fix in her PR, will see that and fix this once and for all.

@GemmaTuron
Copy link
Member

Great thanks for the explanation I think I get it! @OlawumiSalaam anything you want to add here?

@OlawumiSalaam
Copy link
Contributor

Hi @GemmaTuron
I have implemented a fix that addresses the issue in my PR by capturing errors from BentoML subprocess calls, handling exceptions properly, and ensuring that any model artifacts are cleaned up if a fetch fails. Specifically, I added JSONDecodeError handling and checks for BentoML command failures, and I now raise a custom BentoMLException (which encapsulates these errors) to trigger the artifact cleanup proces. I also ensured that stderr from BentoML subprocess calls is captured and logged to surface hidden errors, such as network failures. This approach should guard against silent failures by handling all BentoML subprocess errors under a general umbrella as discussed with @DhanshreeA .

@GemmaTuron
Copy link
Member

amazing thanks @OlawumiSalaam for the detailed feedback
Can you guys let me know once this PR is merged?
and @arnaucoma24 can you confirm if the fix suggested by @Abellegese works? To be sure this is the issue we are facing. I also suggest once the PR is merged @arnaucoma24 could try installing Ersilia from scratch with the new code (and without manually uninstalling bentoml) to see if the bug is still there.
Thanks all!

@arnaucoma24
Copy link
Author

I can confirm pip uninstall bentoml fixes the issue and leads to smooth model fetching. Thanks a lot guys! Let me know whenever the PR is mergerd and I will give it a full try from scratch.

@Abellegese
Copy link
Contributor

Abellegese commented Feb 4, 2025

Hi @arnaucoma24 glad it works. Will let you know the ultimate fix for this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: On Hold
Development

No branches or pull requests

4 participants