Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use poetry for python dependencies #72

Merged
merged 1 commit into from
May 31, 2024
Merged

chore: use poetry for python dependencies #72

merged 1 commit into from
May 31, 2024

Conversation

plexoos
Copy link
Member

@plexoos plexoos commented May 15, 2024

No description provided.

@rahmans1
Copy link
Contributor

rahmans1 commented May 16, 2024

Just curious, is it not possible to handle this through spack alone? Also is this just for pyvista https://packages.spack.io/package.html?name=py-pyvistaqt ?

@plexoos
Copy link
Member Author

plexoos commented May 16, 2024

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

@rahmans1
Copy link
Contributor

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

We can request a version update or I can add myself as a maintainer for that spack package. It usually requires a small update like spack/spack#41246 to have the version update if we require the most latest version.

@plexoos
Copy link
Member Author

plexoos commented May 16, 2024

We can certainly consider Spack as a package manager for python... But I think for now a more pressing issue is that pyvista does not work on our GPU machine. Here I created an issue: #76
Could you please take a look?

@buddhasystem
Copy link

@plexoos so installing "poetry" and its paraphernalia is just another attempt to display pyvista on your Mac?

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024

No

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024

It's for non-Mac user like you

@plexoos plexoos force-pushed the pr/poetry branch 2 times, most recently from 314c755 to 9d09f5f Compare May 24, 2024 17:18
@rahmans1
Copy link
Contributor

I think the major python packages are available through spack. I am tempted to branch of beta and see how well that works instead of adding poetry as a separate dependency management tool.

https://packages.spack.io/package.html?name=py-ipython
https://packages.spack.io/package.html?name=py-numpy

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

@rahmans1
Copy link
Contributor

rahmans1 commented May 24, 2024

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~# 

@rahmans1
Copy link
Contributor

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.
Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~# 

Anyways. I figured it out. It just doesn't know where the .toml file is automatically. I have to launch it from the right folder or have it in path.

@rahmans1
Copy link
Contributor

rahmans1 commented May 24, 2024

This system does not appear to be running an xserver.
PyVista will likely segfault when rendering.

Try starting a virtual frame buffer with xvfb, or using
  ``pyvista.start_xvfb()``
  warnings.warn(
opticks/g4cx/tests/G4CXTest_raindrop.sh: line 189:   302 Segmentation fault      (core dumped) ${IPYTHON:-ipython} --pdb -i $script
opticks/g4cx/tests/G4CXTest_raindrop.sh : ana error with script G4CXTest_raindrop.py

Is this the error you get now? I guess we will just need to modify the plotting script so that it doesn't try to launch the headless window with xfvb which is not in our system.

@buddhasystem
Copy link

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024 via email

@rahmans1
Copy link
Contributor

rahmans1 commented May 24, 2024

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

[rahmans@npps0 ~]$ startx
xauth: file /home/rahmans/.serverauth.1541623 does not exist

/usr/libexec/Xorg.wrap: Only console users are allowed to run the X server

xinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: server error
Couldn't get a file descriptor referring to the console

@rahmans1
Copy link
Contributor

Sakib, please use the updated esi-shell from this pr

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

curl -Os https://raw.githubusercontent.com/BNLNPPS/esi-shell/pr/poetry/esi-shell && chmod u+x esi-shell

@rahmans1
Copy link
Contributor

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

I get the same error after getting the updated esi-shell script

@plexoos
Copy link
Member Author

plexoos commented May 24, 2024

You reported two errors. The updated esi-shell addressed the first one where it could not find a pyproject.toml. Now, you don't need to install with poetry by hand.

The second one about "This system does not appear to be running an xserver ... " is because you are running remotely and X windows cannot be opened locally. If you have a direct access to a GPU machine (NVidia of course) with X11, this error should not appear. Unfortunately, I don't have such machine to test but that is what I expect.

Maxim @buddhasystem it seems like you have a local NVidia GPU. Could you try the test I suggested above to see if we get any visuals from it?

@plexoos plexoos force-pushed the pr/poetry branch 2 times, most recently from bb2d1d1 to e379ba6 Compare May 31, 2024 13:50
@plexoos plexoos merged commit 3e079dd into beta May 31, 2024
1 check passed
@plexoos plexoos deleted the pr/poetry branch May 31, 2024 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants