Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uberenv updates to support dev with python in spack envs #124

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

cyrush
Copy link
Member

@cyrush cyrush commented Sep 6, 2023

Updates that adjust uberenv spack env setup so that folks can develop using python modules installed to a spack view.

Addresses most challenges outlined #120.

May still need some cleanup, so comments welcome.

Details:

  • Adds a new step that patches host configs post build to use the paths in the spack view.

Uses patterns from .uberenv_config.json spack_host_config_patches` entry to drive what is patched:

Example input:

  "spack_host_config_patches":
 {
   "PYTHON_EXECUTABLE": "view/python*/bin/python3",
   "SPHINX_EXECUTABLE": "view/python*/bin/sphinx-build"
   }

Creates a -patchd.cmake file in the prefix dir (does not modify original host config)

  • adds --spack-skip-externals option

This is mostly driven by the limitations of using an external python, but is generally useful for cases where system libs undermine you.

  • symlink the default view of the spack env into {prefix}/spack_view for easy access.

- patch spack.yaml to create the view dir under spack_env, instead of a hidden file deeper in the dir hierarchy

This should only happen in cases where spack.yaml lacks view customization.

@cyrush
Copy link
Member Author

cyrush commented Sep 6, 2023

Looks like i need to fix the view dir patching logic.

@@ -788,17 +797,22 @@ def find_spack_pkg_path_from_hash(self, pkg_name, pkg_hash):
# TODO: at least print a warning when several choices exist. This will
# pick the first in the list.
if l.startswith(pkg_name) and len(l.split()) > 1:
return {"name": pkg_name, "path": l.split()[-1]}
pkg_path = l.split()[-1]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

part of a fix for #123

for l in out.split("\n"):
# TODO: at least print a warning when several choices exist. This will
# pick the first in the list.
if l.startswith(pkg_name) and len(l.split()) > 1:
return {"name": pkg_name, "path": l.split()[-1]}
pkg_path = l.split()[-1]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

part of a fix for #123

@cyrush
Copy link
Member Author

cyrush commented Sep 7, 2023

I removed the logic that tried to present a better default view by hacking yaml, and instead opt-ed to symlink whatever the view directory is under the prefix. This should be more robust and work with any view setup even if it was customized in an input spack.yaml

@cyrush
Copy link
Member Author

cyrush commented Sep 7, 2023

I need to test why we can't inspect and symlink the view for the --install case.

==> Updating view at /home/runner/work/uberenv/uberenv/.ci/test-project/uberenv_libs/spack_env/.spack-env/view
[exe: /home/runner/work/uberenv/uberenv/.ci/test-project/uberenv_libs/spack/bin/spack -D /home/runner/work/uberenv/uberenv/.ci/test-project/uberenv_libs/spack_env python -c 'env = spack.environment.active_environment();print(env.views["default"].get_projection_for_spec(env.matching_spec("'magictestlib_cached@1.0.0%gcc ^hdf5@1.14.0'")))']
[failed to find spack view info]

@cyrush
Copy link
Member Author

cyrush commented Dec 6, 2023

@white238 @kennyweiss @chapman39

This branch has been tested and is now in use for both conduit and ascent CI.
I would like to find a path to get it into uberenv main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

symlinking host config is broken with envs + extra spec constraints
1 participant