Find externals/ compilers #108

chapman39 · 2023-03-17T00:12:50Z

Introduces 2 new arguments:
- spack-compiler-paths: Space delimited string of paths for Spack to search for compilers (if spack_setup_environment is True)
- spack-externals: Space delimited string of packages for Spack to search for externals (if spack_setup_environment is True)
If Uberenv cannot find a spack.yaml, it will automatically create one, search for compilers/ externals, and copy it over to package_source_dir.
Uberenv no longer stops if it cannot find a spack.yaml
Using argparse instead of optparse

Misc Notes:

I've found that hdf5-1.14.1-2 inconsistently works. It often gets a FetchError, which is why I try to build @1.14.0 in CI.

Fixes #105
Fixes #104

chapman39 · 2023-04-10T17:51:07Z

@white238 Was wondering about the workflow of --spack-create-environment (or maybe spack-setup-environment makes more sense?)

Were you thinking the user would run uberenv with --spack-setup-environment first, and uberenv would copy the spack.yaml over to a user-specified location (like `/scripts/spack/configs/)? And then later on you'd modify the spack.yaml and run uberenv again.

uberenv.py

white238 · 2023-04-14T23:26:37Z

.ci/test-project/uberenv_configs/install.json

@@ -5,6 +5,7 @@
 "spack_build_mode" : "install",
 "spack_configs_path": "spack_configs",
 "spack_packages_path": "packages",
+"spack_not_buildable_packages": "cmake pkg-config zlib",


I don't like this name but can't come up with a better one right now. Let me ponder this.

spack_packages_to_find ?

spack_externals ?

chapman39 · 2023-04-20T21:47:45Z

i've noticed spack compiler find finds way more compilers than users would probably want to find. unlike external find, you cant ask for a specific compiler to look for (e.g. gcc)-- as far as i know

[exe: /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack/bin/spack -D /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env compiler find]
==> Added 61 new compilers to /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env/spack.yaml

white238 · 2023-04-20T21:52:03Z

i've noticed spack compiler find finds way more compilers than users would probably want to find. unlike external find, you cant ask for a specific compiler to look for (e.g. gcc)-- as far as i know
[exe: /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack/bin/spack -D /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env compiler find]
==> Added 61 new compilers to /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env/spack.yaml

LC machines are very special that they have a million compilers. I don't think this is going to be true on most peoples machines.

But we might want to add how to specify a specific family, gcc vs clang

cyrush · 2023-04-20T21:57:06Z

In the past when no config was given, uberenv would look for compilers and a user could ask for the family they wanted as part of the spec passed via command line args (--spec=clang,etc).

It would be good to know how to do that easily with the env setup.

chapman39 · 2023-04-20T21:57:18Z

also, running plain spack compiler find on ruby doesn't even find the gcc versions we want. (it only found gcc@4.8.5)

the only way i could find the gcc compilers with compiler find is including the compiler path directly.

[meemee@ruby967:test-project]$ spack compiler find /usr/tce/packages/gcc/*
==> Added 7 new compilers to /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env/spack.yaml
    gcc@10.2.1  gcc@9.3.1  gcc@8.3.1  gcc@8.1.0  gcc@7.3.0  gcc@7.1.0  gcc@6.1.0
==> Compilers are defined in the following files:
    /usr/WS2/meemee/uberenv/repo/.ci/test-project/uberenv_libs/spack_env/spack.yaml

chapman39 · 2023-04-20T21:58:13Z

as far as i know spack compiler find doesn't have an option to search for a specific family
https://spack.readthedocs.io/en/latest/command_index.html#spack-compiler-find

cyrush · 2023-04-20T22:00:08Z

correct, but there was still an easy path to select from what was found. Seems I restated Chris' suggestion. Just need to know how to select.

white238 · 2023-04-20T22:42:24Z

as far as i know spack compiler find doesn't have an option to search for a specific family https://spack.readthedocs.io/en/latest/command_index.html#spack-compiler-find

There is a way to tell it to search a different prefix. My guess is by default it only searches in /usr or /opt. For LC, it also needs to search in /usr/tce/packages.

adrienbernede · 2023-04-21T08:11:00Z

We can pass a path to the find command, that’s correct.
Also, my main issue with spack compilers find is that it will not find certain compilers, you would have to load the appropriate module.

So, here is the work around I can suggest:
given a compiler family (gcc), or a compiler module name (gcc/8.3.1), look for the PATH prepend in the module files and use those paths to instruct spack where to find the compilers.

adrienbernede · 2023-04-21T08:16:44Z

I guess since Spack is meant to install a full software stack on a machine from scratch, they can’t assume modules are already set.

I think we can make that assumption. It is true that this may not be portable to machines where modules are not set, but that could be tested by the script.

…ind spack.yaml

chapman39 · 2023-04-27T22:28:32Z

I've been having trouble implementing spack_externals and spack_compiler_paths as a command line argument due to the fact that for some reason when I have spaces in the argument it only takes the first thing.

For example:
uberenv.py --spack-externals="cmake hdf5 zlib", will only take cmake.

I believe this could be fixed it we switched from OptParse (which has been deprecated since Python 3.2) to ArgParse.

Alternatively of course, we can delimit the argument without spaces (with ; or ,).

white238 · 2023-04-27T22:30:15Z

I've been having trouble implementing spack_externals and spack_compiler_paths as a command line argument due to the fact that for some reason when I have spaces in the argument it only takes the first thing.

For example: uberenv.py --spack-externals="cmake hdf5 zlib", will only take cmake.

I believe this could be fixed it we switched from OptParse (which has been deprecated since Python 3.2) to ArgParse.

Alternatively of course, we can delimit the argument without spaces (with ; or ,).

Please upgrade to argparse.

chapman39 · 2023-04-27T22:31:06Z

Also, thoughts on new variable names?

spack-compiler-paths: Space delimited string of paths for Spack to search for compilers (if spack_setup_environment is True)
spack-externals: Space delimited string of packages for Spack to search for externals (if spack_setup_environment is True)

kennyweiss

Thanks @chapman39 for this nice addition!

We should probably add some new CI tests to ensure this continues to work.

uberenv.py

kennyweiss · 2023-05-11T17:33:33Z

uberenv.py

+            if self.spack_compiler_paths is None:
+                spack_compiler_find_cmd = "{0} compiler find".format(self.spack_exe())
+            else:
+                spack_compiler_find_cmd = "{0} compiler find {1}".format(self.spack_exe(), self.spack_compiler_paths)


Meta-question (outside the scope of this PR):

Should we have a spack_command(command, additional_args) function in uberenv?
That way, this line could be written as something like:

spack_compiler_find_cmd = spack_command("compiler find", self.spack_compiler_paths)

…s://github.com/LLNL/uberenv into feature/chapman39/find_externals_and_compilers

uberenv.py

white238 · 2023-06-09T20:21:24Z

The documentation needs to be updated

white238 · 2023-06-09T20:38:05Z

./scripts/uberenv/uberenv.py --prefix=../libs

produces the following error:

Traceback (most recent call last):
  File "/mnt/c/Users/white238/projects/serac/repo/./scripts/uberenv/uberenv.py", line 1380, in <module>
    sys.exit(main())
  File "/mnt/c/Users/white238/projects/serac/repo/./scripts/uberenv/uberenv.py", line 1319, in main
    env.setup_paths_and_dirs()
  File "/mnt/c/Users/white238/projects/serac/repo/./scripts/uberenv/uberenv.py", line 758, in setup_paths_and_dirs
    if self.spack_env_file is None:
AttributeError: 'SpackEnv' object has no attribute 'spack_env_file'. Did you mean: 'spack_env_name'?

…ub.com:LLNL/uberenv into feature/chapman39/find_externals_and_compilers

white238 · 2023-06-15T21:34:37Z

Docs here:

https://uberenv.readthedocs.io/en/feature-chapman39-find_externals_and_compilers/

white238 · 2023-06-15T21:40:20Z

Thanks @chapman39 !

starting on new find externals/ compilers

131fbbe

chapman39 self-assigned this Mar 17, 2023

Merge branch 'main' into feature/chapman39/find_externals_and_compilers

1dd2837

white238 reviewed Apr 14, 2023

View reviewed changes

uberenv.py Outdated Show resolved Hide resolved

white238 reviewed Apr 14, 2023

View reviewed changes

copy spack.yaml, keep going after find, keep going after failure to f…

68718ed

…ind spack.yaml

argparse instead of optparse

e3c4d0b

chapman39 changed the title ~~WIP: Find externals/ compilers~~ Find externals/ compilers Apr 28, 2023

chapman39 added the enhancement New feature or request label Apr 28, 2023

chapman39 requested review from white238, cyrush and kennyweiss May 2, 2023 23:58

chapman39 added 2 commits May 11, 2023 10:29

merge conflict

9c6c045

args not opts

c407cfb

kennyweiss approved these changes May 11, 2023

View reviewed changes

chapman39 added 2 commits May 16, 2023 16:36

find ci check

0d7ede5

merge conflict

87df829

chapman39 added 7 commits May 16, 2023 16:41

rm warning if spack env file is none

1823e26

force hdf5 version

4d88add

add toss4, print spack.yaml at right location

8d07f49

ci debug print

8a959af

rm debug print

03973c5

update setup only comment

f06d4b2

Merge branch 'feature/chapman39/find_externals_and_compilers' of http…

2fd1c28

…s://github.com/LLNL/uberenv into feature/chapman39/find_externals_and_compilers

white238 reviewed Jun 9, 2023

View reviewed changes

uberenv.py Outdated Show resolved Hide resolved

chapman39 added 13 commits June 9, 2023 14:44

fix spack_env_file error, rm spack-setup-environment option

7b0d81c

Merge branch 'feature/chapman39/find_externals_and_compilers' of gith…

0d8f375

…ub.com:LLNL/uberenv into feature/chapman39/find_externals_and_compilers

rm invalid ci test, dev-builds test spack_setup_environment

f105710

fix spack configs error

e5ecedf

docs

a115512

fix docs

593e63f

create separate ci test for external/compiler find

45f4d12

fix spack yaml print

ef8ae17

rm hdf5 version

5577de2

only find hdf5, suggest hdf5 1.14.0 (otherwise FetchError)

70f410b

require hdf5 version...

b281989

actually fix spack yaml path

db1b4f8

force hdf5 1.14.0 on create-spack-yaml ci test

9871d50

white238 approved these changes Jun 15, 2023

View reviewed changes

fix project-json formatting line 206

194d08b

chapman39 merged commit d934c61 into main Jun 16, 2023

chapman39 deleted the feature/chapman39/find_externals_and_compilers branch June 16, 2023 00:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Find externals/ compilers #108

Find externals/ compilers #108

chapman39 commented Mar 17, 2023 •

edited

Loading

chapman39 commented Apr 10, 2023

white238 Apr 14, 2023

chapman39 Apr 20, 2023

adrienbernede Apr 21, 2023

chapman39 commented Apr 20, 2023

white238 commented Apr 20, 2023

cyrush commented Apr 20, 2023

chapman39 commented Apr 20, 2023

chapman39 commented Apr 20, 2023

cyrush commented Apr 20, 2023 •

edited

Loading

white238 commented Apr 20, 2023

adrienbernede commented Apr 21, 2023 •

edited

Loading

adrienbernede commented Apr 21, 2023 •

edited

Loading

chapman39 commented Apr 27, 2023

white238 commented Apr 27, 2023

chapman39 commented Apr 27, 2023 •

edited

Loading

kennyweiss left a comment

kennyweiss May 11, 2023

white238 commented Jun 9, 2023

white238 commented Jun 9, 2023

white238 commented Jun 15, 2023

white238 commented Jun 15, 2023

Find externals/ compilers #108

Find externals/ compilers #108

Conversation

chapman39 commented Mar 17, 2023 • edited Loading

chapman39 commented Apr 10, 2023

white238 Apr 14, 2023

Choose a reason for hiding this comment

chapman39 Apr 20, 2023

Choose a reason for hiding this comment

adrienbernede Apr 21, 2023

Choose a reason for hiding this comment

chapman39 commented Apr 20, 2023

white238 commented Apr 20, 2023

cyrush commented Apr 20, 2023

chapman39 commented Apr 20, 2023

chapman39 commented Apr 20, 2023

cyrush commented Apr 20, 2023 • edited Loading

white238 commented Apr 20, 2023

adrienbernede commented Apr 21, 2023 • edited Loading

adrienbernede commented Apr 21, 2023 • edited Loading

chapman39 commented Apr 27, 2023

white238 commented Apr 27, 2023

chapman39 commented Apr 27, 2023 • edited Loading

kennyweiss left a comment

Choose a reason for hiding this comment

kennyweiss May 11, 2023

Choose a reason for hiding this comment

white238 commented Jun 9, 2023

white238 commented Jun 9, 2023

white238 commented Jun 15, 2023

white238 commented Jun 15, 2023

chapman39 commented Mar 17, 2023 •

edited

Loading

cyrush commented Apr 20, 2023 •

edited

Loading

adrienbernede commented Apr 21, 2023 •

edited

Loading

adrienbernede commented Apr 21, 2023 •

edited

Loading

chapman39 commented Apr 27, 2023 •

edited

Loading