Failure to find VTK libraries after updating MOOSE environment

[aarograh]

Bug Description

Based on guidance in the 2021-11 MOOSE newsletter, I tried to update the MOOSE environment on our ORNL machines. The process consistently failed to update the moose-libmesh and moose-tools libraries (no matter how much I tried to manually force it to install certain versions, conda kept claiming that it was already installed). I finally wiped the environment and started from scratch following the MOOSE "getting started" instructions.

For my normal development setup, this worked fine. However, the CI we have for our Mole code is failing with the following errors during compilation:

Compiling C++ (in opt mode) /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/build/unity_src/kernels_Unity.C...
Compiling C++ (in opt mode) /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/build/unity_src/materials_Unity.C...
Compiling C++ (in opt mode) /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/build/unity_src/postprocessors_Unity.C...
Compiling C++ (in opt mode) /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/build/unity_src/utils_Unity.C...
Linking Library /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/moose/modules/chemical_reactions/lib/libchemical_reactions-opt.la...
Linking Library /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/moose/modules/module_loader/lib/libmodule_loader_with_cr_ls-opt.la...
Linking Library /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/test/lib/libmole_test-opt.la...
Linking Library /localhome/gitlab-runner/builds/43d03e71/0/mole/mole/moose/framework/libmoose-opt.la...
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkIOCore-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkCommonCore-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkCommonDataModel-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkFiltersCore-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkIOXML-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkImagingCore-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkIOImage-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkImagingMath-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkIOParallelXML-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkParallelMPI-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkParallelCore-6.3
/localhome/gitlab-runner/miniconda3/envs/moose/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lvtkCommonExecutionModel-6.3
collect2: error: ld returned 1 exit status
make: *** [/localhome/gitlab-runner/builds/43d03e71/0/mole/mole/moose/framework/libmoose-opt.la] Error 1
make: *** Waiting for unfinished jobs....

Steps to Reproduce

It is unclear how to reproduce this since I did the same process one several different machines. MOOSE builds fine on my normal development machine (where I am manually building it), but it fails to build on the CI machine (where the build process is automated but follows the same set of commands as what I would do manually).

Impact

It prevents our automated testing from working properly, which is requirement to accept changes to Mole. We can manually test everything and post results but that's a nuisance and is definitely not as good of a QA procedure.

[loganharbour]

Switched this to a discussion.

Does your application require VTK? Are you missing the moose-libmesh-vtk conda package?

[aarograh]

Yep. Basically completely wiped out the old conda stuff and then followed these instructions: https://mooseframework.inl.gov/getting_started/installation/conda.html

[loganharbour]

We're working on cleaning those instructions as well.

Would you mind trying the instructions here: https://mooseframework.inl.gov/docs/PRs/19453/site/getting_started/installation/conda.html and pasting the resulting conda list once you're done?

One more question - do you have state within your CI builds? If so, I would suggest nuking the install every time and redoing it. Conda can be very funky with state

[aarograh]

I'm not sure what you mean by state. Do you mean a build which carries over from one build to the next? If so, then yes.

And to that point, I added a make clobberall line in the CI instructions just now and things seem to be working properly now. It looks like some old configuration information was messing something up and just needed to be wiped out. I probably should've thought to try that sooner.

[loganharbour]

Yep - that's exactly it (definition for state).

For a while we had state between CI builds, but for the sake of consistency we decided to start fresh every time.

[aarograh]

Good to know. Since that's what you guys do, I'll just leave that clobberall in there so that we're always doing a fresh MOOSE build. Hopefully that will make this simpler in the future.

Thanks for the help/information.