fail to import pyhit in HPC

[ada-ayy]

Hello, everyone!
I have installed moose on the HPC cluster followed by https://mooseframework.inl.gov/getting_started/installation/conda.html.
When I try "run_tests" on my login node, they all go on smoothly. But when I submit a task to compute nodes, it always fails with "fail to import pyhit".
$ which python
~/miniconda3/envs/moose/bin/python
$ module list
No modules loaded
(for the compute nodes, I sometimes module load intel, but it still fails to import pyhit)
And here is my conda list and ldd result.
conda list and ldd.txt

Thanks!

[GiudGiud]

Hello

In your submit script, do you load the environment?
eg do you have conda activate moose ?

Note that the conda install isnt optimal for HPC clusters. You want to look at these instructions for that:
https://mooseframework.inl.gov/getting_started/installation/hpc_install_moose.html

Guillaume

[ada-ayy]

yeah, I have loaded the environment, when I "echo $CONDA_DEFAULT_ENV", both of the login node and compute node return "moose"
I have tried https://mooseframework.inl.gov/getting_started/installation/hpc_install_moose.html, but it fails with many problems, so I wonder whether I could use conda to install it.

[GiudGiud]

My experience is that you can but it wont be as fast. Not sure for the error message. @milljm will know

[ada-ayy]

Thanks! I will try the hpc_install guide again if the conda doesn't fit.

[ada-ayy]

@milljm Hello! Do you have any suggestions for me?

[milljm]

Are you using qsub to launch a job? If so, can we see the qsub script?

Have you tried to perform the same things in this qsub script, manually, using qsub -I. -I launches an interactive session on a node, allowing you to troubleshoot things like this.

"fail to import pyhit" usually means MOOSE is attempting to be run with a different environment than it was built with. So we're looking for reasons why "conda activate moose" may not be occurring during your job submission routine.

INL HPC?

Which HPC cluster are we talking about? One of ours at INL? If so, we have modules available that are better suited to running MOOSE instead of using Conda's "non-optimized compilers". Concerning Falcon, Lemhi, and Sawtooth, the general instructions for obtaining these compilers is:

module load use.moose PETSc

[milljm]

Also, sorry @ada-ayy. I am going through my email after a long period of inactivity (thanksgiving holiday for us), and I just made it to yours (about me asking you to create a new discussion). I was keeping my eye out for one from you, but it slipped through my attention...

[ada-ayy]

Never mind, I'm not in a hurry :）
I use sbatch script (slurm system) to submit my task.
The HPC cluster I use is from my university, not INL. I do not use the default conda environment in my cluster, but create a new one within my home directory. Maybe this is the problem?
The operating system is CentOS 7.6.

This is my script:

#!/bin/bash
#SBATCH -p cpu-normal 
#SBATCH -J moosetest
#SBATCH -N 1
#SBATCH -n 4
#SBATCH  -t 1:00:00

srun hostname | sort > machinefile.${SLURM_JOB_ID}

NP=`cat machinefile.${SLURM_JOB_ID} | wc -l`

#conda activate moose
#module load intel/19.0.5.281

#make hit

#srun -n 4 ./run_tests 
./run_tests -j 4

[milljm]

I am not understanding this batch script... it looks like conda activate moose is commented out. Pretty sure you want this line to actually run. I would uncomment that as the first step.

[ada-ayy]

yeah, I have tried, but failed with the same problem again. Maybe it is more related to the environment of my HPC cluster. I will ask the administer for advice or try another way to install moose. Thanks again!