Consider adding parallelization to the compile_MET_all.sh script

[jprestop]

Describe the Task

This issue arose from the METplus Discussion Integration of OpenMP/MPI for MET_COMPILE_ALL.sh #1678.

Good afternoon METplus team,

I was wondering if there is a way to add parallelization installation option for the MET_Compile_all.sh script. As it stands the dependencies are installed in serial as well as MET which can take several minutes to over half an hour or more due to system resources.

By utilizing multiple processors this installation could be cut down by 25-50% ideally.

Would this be something to consider if funding is available for it?

So by looking at the compile_met_all script. mpicc can be called as an argument in front of the script ($FC $CC $CXX etc.). However the compile script is not parallel enabled with (-j (# of cores) ). So mpich would have to be installed ahead of running compile_met_all.sh

Time Estimate

1-3 days of work

Sub-Issues

Consider breaking the task down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

NONE

Funding Source

TBD

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[ghost]

Possible solution for core management.

Add an IF/ELSE statement for core optimization in compile_MET_all.sh.

#############################Core Management####################################

export CPU_CORE=$(nproc)                                             #number of available cores on system
export CPU_6CORE="6"
export CPU_HALF=$(($CPU_CORE / 2))                                   #half of availble cores on system
export CPU_HALF_EVEN=$(( $CPU_HALF - ($CPU_HALF % 2) ))              #Forces CPU cores to even number to avoid partial core export. ie 7 cores would be 3.5 cores.

if [ $CPU_CORE -le $CPU_6CORE ]                                  #If statement for low core systems.  Forces computers to only use 1 core if there are 4 cores or less on the system.
then
  export CPU_HALF_EVEN="2"
else
  export CPU_HALF_EVEN=$(( $CPU_HALF - ($CPU_HALF % 2) )) 
fi


echo "##########################################"
echo "Number of cores being used $CPU_HALF_EVEN"
echo "##########################################"

Possible solution for argument in front of compile_MET_all.sh
CC=$MPICC FC=$MPIFC F77=$MPIF77 F90=$MPIF90 CXX=$MPICXX ./compile_MET_all.sh

Possible solution for compile_MET_all.sh if statement

# Compile GSL
if [ $COMPILE_GSL -eq 1 ]; then

  if [ ${COMPILER_FAMILY} = "pgi" ]; then
    vrs="1.11";
  else
    vrs="2.1";
  fi

  echo
  echo "Compiling GSL_${vrs} at `date`"
  mkdir -p ${LIB_DIR}/gsl
  cd ${LIB_DIR}/gsl
  rm -rf gsl*
  tar -xf ${TAR_DIR}/gsl-${vrs}.tar.gz
  cd gsl*
  echo "cd `pwd`"
  echo "./configure --prefix=${LIB_DIR} > configure.log 2>&1"
  ./configure --prefix=${LIB_DIR} > configure.log 2>&1
  ret=$?
  if [ $ret != 0 ]; then
    echo "configure returned with non-zero ($ret) status"
    exit 1
  fi
  echo "make > make.log 2>&1"
  >> make -j CPU_HALF_EVEN > make.log 2>&1
  ret=$?
  if [ $ret != 0 ]; then
    echo "make returned with non-zero ($ret) status"
    exit 1
  fi
  echo "make install > make_install.log 2>&1"
>>  make -j CPU_HALF_EVEN  install > make_install.log 2>&1
  ret=$?
  if [ $? != 0 ]; then
    echo "make install returned with non-zero ($ret) status"
    exit 1
  fi
fi

Just my thoughts on this. If it is outside the scope of the project I will try to implement in my community driven code.

[jprestop]

This issue was completed with PR Bugfix #2390 develop compile zlib #2404 as part of Issue Bugfix: Issues with the met_compile_all.sh script and associated tar files. #2390:

Added support for setting make arguments used to set -j argument to use multiple jobs to compile

Closing this issue.