Bugfix: Unknown ld Flag on Macos causes MET compilation to fail

[HathewayWill]

Describe the Problem

When using GCC@12 from the homebrew package manager MET fails to build due to an unknown ld flag --disable-new-dtags. Gives an error thatC compiler cannot create executables.

All the libraries appear to have built successfully.

Expected Behavior

MET tools should be able to determine what ld flags are needed based on OS.

Environment

Describe your runtime environment:
1. Machine: Virtual Mac Laptop
*2. OS: Mac OS 13.1.4
*3. Software version number(s) MET 11.0.2

To Reproduce

Describe the steps to reproduce the behavior:
` echo "MET INSTALLING"
brew install wget
brew install git
brew install gcc@12
brew install libtool
brew install automake
brew install autoconf
brew install make
brew install m4
brew install java
brew install ksh
brew install grads
brew install ksh
brew install tcsh
brew install snap
brew install python@3.11
brew install cmake
brew install xorgproto
brew install xorgrgb
brew install xauth
brew install curl
brew install flex
brew install byacc
brew install bison
brew install gnu-sed

pip3.11 install python-dateutil

#Directory Listings
mkdir $HOME/DTC
export WRF_FOLDER=$HOME/DTC

if [ "$WRFCHEM_PICK" = "1" ]; then
  mkdir $HOME/WRFCHEM
  export WRF_FOLDER=$HOME/WRFCHEM
fi

if [ "$WRFHYDRO_COUPLED_PICK" = "1" ]; then
  mkdir $HOME/WRFHYDRO_COUPLED
  export WRF_FOLDER=$HOME/WRFHYDRO_COUPLED
fi

if [ "$WRFHYDRO_STANDALONE_PICK" = "1" ]; then
  mkdir $HOME/WRFHYDRO_STANDALONE
  export WRF_FOLDER=$HOME/WRFHYDRO_STANDALONE
fi


if [ "$WRF_PICK" = "1" ]; then
  mkdir $HOME/WRF
  export WRF_FOLDER=$HOME/WRF
fi

if [ "$HWRF_PICK" = "1" ]; then
  mkdir $HOME/HWRF
  export WRF_FOLDER=$HOME/HWRF
fi



mkdir $WRF_FOLDER/MET-11.0.2
mkdir $WRF_FOLDER/MET-11.0.2/Downloads
mkdir $WRF_FOLDER/METplus-5.0.1
mkdir $WRF_FOLDER/METplus-5.0.1/Downloads




  #Symlink to avoid clang conflicts with compilers
  #default gcc path /usr/bin/gcc
  #default homebrew path /usr/local/bin

  echo "Please enter password for linking GNU libraries"

  echo $PASSWD | sudo -S ln -sf /usr/local/bin/gcc-12 /usr/local/bin/gcc
  echo $PASSWD | sudo -S ln -sf /usr/local/bin/g++-12 /usr/local/bin/g++
  echo $PASSWD | sudo -S ln -sf /usr/local/bin/gfortran-12 /usr/local/bin/gfortran
  echo $PASSWD | sudo -S ln -sf /usr/local/bin/python3.11 /usr/local/bin/python3



cd $WRF_FOLDER/MET-11.0.2/Downloads

wget -c -4 https://raw.githubusercontent.com/dtcenter/MET/main_v11.0/internal/scripts/installation/compile_MET_all.sh

wget -c -4 https://dtcenter.ucar.edu/dfiles/code/METplus/MET/installation/tar_files.tgz

wget -c -4 https://github.com/dtcenter/MET/archive/refs/tags/v11.0.2.tar.gz

cp compile_MET_all.sh $WRF_FOLDER/MET-11.0.2
tar -xvzf tar_files.tgz -C $WRF_FOLDER/MET-11.0.2
cp v11.0.2.tar.gz $WRF_FOLDER/MET-11.0.2/tar_files
cd $WRF_FOLDER/MET-11.0.2



cd $WRF_FOLDER/MET-11.0.2



export PYTHON_VERSION=$(python3 -V 2>1|awk '{print $2}')
export PYTHON_VERSION_MAJOR_VERSION=$(echo $PYTHON_VERSION | awk -F. '{print $1}')
export PYTHON_VERSION_MINOR_VERSION=$(echo $PYTHON_VERSION | awk -F. '{print $2}')
export PYTHON_VERSION_COMBINED=$PYTHON_VERSION_MAJOR_VERSION.$PYTHON_VERSION_MINOR_VERSION

export CC=/usr/local/bin/gcc
export CXX=/usr/local/bin/g++
export CFLAGS="-fPIC -fPIE -O3 -Wno-implicit-function-declaration"
export FC=/usr/local/bin/gfortran
export F77=/usr/local/bin/gfortran
export F90=/usr/local/bin/gfortran
export gcc_version=$(gcc -dumpfullversion)
export TEST_BASE=$WRF_FOLDER/MET-11.0.2
export COMPILER=gnu_$gcc_version
export MET_SUBDIR=${TEST_BASE}
export MET_TARBALL=v11.0.2.tar.gz
export USE_MODULES=FALSE
export MET_PYTHON=/usr/local/

export MET_PYTHON_CC="-I$MET_PYTHON/opt/python@$PYTHON_VERSION_COMBINED/Frameworks/Python.framework/Versions/$PYTHON_VERSION_COMBINED/include/python$PYTHON_VERSION_COMBINED"
export MET_PYTHON_LD="-L$MET_PYTHON/opt/python@$PYTHON_VERSION_COMBINED/Frameworks/Python.framework/Versions/$PYTHON_VERSION_COMBINED/lib/python$PYTHON_VERSION_COMBINED/config-$PYTHON_VERSION_COMBINED-darwin -lpython$PYTHON_VERSION_COMBINED -ldl -framework CoreFoundation"


export SET_D64BIT=FALSE


chmod 775 compile_MET_all.sh

sed -i'' -e "831s/export/#export/g" compile_MET_all.sh
time ./compile_MET_all.sh 2>&1 | tee compile_MET_all.log

config.log
configure.log
compile_MET_all.log

Relevant Deadlines

NONE.

Funding Source

2792543

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Development issue
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[jprestop]

@jprestop I am not able to reproduce this problem on Cheyenne using gcc-12.1.0. Note the user is on a MacOS and is using gcc-12.3.0. Will need to set up similar environment to test.

[HathewayWill]

@jprestop I am not able to reproduce this problem on Cheyenne using gcc-12.1.0. Note the user is on a MacOS and is using gcc-12.3.0. Will need to set up similar environment to test.

If you have virtual machine, i can send you the image

[jprestop]

Hi @HathewayWill. I don't have a VM, but I was able to get gcc 12.3.0 using MacPorts and was able to reproduce the problem you are experiencing. I'll get a fix in place and will update here once it's ready.

[HathewayWill]

Hi @HathewayWill. I don't have a VM, but I was able to get gcc 12.3.0 using MacPorts and was able to reproduce the problem you are experiencing. I'll get a fix in place and will update here once it's ready.

@jprestop did this fix get merged into the main branch or the next release?

[jprestop]

Hi @HathewayWill. No, it did not. On Monday, after fixing this problem, I discovered that HDF5 wasn't compiling successfully on my Mac. I wanted to get that issue resolved before merging in the changes in case additional changes to the script were necessary. I haven't been able to get back to this task, so it is not yet complete.

[HathewayWill]

@jprestop I cansend my hdf5 commands for Mac ports

…

On Wed, Jul 5, 2023, 9:37 PM jprestop ***@***.***> wrote: Hi @HathewayWill <https://github.com/HathewayWill>. No, it did not. On Monday, after fixing this problem, I discovered that HDF5 wasn't compiling successfully on my Mac. I wanted to get that issue resolved before merging in the changes in case additional changes to the script were necessary. I haven't been able to get back to this task, so it is not yet complete. — Reply to this email directly, view it on GitHub <#2596 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4FUT5UIOT3BSMRD65BYYKLXOYQGXANCNFSM6AAAAAAZZ3VCQM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jprestop]

Thanks, @HathewayWill. That is kind of you. However, I'm hoping to get the compilation working via the script. I'd like users who don't have access to MacPorts and don't have much IT support still be able to install MET and its dependent libraries,

[HathewayWill]

Thanks, @HathewayWill. That is kind of you. However, I'm hoping to get the compilation working via the script. I'd like users who don't have access to MacPorts and don't have much IT support still be able to install MET and its dependent libraries,

@jprestop sounds good. Hopefully something will work out without too much pain.

[HathewayWill]

@HathewayWill Since this issue is resolved, I'm going to go ahead and merge it. I will work on the HDF5 library problem another time.

Is it merged into the main branch or next update I'll test it out

@jprestop

[jprestop]

@HathewayWill It has been merged into main_v11.0 (current official release), but not yet main_v11.1 (official release scheduled for 7/31).

[HathewayWill]

@jprestop @georgemccabe

I can confirm that the fix for #2600 works for homebrew gcc. See attached log

compile_MET_all.log

[jprestop]

@HathewayWill Thank you for the confirmation!

Reopening as the work isn't complete for the main_v11.1 branch.

[jprestop]

@HathewayWill Would you be able to test the compile_MET_all.sh script (linked) from my branch with the MET-11.1.0-rc1 release (linked)?

[HathewayWill]

@HathewayWill Would you be able to test the compile_MET_all.sh script (linked) from my branch with the MET-11.1.0-rc1 release (linked)?

@jprestop @georgemccabe

Testing it right now. So far it has some bugs but I'm not sure if it is the MAKE_ARGS variable causing it or something else. I sent a email to @jprestop

[HathewayWill]

@jprestop @georgemccabe

MET fails to build at NETCDF-CXX. Log files and script used to test the code on mac is attached in a zip file.
MET11.1.0_METPLUS5.1.0.zip
MET-11.1.0-rc1.txt

[jprestop]

It seems that when MET fails to build due to the unknown ld flag "--disable-new-dtags" is not a gcc 12 problem, but rather a Mac problem. When I was compiling with gcc 11.3.0 on my Mac I had the same problem. I have modified the code and will submit a new pull request.

[HathewayWill]

Great looking forward to testing it. Do you want to do another webcall when I do? We can real time test

…

On Wed, Jul 19, 2023, 5:04 PM jprestop ***@***.***> wrote: It seems that when MET fails to build due to the unknown ld flag "--disable-new-dtags" is not a gcc 12 problem, but rather a Mac problem. When I was compiling with gcc 11.3.0 on my Mac I had the same problem. I have modified the code and will submit a new pull request. — Reply to this email directly, view it on GitHub <#2596 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4FUT5RABZDHOBHNFTWESPDXRBKVTANCNFSM6AAAAAAZZ3VCQM> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[jprestop]

@HathewayWill Please feel free to test whenever it's convenient for you. If you'd like to use the script from the branches below are the links for easy access:

For main_v11.0: https://raw.githubusercontent.com/dtcenter/MET/bugfix_2596_main_v11.0_rpath_compilation/internal/scripts/installation/compile_MET_all.sh

For main_v11.1:
https://raw.githubusercontent.com/dtcenter/MET/bugfix_2596_main_v11.1_rpath_compilation/internal/scripts/installation/compile_MET_all.sh

[HathewayWill]

@georgemccabe @jprestop

Results from #2613 (PASSES)
Zip file with relevant log files and new variables.
MET11.0.zip

Results from #2614 (FAILS)

• Builds all libraries but MET fails to build
• Zip file with relevant log files, script to build MET, new variables etc.
  MET11.1.zip

Worth noting new commands that work well:

export MET_PYTHON_CC="$(python3.10-config --cflags --embed)"
export MET_PYTHON_LD="$(python3.10-config --ldflags --embed) -L${MET_PYTHON}/lib -lpython${PYTHON_VERSION_COMBINED}"

[jprestop]

Thanks @HathewayWill for confirming that for #2614 you were able to compile the dependent libraries, but noted that MET also fails for you. These errors, while different from mine, also seem unrelated to the script. I have noted your errors along with mine in issue #2615.

[jprestop]

@HathewayWill I just wanted to send an update. I thought that your errors were different from mine, but it turns out that you had additional warnings in your file, but the actual error was the same. I have updated the description in issue #2615 to note that we did, indeed, receive the same error. It's great that you were able to reproduce the same error I received.

[HathewayWill]

@jprestop

I added a few comments to the other issues. Also someone found a solution to 11.1 due to a header issue.