Add Jet support to fit2obs

[KateFriedman-NOAA]

@DavidHuber-NOAA is working on porting global-workflow to RDHPCS Jet. Add in support for building and running fit2obs on Jet.

Related to issue #4 being worked on by @KateFriedman-NOAA.

[DavidHuber-NOAA]

I am giving this a test run presently.

[KateFriedman-NOAA]

I have edits in a branch for functionality on WCOSS2. Mind seeing if those edits work on Jet? I just committed them to my branch: https://github.com/KateFriedman-NOAA/Fit2Obs/tree/feature/wcoss2_jet_support

Still testing my branch on WCOSS2 with a copy of global-workflow develop but so far it appears to be working. Need to look at logs for silent errors still and have Jack look at output.

[DavidHuber-NOAA]

@KateFriedman-NOAA Sure will do. Thanks for the heads up.

[DavidHuber-NOAA]

@KateFriedman-NOAA I ran the workflow overnight through 12 cycles and the fit2obs jobs are running without error. However, I am only seeing fits for forecast hours 00 and 06 being generated in archive/EXP/fits. Also, the horiz/fcs directory in archive/ is not being populated.

[DavidHuber-NOAA]

@KateFriedman-NOAA Disregard, I am seeing out of memory errors on kjet. I'm going to investigate further.

[DavidHuber-NOAA]

Fixed by requesting all available memory on the node (--mem=0). I'm now seeing other hours populate in archive/ and horiz/fcs is populating as well.

[KateFriedman-NOAA]

Excellent! This is good resource info to know. When the vrfy job finally gets broken into separate jobs (hopefully soon) we'll need to know the resources to assign to the individual fit2obs job in the workflow. Thanks for the update!

[DavidHuber-NOAA]

@jack-woollen Was able to verify that a recent test of Fit2Obs generated outputs as expected on Jet. PR forthcoming.

[jack-woollen]

@KateFriedman-NOAA I reviewed the PR and have one problem. This morning I pushed changes to the repo that added back the .x on the executables, per Rahul. Some of the changes in the pull request overwrite some of those changes. Can you merge the new develop branch into your changes and make another PR? Sorry about the mixup!

[KateFriedman-NOAA]

@jack-woollen Oh, I just resolved conflicts and undid that in my branch (not knowing about that other commit). I'll redo that merge from develop and note it in this PR when done. Thanks for letting me know!