Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Ocean submit script #6094

Merged
merged 1 commit into from
Jul 26, 2024
Merged

Update Ocean submit script #6094

merged 1 commit into from
Jul 26, 2024

Conversation

Bronoulli
Copy link
Member

@Bronoulli Bronoulli commented Jun 16, 2024 •
edited
Loading

Proposed changes

Some changes that will bring Ocean's submit script up to date:
Added Ocean as a machine on develop

Updated Ocean's submit script to be compatible with spectre schedule

Removed Clang submit script since Ocean compiles with GCC.

Code review checklist

  • The code is documented and the documentation renders correctly. Run
    make doc to generate the documentation locally into BUILD_DIR/docs/html.
    Then open index.html.
  • The code follows the stylistic and code quality guidelines listed in the
    code review guide.
  • The PR lists upgrade instructions and is labeled bugfix or
    new feature if appropriate.

AlexCarpenter46
AlexCarpenter46 suggested changes Jul 1, 2024
View reviewed changes
support/SubmitScripts/Ocean.sh
bash -c "${SPECTRE_COMMAND} --input-file ${SPECTRE_INPUT_FILE}"
# Ocean is a supercomputer at Cal State, Fullerton.
# More information:
# https://github.com/sxs-collaboration/WelcomeToSXS/wiki/Ocean
Copy link
Contributor

@AlexCarpenter46 AlexCarpenter46 Jul 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This link currently doesn't go anywhere so you can just say it's hosted by Geoffrey Lovelace or add a wiki page for Ocean. That's probably a question for @geoffrey4444
edit after talking to Geoffrey: "Ocean is a supercomputer at Cal State, Fullerton. For more information, contact Geoffrey Lovelace."

@AlexCarpenter46
Copy link
Contributor

Works really well, thanks for changing it to work with the scheduler! :)

@Bronoulli Bronoulli requested a review from knelli2 July 1, 2024 23:58
nilsvu
nilsvu reviewed Jul 2, 2024
View reviewed changes
Copy link
Member

@nilsvu nilsvu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@geoffrey4444 do you want to take a look before we merge this?

support/SubmitScripts/Ocean.sh

# These commands load the relevant modules and cd into the run directory,
# creating it if it doesn't exist
module load ohpc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line not needed anymore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We typically don't load modules in the other submit scripts. They have to be loaded on the head node when you submit and they'll carry over.

support/SubmitScripts/Ocean.sh
export PATH=${SPECTRE_BUILD_DIR}/bin:$PATH

# Flag to stop blas in CCE from parallelizing without charm++
export OPENBLAS_NUM_THREADS=1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this one

@nilsvu nilsvu requested a review from geoffrey4444 July 2, 2024 03:27
knelli2
knelli2 reviewed Jul 2, 2024
View reviewed changes
Copy link
Contributor

@knelli2 knelli2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this @Bronoulli!

support/SubmitScripts/Ocean.sh

# These commands load the relevant modules and cd into the run directory,
# creating it if it doesn't exist
module load ohpc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We typically don't load modules in the other submit scripts. They have to be loaded on the head node when you submit and they'll carry over.

support/SubmitScripts/Ocean.sh
Comment on lines -65 to -69
SPECTRE_COMMAND="${SPECTRE_EXECUTABLE} +ppn 19 +pemap 0-18 +commap 19"

mpirun -np ${SLURM_JOB_NUM_NODES} --map-by ppr:1:node singularity exec \
/opt/ohpc/pub/containers/spectre_ocean.sif \
bash -c "${SPECTRE_COMMAND} --input-file ${SPECTRE_INPUT_FILE}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexCarpenter46 or @geoffrey4444 do you know if all this extra core mapping is still necessary? I wouldn't think so, but just to be sure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, I tested it out and it was using the correct number of workers and comm so it seems fine without it

AlexCarpenter46
AlexCarpenter46 suggested changes Jul 6, 2024
View reviewed changes
Copy link
Contributor

@AlexCarpenter46 AlexCarpenter46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only tested with 1 node, but our summer student tested with multiple nodes and it's not doing what we expect, it's duplicating Executable Output and only utilizing one of the nodes so we'll need to investigate that before merging.

@knelli2 knelli2 added the in progress Don't review, used for sharing code and getting feedback label Jul 8, 2024
@AlexCarpenter46 @Bronoulli
Ocean Submit Script Update
fa50321 
Co-authored-by: Kyle Pannone <kylepannone@csu.fullerton.edu>
@AlexCarpenter46 AlexCarpenter46 removed the in progress Don't review, used for sharing code and getting feedback label Jul 26, 2024
@knelli2 knelli2 enabled auto-merge July 26, 2024 18:26
@knelli2 knelli2 added the auto-merge GitHub's auto-merge has been enabled for this PR. label Jul 26, 2024
@knelli2 knelli2 merged commit a6a8ee4 into sxs-collaboration:develop Jul 26, 2024
22 checks passed
@knelli2 knelli2 added the clusters Super computer support (env, submit, machine files) label Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nilsvu nilsvu nilsvu left review comments

@AlexCarpenter46 AlexCarpenter46 AlexCarpenter46 approved these changes

@knelli2 knelli2 knelli2 approved these changes

@geoffrey4444 geoffrey4444 geoffrey4444 approved these changes

Assignees
No one assigned
Labels
auto-merge GitHub's auto-merge has been enabled for this PR. clusters Super computer support (env, submit, machine files)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Bronoulli @AlexCarpenter46 @nilsvu @geoffrey4444 @knelli2