-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework file I/O in mesh converters for thread-safety #437
Conversation
- Write graph.info files into same directory as output mesh files. - Remove related os.chdir() pattern from python wrapper to ensure calls are thread-safe.
To run additional tests (attached):
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, just requesting a few formatting fixes. I'll test this in compass tomorrow. Thanks for the nice example tests and instructions!
@mgduda, @akturner, @matthewhoffman and @mark-petersen, this work is needed as part of working on supporting parallel tasks via Parsl. If we make meshes in parallel with threads, it turns out that we can't be doing |
@dengwirda, could you add to the PR description that the approach for modifying the C++ code is to use the directory of the output file also as the directory for the other files (e.g. graph.info)? I think that's missing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm just going to approve by inspection. This makes sense and the changes look appropriate. Thank for working on this Darren.
@dengwirda, I tested this in I will approve as soon as the formatting issues are fixed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Built and ran successfully for me
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I compiled the c++ MpasCellCuller and MpasMeshConverter on grizzly with both gnu and intel and tested them individually with a QU240 mesh. Everything appears to work fine. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll second @xylar's suggestion to fix the whitespace in mpas_cell_culler.cpp
and mpas_mesh_converter.cpp
.
@mgduda, I went ahead and fixed the white space since it seems like @dengwirda hasn't had time. Could you give it another look to make sure you're okay with the changes otherwise? I would like to do a new release of MPAS-Tools soon and it would be nice if this were included. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving now that white space issues are fixed.
@mgduda, is this something you might have time to try out soon? If so, the timing would be good with other fixes we want to make for another MPAS-Tools release. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about the delay!
Thanks @mgduda! |
This PR reworks the way the C++
MpasMeshConverter.x
andMpasCellCuller.x
, as well as the MPAS-Toolsconversion.py
wrapper, handles file I/O to eliminate the need to change into local directories to account for hard-coded relative file names (e.g.graph.info
).Eliminating the
os.getcwd()
+os.chdir()
pattern inconversion.py
makes these routines thread-safe, allowing them to be called across parallel threads.These changes write all output files generated by the mesh-converter/culler (e.g.
graph.info
,cellMapForward.txt
, etc) into the directory defined by the output filename.