Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add singularity and slurm support #403

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
dba5d7d
Added code to support Singularity on HiperGator
suhasthegame Jan 3, 2024
75b8156
Testing for docker_entrypoint
suhasthegame Jan 9, 2024
2a9b195
Intermediate Commit --ignore
suhasthegame Feb 14, 2024
0705809
Added Dependencies --ignore
suhasthegame Feb 19, 2024
9957574
Ignore --all
suhasthegame Feb 21, 2024
5f2aac3
Added version number to girder worker
suhasthegame Feb 27, 2024
03b3717
Changed version
suhasthegame Feb 27, 2024
ba0c9c3
Reverted version back to original
suhasthegame Feb 28, 2024
10a4740
Intermediate Changes Please ignore
Apr 16, 2024
b1889c1
Temporary commit
Apr 30, 2024
b3429fe
Almost working
May 7, 2024
0ff3fec
Bug Fixes
May 14, 2024
27f790b
Addressed GPU config issue for plugins
May 16, 2024
a0f1b72
UF Progress
willdunklin May 21, 2024
9524112
Intermediate commit
Jun 24, 2024
31367c6
Fixed GPU and CPU allocation
Jul 1, 2024
3b08406
Added code to clean up tmp folders after job
Jul 10, 2024
8dadcc3
Merge branch 'migration' of https://github.com/suhasthegame/girder_wo…
willdunklin Jul 17, 2024
c7a9d3b
Split girder-worker-singularity into a seperate package
willdunklin Jul 22, 2024
8baadbc
Update singularity package dependencies
willdunklin Aug 1, 2024
66e99b3
Update singularity threads to track jobId
willdunklin Aug 21, 2024
6c326ee
Refactor singularity-slurm into separate package
willdunklin Aug 27, 2024
32c8e77
Format code
willdunklin Sep 16, 2024
2399313
Add slurm configuration settings
willdunklin Sep 16, 2024
d36a992
Merge branch 'master' of https://github.com/girder/girder_worker into…
willdunklin Sep 25, 2024
5daf316
Format code
willdunklin Sep 26, 2024
7d3783f
Undo test case removal
willdunklin Sep 26, 2024
971a918
Clean up dependencies
willdunklin Oct 2, 2024
bc129a9
Split slurm configuration setting out from worker
willdunklin Oct 2, 2024
2058ecc
Update slurm/singularity extensions to use scm versioning
willdunklin Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Intermediate Changes Please ignore
  • Loading branch information
User pinaki.sarder-web committed Apr 16, 2024
commit 10a47404b2c779af8f13cdb59d56b35de14d020a
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -11,3 +11,6 @@ __pycache__/
.tox/
.cache/
htmlcov/
build/
.eggs

29 changes: 29 additions & 0 deletions examples/plugin_example/gwexample/analyses/tasks.py
Original file line number Diff line number Diff line change
@@ -10,3 +10,32 @@ def fibonacci(n):
if n == 1 or n == 2:
return 1
return fibonacci(n-1) + fibonacci(n-2)

@app.task
# @argument('image_name', 'slide_name', 'path')
def nuclei(image_name, slide_name, path):
"running nuclei"
print(app, '++++++++++')
if path:
print('using arg path !!')
os.chdir(path)
else:
print('using default path !!')
os.chdir('/home/rc-svc-pinaki.sarder-web/digital_slide_archive/devops/singularity-minimal')
print('Current Path => ', os.getcwd())
path = os.getcwd()
flags = os.O_RDWR | os.O_CREAT
sif_image = os.open('sarderlab_histomicstk_latest.sif', flags)
sif_image_path = path + image_name if image_name else '/sarderlab_histomicstk_latest.sif'
slide_image = os.open(slide_name, flags)
slide_image_path = path + slide_name if slide_name else '18-142_PAS_1of6.svs'
output = os.open('Nuclei-outputNucleiAnnotationFile.anot', flags)
output_path = path + '/Nuclei-outputNucleiAnnotationFile.anot'
run_container = f'apptainer run --pwd /HistomicsTK/histomicstk/cli {sif_image} NucleiDetection {slide_image} {output}'
try:
res = subprocess.call(f'apptainer run --pwd /HistomicsTK/histomicstk/cli {sif_image_path} NucleiDetection {slide_image_path} {output_path}', shell=True, bufsize=0,stdin=subprocess.PIPE,stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding="UTF8")
print(res, '----1')

except Exception as e:
print(f"Exception occured {e}")

Loading