Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying out an easy fix for setting the custom tmp folder. #1908

Merged
merged 4 commits into from
Aug 10, 2023

Conversation

kockan
Copy link
Contributor

@kockan kockan commented Aug 9, 2023

Picard command-line programs that check for R installation do so in the customCommandLineValidation() method under parseArgs(). If the user wants to set a custom tmp directory via the --TMP_DIR flag, this will not go into effect since any usages of --TMP_DIR happen after parseArgs(). Therefore RExecutor will try to copy the R script that checks whether R is installed to the default system tmp folder and attempt to run the script from that location.

This is an attempt at fixing this timing issue by moving the work related to --TMP_DIR inside parseArgs(), before customCommandLineValidation() is ever called. I'm not sure whether this is the best long-term solution, but it seems to fix the issue.

Related issue: #1907

@kockan kockan marked this pull request as ready for review August 9, 2023 19:19
@kockan kockan requested a review from lbergelson August 9, 2023 19:19
Copy link
Member

@lbergelson lbergelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kockan This seems sane to me. I might put a comment mentioning the timing issue in it.

@lbergelson
Copy link
Member

@kockan Have you tested that this fixes the issue? Can you reproduce it/

@kockan
Copy link
Contributor Author

kockan commented Aug 9, 2023

@lbergelson Yes, it does. Here's a run with the main branch:

/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -jar /Users/kockan/tools/picard/build/libs/picard.jar CollectInsertSizeMetrics -I /Users/kockan/tools/picard/testdata/picard/sam/insert_size_metrics_test.sam -O test.txt -H test.pdf --TMP_DIR /Users/kockan/picard_tmp_test
INFO	2023-08-09 16:34:26	RExecutor	Executing R script via command: Rscript /var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/script16292753669425580283.R
INFO	2023-08-09 16:34:26	ProcessExecutor	[1] "Checking if R is installed"
Aug 09, 2023 4:34:26 PM com.intel.gkl.NativeLibraryLoader load
INFO: Loading libgkl_compression.dylib from jar:file:/Users/kockan/tools/picard/build/libs/picard.jar!/com/intel/gkl/native/libgkl_compression.dylib
Aug 09, 2023 4:34:26 PM com.intel.gkl.NativeLibraryLoader load
WARNING: Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression6055280550537139184.dylib: dlopen(/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression6055280550537139184.dylib, 0x0001): tried: '/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression6055280550537139184.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression6055280550537139184.dylib' (no such file), '/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression6055280550537139184.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))
Aug 09, 2023 4:34:26 PM com.intel.gkl.NativeLibraryLoader load
INFO: Loading libgkl_compression.dylib from jar:file:/Users/kockan/tools/picard/build/libs/picard.jar!/com/intel/gkl/native/libgkl_compression.dylib
Aug 09, 2023 4:34:26 PM com.intel.gkl.NativeLibraryLoader load
WARNING: Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression15062089678617180362.dylib: dlopen(/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression15062089678617180362.dylib, 0x0001): tried: '/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression15062089678617180362.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression15062089678617180362.dylib' (no such file), '/private/var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/libgkl_compression15062089678617180362.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))
[Wed Aug 09 16:34:26 EDT 2023] CollectInsertSizeMetrics --Histogram_FILE test.pdf --INPUT /Users/kockan/tools/picard/testdata/picard/sam/insert_size_metrics_test.sam --OUTPUT test.txt --TMP_DIR /Users/kockan/picard_tmp_test --DEVIATIONS 10.0 --MINIMUM_PCT 0.05 --METRIC_ACCUMULATION_LEVEL ALL_READS --INCLUDE_DUPLICATES false --ASSUME_SORTED true --STOP_AFTER 0 --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[Wed Aug 09 16:34:26 EDT 2023] Executing as kockan@wmda4-778 on Mac OS X 13.4 aarch64; OpenJDK 64-Bit Server VM 17.0.7+7; Deflater: Jdk; Inflater: Jdk; Provider GCS is available; Picard version: Version:3.1.0-SNAPSHOT
INFO	2023-08-09 16:34:26	RExecutor	Executing R script via command: Rscript /var/folders/tk/xx6p5mgs6fxgxm9gcwh2td700000gq/T/script9335001090116671995.R /Users/kockan/tools/picard/test.txt /Users/kockan/tools/picard/test.pdf insert_size_metrics_test.sam
INFO	2023-08-09 16:34:26	ProcessExecutor	null device 
INFO	2023-08-09 16:34:26	ProcessExecutor	          1 
[Wed Aug 09 16:34:26 EDT 2023] picard.analysis.CollectInsertSizeMetrics done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=41943040

Process finished with exit code 0

vs. a run with the feature branch:

/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -jar /Users/kockan/tools/picard/build/libs/picard.jar CollectInsertSizeMetrics -I /Users/kockan/tools/picard/testdata/picard/sam/insert_size_metrics_test.sam -O test.txt -H test.pdf --TMP_DIR /Users/kockan/picard_tmp_test
INFO	2023-08-09 16:37:05	RExecutor	Executing R script via command: Rscript /Users/kockan/picard_tmp_test/script17597955685019068752.R
INFO	2023-08-09 16:37:05	ProcessExecutor	[1] "Checking if R is installed"
Aug 09, 2023 4:37:05 PM com.intel.gkl.NativeLibraryLoader load
INFO: Loading libgkl_compression.dylib from jar:file:/Users/kockan/tools/picard/build/libs/picard.jar!/com/intel/gkl/native/libgkl_compression.dylib
Aug 09, 2023 4:37:05 PM com.intel.gkl.NativeLibraryLoader load
WARNING: Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/Users/kockan/picard_tmp_test/libgkl_compression1049026855005558731.dylib: dlopen(/Users/kockan/picard_tmp_test/libgkl_compression1049026855005558731.dylib, 0x0001): tried: '/Users/kockan/picard_tmp_test/libgkl_compression1049026855005558731.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/kockan/picard_tmp_test/libgkl_compression1049026855005558731.dylib' (no such file), '/Users/kockan/picard_tmp_test/libgkl_compression1049026855005558731.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))
Aug 09, 2023 4:37:05 PM com.intel.gkl.NativeLibraryLoader load
INFO: Loading libgkl_compression.dylib from jar:file:/Users/kockan/tools/picard/build/libs/picard.jar!/com/intel/gkl/native/libgkl_compression.dylib
Aug 09, 2023 4:37:05 PM com.intel.gkl.NativeLibraryLoader load
WARNING: Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/Users/kockan/picard_tmp_test/libgkl_compression192450232016518128.dylib: dlopen(/Users/kockan/picard_tmp_test/libgkl_compression192450232016518128.dylib, 0x0001): tried: '/Users/kockan/picard_tmp_test/libgkl_compression192450232016518128.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/kockan/picard_tmp_test/libgkl_compression192450232016518128.dylib' (no such file), '/Users/kockan/picard_tmp_test/libgkl_compression192450232016518128.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))
[Wed Aug 09 16:37:05 EDT 2023] CollectInsertSizeMetrics --Histogram_FILE test.pdf --INPUT /Users/kockan/tools/picard/testdata/picard/sam/insert_size_metrics_test.sam --OUTPUT test.txt --TMP_DIR /Users/kockan/picard_tmp_test --DEVIATIONS 10.0 --MINIMUM_PCT 0.05 --METRIC_ACCUMULATION_LEVEL ALL_READS --INCLUDE_DUPLICATES false --ASSUME_SORTED true --STOP_AFTER 0 --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[Wed Aug 09 16:37:06 EDT 2023] Executing as kockan@wmda4-778 on Mac OS X 13.4 aarch64; OpenJDK 64-Bit Server VM 17.0.7+7; Deflater: Jdk; Inflater: Jdk; Provider GCS is available; Picard version: Version:3.1.0-1-g4cb25b466-SNAPSHOT
INFO	2023-08-09 16:37:06	RExecutor	Executing R script via command: Rscript /Users/kockan/picard_tmp_test/script2092906871635032078.R /Users/kockan/tools/picard/test.txt /Users/kockan/tools/picard/test.pdf insert_size_metrics_test.sam
INFO	2023-08-09 16:37:06	ProcessExecutor	null device 
INFO	2023-08-09 16:37:06	ProcessExecutor	          1 
[Wed Aug 09 16:37:06 EDT 2023] picard.analysis.CollectInsertSizeMetrics done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=41943040

Process finished with exit code 0

@kockan
Copy link
Contributor Author

kockan commented Aug 9, 2023

@lbergelson I am also worried about the "instantiating the intel libraries" part, though this comes before any of that really happens, so unless those require the default system tmp dir, it should be fine, but happy to look at alternatives if it actually might break those.

@lbergelson
Copy link
Member

@kockan I had misread it when I posted (and deleted) that comment. I think it should be good. It happens before those still.

@kockan kockan merged commit 9206efc into master Aug 10, 2023
6 checks passed
@kockan kockan deleted the can_tmpdir_fix branch August 10, 2023 16:43
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants