Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nf-validation cannot find the manifest path #312

Closed
Carldeboer opened this issue Jun 12, 2024 · 4 comments · Fixed by #315 or #318
Closed

nf-validation cannot find the manifest path #312

Carldeboer opened this issue Jun 12, 2024 · 4 comments · Fixed by #315 or #318
Assignees
@nebfield
Labels
bug Something isn't working

Comments

@Carldeboer
Copy link

Description of the bug

Hi all
I downloaded a recent nextflow via bioconda. Using the cluster's apptainer1.2.2.
The error happens when running the test command:
nextflow run pgscatalog/pgsc_calc -profile test,apptainer
But it fails with an error shown in the example below
The issue maybe related to #293
For context, I had a previous version of pgs_calc and nextflow with singularity working on this system, but a lot has changed since then.

Command used and terminal output

$ nextflow info
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
$ nextflow run pgscatalog/pgsc_calc -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Pulling pgscatalog/pgsc_calc ...
 downloaded from https://github.com/PGScatalog/pgsc_calc.git
Launching `https://github.com/pgscatalog/pgsc_calc` [mad_pesquet] DSL2 - revision: 45326ac066 [main]

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script 'my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details

Relevant files

.nextflow.log

Jun-12 15:03:58.141 [main] DEBUG nextflow.cli.Launcher - $> nextflow run pgscatalog/pgsc_calc -profile test,apptainer
Jun-12 15:03:58.366 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2
Jun-12 15:03:58.450 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=./my-pipelines2/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Jun-12 15:03:58.477 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-12 15:03:58.480 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-12 15:03:58.486 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jun-12 15:03:58.517 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-12 15:03:58.546 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: ./my-pipelines2/scm
Jun-12 15:03:58.601 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jun-12 15:03:58.618 [main] INFO  nextflow.cli.CmdRun - Pulling pgscatalog/pgsc_calc ...
Jun-12 15:03:58.630 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc/contents/nextflow.config
Jun-12 15:04:01.124 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc/contents/main.nf
Jun-12 15:04:01.262 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc
Jun-12 15:04:01.411 [main] DEBUG nextflow.scm.AssetManager - Pulling pgscatalog/pgsc_calc -- Using remote clone url: https://github.com/PGScatalog/pgsc_calc.git
Jun-12 15:04:06.719 [main] INFO  nextflow.cli.CmdRun -  downloaded from https://github.com/PGScatalog/pgsc_calc.git
Jun-12 15:04:06.806 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/nextflow.config
Jun-12 15:04:06.810 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/nextflow.config
Jun-12 15:04:06.829 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: ./my-pipelines2/secrets/store.json
Jun-12 15:04:06.833 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@4c635edc] - activable => nextflow.secret.LocalSecretsProvider@4c635edc
Jun-12 15:04:06.842 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,apptainer`
Jun-12 15:04:08.435 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, cfc_dev, uzl_omics, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, mccleary, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, vsc_calcua, pdc_kth, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, imperial, maestro, software_license, utd_europa, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, seg_globe, sanger, dkfz, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, azurebatch, hki, crukmi, csiro_petrichor, qmul_apocrita, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, self_hosted_runner, tufts, uw_hyak_pedslabs, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uod_hpc, fub_curta, uct_hpc, aws_tower, binac]
Jun-12 15:04:08.523 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jun-12 15:04:08.525 [main] DEBUG nextflow.cli.CmdRun - Launching `https://github.com/pgscatalog/pgsc_calc` [mad_pesquet] DSL2 - revision: 45326ac066 [main]
Jun-12 15:04:08.528 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-12 15:04:08.528 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-12 15:04:08.619 [main] DEBUG nextflow.Session - Session UUID: 2cdff23a-7844-4dc5-ae27-e79aac515d43
Jun-12 15:04:08.619 [main] DEBUG nextflow.Session - Run name: mad_pesquet
Jun-12 15:04:08.620 [main] DEBUG nextflow.Session - Executor pool size: 16
Jun-12 15:04:08.633 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-12 15:04:08.641 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-12 15:04:08.675 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 204776@login02 [206.12.119.244]
  CPUs: 16 - Mem: 187.5 GB (7.8 GB) - Swap: 0 (0)
Jun-12 15:04:08.719 [main] DEBUG nextflow.Session - Work-dir: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/work [nfs]
Jun-12 15:04:08.720 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/bin
Jun-12 15:04:08.746 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-12 15:04:08.760 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-12 15:04:08.885 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-12 15:04:08.927 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Jun-12 15:04:09.099 [main] DEBUG nextflow.Session - Session start
Jun-12 15:04:09.103 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/results/pipeline_info/execution_trace_2024-06-12_15-04-08.txt
Jun-12 15:04:09.122 [main] DEBUG nextflow.Session - Using default localLib path: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/lib
Jun-12 15:04:09.128 [main] DEBUG nextflow.Session - Adding to the classpath library: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/lib
Jun-12 15:04:09.896 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-12 15:04:11.023 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: latest
Jun-12 15:04:11.187 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-validation@1.1.3
Jun-12 15:04:11.897 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-validation-1.1.3.zip' in 'nf-validation-1.1.3'
Jun-12 15:04:11.905 [main] DEBUG nextflow.plugin.PluginUpdater - Failed atomic move for plugin /tmp/pf4j-update-downloader6398866501532599210/nf-validation-1.1.3 -> ./my-pipelines2/plugins/nf-validation-1.1.3 - Reason: /tmp/pf4j-update-downloader6398866501532599210/nf-validation-1.1.3 -> ./my-pipelines2/plugins/nf-validation-1.1.3: Invalid cross-device link - Fallback on safe move
Jun-12 15:04:12.183 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_6ee8db14b0ece2bc: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/main.nf
  Script_b4ded561a904b4b9: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf
Jun-12 15:04:12.184 [main] DEBUG nextflow.Session - Session aborted -- Cause: Cannot find the manifest path
Jun-12 15:04:12.200 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Cannot find the manifest path
        at org.pf4j.ManifestPluginDescriptorFinder.readManifestFromDirectory(ManifestPluginDescriptorFinder.java:142)
        at org.pf4j.ManifestPluginDescriptorFinder.readManifest(ManifestPluginDescriptorFinder.java:72)
        at org.pf4j.ManifestPluginDescriptorFinder.find(ManifestPluginDescriptorFinder.java:58)
        at org.pf4j.AbstractPluginManager.loadPluginFromPath(AbstractPluginManager.java:829)
        at org.pf4j.DefaultPluginManager.loadPluginFromPath(DefaultPluginManager.java:139)
        at nextflow.plugin.CustomPluginManager.loadPluginFromPath(CustomPluginManager.groovy:46)
        at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:98)
        at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:342)
        at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:198)
        at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:145)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:335)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:326)
        at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:453)
        at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:87)
        at nextflow.script.IncludeDef.loadPlugin0(IncludeDef.groovy:205)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:106)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_b4ded561a904b4b9.runScript(Script_b4ded561a904b4b9:7)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:222)
        at nextflow.script.IncludeDef.memoizedMethodPriv$loadModule0PathMapSession(IncludeDef.groovy:140)
        at nextflow.script.IncludeDef.access$0(IncludeDef.groovy)
        at nextflow.script.IncludeDef$__clinit__closure2.doCall(IncludeDef.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at groovy.lang.Closure.call(Closure.java:433)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at nextflow.script.IncludeDef.loadModule0(IncludeDef.groovy)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:112)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_6ee8db14b0ece2bc.runScript(Script_6ee8db14b0ece2bc:35)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)

System information

Nextflow version: 24.04.2
Hardware: HPC
Excutor: local
Container engine: Apptainer (tried singularity too, with similar results)
OS: some linux
pgs_calc: latest
@Carldeboer Carldeboer added the bug Something isn't working label Jun 12, 2024
@nebfield
Copy link
Member

Your local nextflow installation is trying to pull files into an odd directory:

my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf'

Perhaps try manually installing:

$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
$ cd pgsc_calc
$ nextflow run main.nf -profile test,apptainer

@Carldeboer
Copy link
Author

Doesn't seem to fix it:

$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
pgscatalog/pgsc_calc cloned to: pgsc_calc.5 ...
$ cd pgsc_calc/
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [sad_faraday] DSL2 - revision: c5aaae71cd

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script './workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details
$ cat .nextflow.log
Jun-13 09:20:49.846 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf -profile test,apptainer
Jun-13 09:20:49.958 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2
Jun-13 09:20:49.994 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=~/my-pipelines/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Jun-13 09:20:50.019 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-13 09:20:50.021 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-13 09:20:50.026 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jun-13 09:20:50.045 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-13 09:20:50.077 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/nextflow.config
Jun-13 09:20:50.081 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/nextflow.config
Jun-13 09:20:50.127 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: ~/my-pipelines/secrets/store.json
Jun-13 09:20:50.132 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6345e7b9] - activable => nextflow.secret.LocalSecretsProvider@6345e7b9
Jun-13 09:20:50.142 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,apptainer`
Jun-13 09:20:52.382 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, cfc_dev, uzl_omics, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, mccleary, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, vsc_calcua, pdc_kth, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, imperial, maestro, software_license, utd_europa, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, seg_globe, sanger, dkfz, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, azurebatch, hki, crukmi, csiro_petrichor, qmul_apocrita, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, self_hosted_runner, tufts, uw_hyak_pedslabs, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uod_hpc, fub_curta, uct_hpc, aws_tower, binac]
Jun-13 09:20:52.441 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jun-13 09:20:52.525 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [sad_faraday] DSL2 - revision: c5aaae71cd
Jun-13 09:20:52.527 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-13 09:20:52.527 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-13 09:20:52.641 [main] DEBUG nextflow.Session - Session UUID: b0bfe352-a33c-4121-a56f-210b5a4efa7d
Jun-13 09:20:52.641 [main] DEBUG nextflow.Session - Run name: sad_faraday
Jun-13 09:20:52.642 [main] DEBUG nextflow.Session - Executor pool size: 16
Jun-13 09:20:52.656 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-13 09:20:52.666 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-13 09:20:52.702 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 47562@login01 [206.12.119.243]
  CPUs: 16 - Mem: 187.5 GB (10.5 GB) - Swap: 0 (0)
Jun-13 09:20:52.760 [main] DEBUG nextflow.Session - Work-dir: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/work [nfs]
Jun-13 09:20:52.810 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-13 09:20:52.828 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-13 09:20:52.941 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-13 09:20:52.966 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Jun-13 09:20:53.166 [main] DEBUG nextflow.Session - Session start
Jun-13 09:20:53.170 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/results/pipeline_info/execution_trace_2024-06-13_09-20-52.txt
Jun-13 09:20:53.189 [main] DEBUG nextflow.Session - Using default localLib path: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/lib
Jun-13 09:20:53.195 [main] DEBUG nextflow.Session - Adding to the classpath library: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/lib
Jun-13 09:20:53.850 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-13 09:20:54.876 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: latest
Jun-13 09:20:55.140 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-validation@1.1.3
Jun-13 09:20:56.068 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-validation-1.1.3.zip' in 'nf-validation-1.1.3'
Jun-13 09:20:56.073 [main] DEBUG nextflow.plugin.PluginUpdater - Failed atomic move for plugin /tmp/pf4j-update-downloader11363677661301068091/nf-validation-1.1.3 -> ~/my-pipelines/plugins/nf-validation-1.1.3 - Reason: /tmp/pf4j-update-downloader11363677661301068091/nf-validation-1.1.3 -> ~/my-pipelines/plugins/nf-validation-1.1.3: Invalid cross-device link - Fallback on safe move
Jun-13 09:20:56.328 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_10147630fe6ec4e7: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/./workflows/pgsc_calc.nf
  Script_6ee8db14b0ece2bc: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/main.nf
Jun-13 09:20:56.328 [main] DEBUG nextflow.Session - Session aborted -- Cause: Cannot find the manifest path
Jun-13 09:20:56.340 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Cannot find the manifest path
        at org.pf4j.ManifestPluginDescriptorFinder.readManifestFromDirectory(ManifestPluginDescriptorFinder.java:142)
        at org.pf4j.ManifestPluginDescriptorFinder.readManifest(ManifestPluginDescriptorFinder.java:72)
        at org.pf4j.ManifestPluginDescriptorFinder.find(ManifestPluginDescriptorFinder.java:58)
        at org.pf4j.AbstractPluginManager.loadPluginFromPath(AbstractPluginManager.java:829)
        at org.pf4j.DefaultPluginManager.loadPluginFromPath(DefaultPluginManager.java:139)
        at nextflow.plugin.CustomPluginManager.loadPluginFromPath(CustomPluginManager.groovy:46)
        at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:98)
        at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:342)
        at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:198)
        at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:145)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:335)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:326)
        at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:453)
        at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:87)
        at nextflow.script.IncludeDef.loadPlugin0(IncludeDef.groovy:205)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:106)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_10147630fe6ec4e7.runScript(Script_10147630fe6ec4e7:7)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:222)
        at nextflow.script.IncludeDef.memoizedMethodPriv$loadModule0PathMapSession(IncludeDef.groovy:140)
        at nextflow.script.IncludeDef.access$0(IncludeDef.groovy)
        at nextflow.script.IncludeDef$__clinit__closure2.doCall(IncludeDef.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at groovy.lang.Closure.call(Closure.java:433)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at nextflow.script.IncludeDef.loadModule0(IncludeDef.groovy)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:112)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_6ee8db14b0ece2bc.runScript(Script_6ee8db14b0ece2bc:35)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)

Not sure if related, but our cluster appears to have masked ~ from refering to the home directory in scripts somehow. This might be something new. Consequently, directories named '~' keep appearing, which I never noticed before. Or maybe that's an apptainer thing?

$ rm pgsc_calc/ -rf      
$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
pgscatalog/pgsc_calc cloned to: pgsc_calc.5 ...
$ cd pgsc_calc/
$ ls
assets        conftest.py   LICENSE   nextflow.config       subworkflows
bin           docs          main.nf   nextflow_schema.json  tests
CITATIONS.md  environments  Makefile  README.md             workflows
conf          lib           modules   RELEASE-CHECKLIST.md
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [nostalgic_meninsky] DSL2 - revision: c5aaae71cd

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script './workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details
$ ls
'~'             conftest.py    main.nf                README.md              work
 assets         docs           Makefile               RELEASE-CHECKLIST.md   workflows
 bin            environments   modules                results
 CITATIONS.md   lib            nextflow.config        subworkflows
 conf           LICENSE        nextflow_schema.json   tests

@nebfield
Copy link
Member

Thanks for the info! I've set up a new branch that is a bit more explicit about plugins. Could you please try:

$ git clone -b nf-schema https://github.com/PGScatalog/pgsc_calc.git
$ cd pgsc_calc
$ nextflow run main.nf -profile test,apptainer

Masking home is interesting. By default nextflow pulls plugins and assets into $HOME/.nextflow/.... Maybe setting an environment variable will help:

$ export NXF_HOME=/path/to/normal/dir/
$ nextflow run main.nf -profile test,apptainer

@Carldeboer
Copy link
Author

Well that one seemed to work:

$ git clone -b nf-schema https://github.com/PGScatalog/pgsc_calc.git
Cloning into 'pgsc_calc'...
remote: Enumerating objects: 6202, done.
remote: Counting objects: 100% (6202/6202), done.
remote: Compressing objects: 100% (2216/2216), done.
remote: Total 6202 (delta 3629), reused 5883 (delta 3439), pack-reused 0
Receiving objects: 100% (6202/6202), 9.50 MiB | 14.99 MiB/s, done.
Resolving deltas: 100% (3629/3629), done.
Updating files: 100% (199/199), done.
$ cd pgsc_calc/
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [amazing_neumann] DSL2 - revision: a10e219a42

Downloading plugin nf-schema@2.0.0
Downloading plugin nf-prov@1.2.2


------------------------------------------------------
  pgscatalog/pgsc_calc v2.0.0-alpha.6
------------------------------------------------------
Core Nextflow options
  runName                   : amazing_neumann
  containerEngine           : apptainer
  launchDir                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc
  workDir                   : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/work
  projectDir                : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc
  userName                  : cdeboer
  profile                   : test,apptainer
  configFiles               :

Input/output options
  input                     : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/examples/samplesheet.csv
  scorefile                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/examples/scorefiles/PGS001229_22.txt
  outdir                    : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/results

Reference options
  ref_samplesheet           : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/reference.csv
  ld_grch37                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/high-LD-regions-hg19-GRCh37.txt
  ld_grch38                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/high-LD-regions-hg38-GRCh38.txt
  ancestry_checksums        : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/checksums.txt

Compatibility options
  target_build              : GRCh37

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

Other parameters
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use pgscatalog/pgsc_calc for your analysis please cite:

* The Polygenic Score Catalog
  https://doi.org/10.1101/2024.05.29.24307783
  https://doi.org/10.1038/s41588-021-00783-5

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/pgscatalog/pgsc_calc/blob/main/CITATIONS.md

[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[-        ] PGS…IBLE:PLINK2_RELABELPVAR -
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
[-        ] PGS…Y_SCORE:SCORE_AGGREGATE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 0 of 1
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 0 of 1
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (6)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
executor >  local (6)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (8)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[92/049bb2] PGS… (cineca chromosome 22) | 1 of 1 ✔
[46/5cbe07] PGS…:MATCH_COMBINE (cineca) | 1 of 1 ✔
[fe/bc4dab] PGS…effect type additive 0) | 1 of 1 ✔
[58/df8c62] PGS…CORE_AGGREGATE (cineca) | 1 of 1 ✔
[d3/20877d] PGS…T:SCORE_REPORT (cineca) | 1 of 1 ✔
[50/350f8c] PGS…UMPSOFTWAREVERSIONS (1) | 1 of 1 ✔
-[pgscatalog/pgsc_calc] Pipeline completed successfully-
Completed at: 14-Jun-2024 15:28:43
Duration    : 3m 50s
CPU hours   : (a few seconds)
Succeeded   : 8

And it also seems that the standard test with nextflow downloading pgsc_calc also works. Maybe my environment was messed up some how.

Thanks for your help!

@nebfield nebfield linked a pull request Jun 17, 2024 that will close this issue
Explicitly declare plugins in nextflow.config #315
Merged
@nebfield nebfield mentioned this issue Jun 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nebfield nebfield

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@Carldeboer @nebfield