Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using conda environment - java.io.IOException: Function not implemented #1771

Closed
nebfield opened this issue Oct 21, 2020 · 2 comments
Closed

Using conda environment - java.io.IOException: Function not implemented #1771

nebfield opened this issue Oct 21, 2020 · 2 comments
Milestone
v20.10.0

Comments

@nebfield
Copy link

Bug report

Expected behavior and actual behavior

I would like to use the conda directive in a process running on a cluster environment.

Whenever I specify the conda directive Nextflow returns an error:

Error executing process > 'multiqc'

Caused by:
  java.io.IOException: Function not implemented

Steps to reproduce the problem

  • Create a conda environment on a shared storage environment (conda create --prefix /mnt/scratch2/users/bwingfield/conda/mqc -c bioconda -c conda-forge python=3.6)
  • Activate in a terminal works OK (conda activate /mnt/scratch2/users/bwingfield/conda/mqc) ✅
  • Using conda in a nextflow process fails:
Channel.from(1,2,3).set{ test_ch }

process multiqc {
    echo true
    executor 'slurm'
    conda '/mnt/scratch2/users/bwingfield/conda/mqc'

    input:
    val x from test_ch

    shell:
    '''
    echo !{x}
    '''
}

Program output

A relevant excerpt from .nextflow.log:

Oct-21 18:15:49.738 [Actor Thread 2] ERROR nextflow.processor.TaskProcessor - Error executing process > 'multiqc'

Caused by:
  java.io.IOException: Function not implemented

java.lang.IllegalStateException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Function not implemented
        at nextflow.conda.CondaCache.getCachePathFor(CondaCache.groovy:334)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:167)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:70)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at nextflow.processor.TaskRun.memoizedMethodPriv$getCondaEnv(TaskRun.groovy:568)
        at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:351)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:64)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
        at nextflow.processor.TaskRun$_closure2.doCall(TaskRun.groovy)
        at nextflow.processor.TaskRun$_closure2.doCall(TaskRun.groovy)
        at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction$1.provide(Memoize.java:139)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:123)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at groovy.lang.Closure.call(Closure.java:399)
        at java_util_concurrent_Callable$call.call(Unknown Source)
        at nextflow.processor.TaskRun.getCondaEnv(TaskRun.groovy)
        at nextflow.processor.TaskRun$getCondaEnv$4.call(Unknown Source)
        at nextflow.processor.TaskProcessor.createTaskHashKey(TaskProcessor.groovy:1955)
        at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
        at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:581)
        at nextflow.processor.InvokeTaskAdapter.call(InvokeTaskAdapter.groovy:59)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 Caused by: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Function not implemented
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1011)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:994)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
        at nextflow.conda.CondaCache$_getLazyImagePath_closure2.doCall(CondaCache.groovy:311)
        at nextflow.conda.CondaCache$_getLazyImagePath_closure2.call(CondaCache.groovy)
        at groovyx.gpars.dataflow.LazyDataflowVariable$1.run(LazyDataflowVariable.java:70)
        ... 3 common frames omitted
Caused by: java.io.IOException: Function not implemented
        at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
        at nextflow.file.FileMutex.lock(FileMutex.groovy:93)
        at nextflow.conda.CondaCache.createLocalCondaEnv(CondaCache.groovy:220)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        ... 12 common frames omitted

Environment

  • Nextflow version: nextflow version 20.07.1.5412
  • Java version:
    java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
  • Operating system: System: Linux 3.10.0-957.5.1.el7.x86_64
  • Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

Additional context

This issue doesn't happen if I specify a conda environment in my home directory, so I think it's related to the shared storage area.
pditommaso added a commit that referenced this issue Oct 23, 2020
@pditommaso
Fix Skip FS lock when Conda dir exist #1771
e4529d4
@pditommaso pditommaso added this to the v20.10.0 milestone Oct 23, 2020
@pditommaso
Copy link
Member

I guess this happens because your file system does not support file lock primitive. But it's fair to assume that file locking is not required when the conda env already exists.

I've push a patched for this that will be included in the next release. Thanks for reporting.

@jammywwh
Copy link

Dear pditommaso,
I met the same issue, is that solved in new version? The version of nextflow installed in my Linux machine is 21.04.3.5560.
Look forward your reply. Thank you.

I guess this happens because your file system does not support file lock primitive. But it's fair to assume that file locking is not required when the conda env already exists.

I've push a patched for this that will be included in the next release. Thanks for reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v20.10.0
Development

No branches or pull requests
3 participants
@pditommaso @nebfield @jammywwh