Not authenticated to use blobs outside of Azure blob container working directory when using Azure Entra

[adamrtalbot]

Related to #5448 and #5444 but both issues refer to using Fusion, this one refers to using azcopy.

They are likely to be solved by the same method, since they have the same underlying challenge: how to pass authentication to the worker node (Batch) from Nextflow.

I seem to be able to recreate the issue without Fusion.

> nextflow run seqeralabs/nf-canary -r main --remoteFile az://igenomes/atacseq_samplesheet_custom.csv --run TEST_STAGE_REMOTE -w az://scidev-useast -c azure.config
N E X T F L O W  ~  version 24.10.3
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [c818260035]
Launching `https://github.com/seqeralabs/nf-canary` [magical_noyce] DSL2 - revision: 2ad4214f51 [main]
Uploading local `bin` scripts folder to az://scidev-useast/tmp/cf/bcc6a54f6a9dd33780a5251d956439/bin
[69/6f65a5] Submitted process > NF_CANARY:TEST_STAGE_REMOTE (1)
ERROR ~ Error executing process > 'NF_CANARY:TEST_STAGE_REMOTE (1)'

Caused by:
  Process `NF_CANARY:TEST_STAGE_REMOTE (1)` terminated with an error exit status (1)


Command executed:

  cat atacseq_samplesheet_custom.csv

Command exit status:
  1

Command output:
  (empty)

Work dir:
  az://scidev-useast/69/6f65a5549f7a3b2357312b12a28996

Container:
  docker.io/library/ubuntu:23.10

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit

azure.config:

process.executor = 'azurebatch'

fusion {
    enabled = false
}

azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = 'node'
        autoPoolMode = true
        allowPoolCreation = true
        deletePoolsOnCompletion = false
    }

    activeDirectory {
        servicePrincipalId = 'redacted'
        servicePrincipalSecret = 'redacted'
        tenantId = 'redacted'
    }
}

And with an access key:

To reiterate what's been said above, the error appears to stem from generateContainerSasWithActiveDirectory, which is only generating a relevant key for the working container and nothing else. Generating an account level SAS seems tricky (according to @alberto-miranda).

Originally posted by @adamrtalbot in #5444 (comment)

[adamrtalbot]

@alberto-miranda here is a method we could tell nextflow to pass the details to the worker task, this could help with #5444 and #5448.

It's pretty crude right now.

[alberto-miranda]

@alberto-miranda here is a method we could tell nextflow to pass the details to the worker task, this could help with #5444 and #5448.

It's pretty crude right now.

Apologies for the delay, but it is great that we are finally moving forward with this 😄. I'm happy to support for this in the Fusion side of things, so let's sync!

[alberto-miranda]

I wrote a couple PRs to support authenticating with Managed Identities with fusion v2.4 and the upcoming v2.5 and system-wide Managed Identities work out of the box (user-assigned require a single environment variable to be injected into worker nodes). So, we should be set if we can make Nextflow:

1. Automatically assign a system-wide MI to pool nodes; or
2. Automatically assign a user-assigned MI to pool nodes and inject an environment variable

(I personally prefer option 1)

[pditommaso]

Likely both should be supported

[adamrtalbot]

I'm not sure option 1 is supported by Azure Batch.

Option 2 is implemented as #5670