Inject namespace files into the Execution context so that they are accessible from the execution's working directory

[anna-geller]

The namespaceFiles property

We want to add an optional namespaceFiles property, which can be set either to:

• a boolean — by default, this property is set to false; if set to true, we inject/load all Namespace Files from that specific namespace into the working directory of that task
• or a map with include and exclude keys and values being a list of strings. Those strings are regular expressions allowing to include or exclude specific files or directories.

WorkingDirectory as a first task to include namespaceFiles property

First, we want to add that property to the WorkingDirectory task. Then, we can add the same property to other tasks in plugins listed in the section below. Example with the include pattern to only load namespace files from the scripts directory:

id: include_scripts
namespace: dev

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    namespaceFiles:
      include:
        - "scripts/**"
    tasks:
      - id: myscript
        type: io.kestra.plugin.scripts.python.Commands
        commands:
          - python scripts/myscript.py

Simple example loading all namespace files into the task's working directory:

id: include_all_namespace_files
namespace: dev

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    namespaceFiles: true
    tasks:
      - id: myscript
        type: io.kestra.plugin.scripts.python.Commands
        commands:
          - python scripts/myscript.py

Problem to be solved

We want to make it easy to orchestrate entire projects containing:

• dbt core models and tests
• SqlMesh
• custom SQL queries
• custom Python scripts and entire modules
• Terraform modules
• Ansible projects
• Docker image builds
• CloudQuery ingestion sync files
• ...and many more.

Currently, users would need to store those in a Git repository and use a combination of WorkingDirectory and a git.Clone task.

To make that easier, we want to allow them to store all these projects as Namespace Files tied to a given namespace. Those files can be used as if you'd be working with a local directory. This way, it's possible to use as simple syntax as follows to orchestrate a Python script stored as a Namespace File in the directory scripts:

id: users
namespace: dev

tasks:
  - id: getUsers
    type: io.kestra.plugin.scripts.python.Commands
    docker:
      image: ghcr.io/kestra-io/pydata:latest
    commands:
      - python scripts/get_users.py
    namespaceFiles:
      include:
        - "scripts/**" 

Task types that will need the namespaceFiles property

Inject the namespace files only for the following tasks, as only those can benefit from them. The syntax may be implemented as described in this issue but on a task level #2341

id: myflow
namespace: dev

tasks:
  - id: dbt-build
    type: io.kestra.plugin.dbt.cli.DbtCLI
    runner: DOCKER
    docker:
      image: ghcr.io/kestra-io/dbt-duckdb
    commands:
      - dbt build
    namespaceFiles:
      include:
        - "dbt/**"  # fetch all files from internal storage from that repository
        - dbt_project.yml
        - profiles.yml
        - packages.yml
      exclude:
        - "dbt/*.py"

• WorkingDirectory

• https://kestra.io/plugins/plugin-script-python/tasks/io.kestra.plugin.scripts.python.commands

• https://kestra.io/plugins/plugin-script-r/tasks/io.kestra.plugin.scripts.r.commands

• https://kestra.io/plugins/plugin-script-julia/tasks/io.kestra.plugin.scripts.julia.commands

• https://kestra.io/plugins/plugin-script-shell/tasks/io.kestra.plugin.scripts.shell.commands

• https://kestra.io/plugins/plugin-script-powershell/tasks/io.kestra.plugin.scripts.powershell.commands

• https://kestra.io/plugins/plugin-script-node/tasks/io.kestra.plugin.scripts.node.commands

• https://kestra.io/plugins/plugin-malloy/tasks/io.kestra.plugin.malloy.cli

• https://kestra.io/plugins/plugin-aws/tasks/cli/io.kestra.plugin.aws.cli.awscli

• https://kestra.io/plugins/plugin-azure/tasks/cli/io.kestra.plugin.azure.cli.azcli

• https://kestra.io/plugins/plugin-gcp/tasks/cli/io.kestra.plugin.gcp.cli.gcloudcli

• https://kestra.io/plugins/plugin-dbt/tasks/cli/io.kestra.plugin.dbt.cli.dbtcli

• https://kestra.io/plugins/plugin-docker/tasks/io.kestra.plugin.docker.build

• TerraformCLI

• AnsibleCLI

• ModalCLI

• SQLMesh

• Dataform

• https://kestra.io/plugins/plugin-cloudquery/tasks/io.kestra.plugin.cloudquery.cloudquerycli

• https://kestra.io/plugins/plugin-fs/tasks/ssh/io.kestra.plugin.fs.ssh.command

Where namespace files could be considered in the future, but not for now

Unsure about the usefulness of namespace files in compression tasks, but those compression + encryption tasks could potentially take a path to namespace files:

• https://kestra.io/plugins/plugin-compress/tasks/io.kestra.plugin.compress.archivecompress
• https://kestra.io/plugins/plugin-compress/tasks/io.kestra.plugin.compress.filecompress
• https://kestra.io/plugins/plugin-crypto/tasks/openpgp/io.kestra.plugin.crypto.openpgp.encrypt

All upload tasks could, in theory, take a file from namespace files, but also here, I am not sure how useful that would be, as namespace files are meant for code, not data files:

• https://kestra.io/plugins/plugin-databricks/tasks/dbfs/io.kestra.plugin.databricks.dbfs.upload
• AWS S3 Upload
• Azure Blob Storage Upload
• GCP Storage Upload
• BigQuery Load
• BigQuery Upload

Singer doesn't need namespace files.
Soda also doesn't.