FileNotFoundError when using dynamic parameters functionality in dbx launch

[JensvandeZande]

I am having trouble using dynamic parameters and configuration files while using the spark_python_task method. It seems that "dbx launch" does not resolve "file:fuse://" properly. This can be related to this issue. Let me sketch the situation.

I am using dbx's Task() abstract base class as described here for its various functionalities. In particular its configuration file reader methods. I use this configuration file to tell each task where to read and write the data. Next to the "--conf-file" parameter I have another parameter "--date" that I want to use as a dynamic parameter. For this, I used argument parser just like the Task._get_conf_file() method. However, when supplying parameters to the "--parameters" argument of "dbx launch" I need to also provide "--conf-file" parameter as the "parameters" argument in the task definition of the deployment.yml file gets overwritten when supplying dynamic parameters.

My deployment.yml looks like this:

environments:
  default:
    workflows:
      - name: "bronze2gold"
        job_clusters:
            - job_cluster_key: "my_cluster"
              <<: *basic-static-cluster

        tasks:
          - task_key: "bronze2silver"
            job_cluster_key: "my_cluster"
            spark_python_task:
              python_file: "file://project1/table1/tasks/bronze2silver.py"
              parameters: ["--conf-file", "file:fuse://project1/table1/conf/tasks/bronze2silver.yml"]

          # - task_key: "silver2gold"
          #   depends_on:
          #       - task_key: "bronze2silver"
          #   job_cluster_key: "my_cluster"
          #   spark_python_task:
          #     python_file: "file://project1/table1/tasks/silver2gold.py"
          #     parameters: ["--conf-file", "file:fuse://project1/table1/conf/tasks/silver2gold.yml"]

My cli call looks like this:

dbx deploy bronze2gold --deployment-file project1/table1/conf/deployment.yml
dbx launch bronze2gold --parameters='{"python_params":["--conf-file", "file:fuse://project1/table1/conf/tasks/bronze2silver.yml", "--date", "2022/11/02"]}'

I can see in the output that the arguments are being parsed, but I get the error FileNotFoundError: [Errno 2] No such file or directory: "file:fuse://project1/table1/conf/tasks/bronze2silver.yml".

I have tried "/dbfs/" and "dbfs://" instead of "file:fuse:" but I get the same error.

When I use "dbx execute" everything works:

dbx execute bronze2gold --task=bronze2silver--cluster-name="somePersonalCluster" --deployment-file project1/table1/conf/deployment.yml --parameters='{"parameters"["--conf-file", "file:fuse://project1/table1/conf/tasks/bronze2silver.yml", "--date", "2022/28/02"]}'

Can anyone explain to me what is happening and how I can solve my problem? Also, how should I solve it when I add a second task to my workflow that has a separate config file? See commented lines in deployment.yml. Per the documentation it is not possible to provide task specific parameters. Does this then mean we cannot use dbx's Task() abstract base class when using dynamic parameters?

KR Jens

[renardeinside]

hi @JensvandeZande ,
could you please provide the dbx deploy logs? I see that file:fuse is not resolved, but I cannot guess why without a log as well as the version of dbx you're using.

[JensvandeZande]

Hi @renardeinside,
Below is the deploy logs I get in the terminal. Im currently using dbx version 0.8.7

(innodata-py3.8) jens@LT414691:~/projects/innodata$ dbx deploy cv_vuro_source2gbr --deployment-file innodata/innolake/calculation_views/cv_vuro/conf/dta/deployment.yml
[dbx][2022-12-14 13:58:31.536] 🆗 Deployment file innodata/innolake/calculation_views/cv_vuro/conf/dta/deployment.yml exists and will be used for deployment
[dbx][2022-12-14 13:58:31.539] Starting new deployment for environment default
[dbx][2022-12-14 13:58:31.540] Using profile provided from the project file
[dbx][2022-12-14 13:58:31.541] Found auth config from provider ProfileEnvConfigProvider, verifying it
[dbx][2022-12-14 13:58:31.541] Found auth config from provider ProfileEnvConfigProvider, verification successful
[dbx][2022-12-14 13:58:31.542] Profile DEFAULT will be used for deployment
[dbx][2022-12-14 13:58:32.700] The workflow cv_vuro_source2gbr was selected for further operations
[dbx][2022-12-14 13:58:32.701] Following the provided build logic
[dbx][2022-12-14 13:58:32.702] 🐍 Building a Python-based project
[dbx][2022-12-14 13:58:32.703] 🧹 Standard package folder dist already exists, cleaning it before Python package build
[dbx][2022-12-14 13:58:33.165] ✅ Python-based project build finished
[dbx][2022-12-14 13:58:33.167] Locating package file
[dbx][2022-12-14 13:58:33.167] Package file located in: dist/innodata-0.0.1-py3-none-any.whl
[dbx][2022-12-14 13:58:33.512] Starting the traversal process
[dbx][2022-12-14 13:58:33.513] Processing libraries for task landing2gtr
[dbx][2022-12-14 13:58:33.513] ✅ Processing libraries for task landing2gtr - done
[dbx][2022-12-14 13:58:33.516] Processing libraries for task gtr2gbr
[dbx][2022-12-14 13:58:33.516] ✅ Processing libraries for task gtr2gbr - done
[dbx][2022-12-14 13:58:33.517] ⬆ Uploading local file innodata/innolake/calculation_views/cv_vuro/tasks/landing2gtr.py
[dbx][2022-12-14 13:58:34.088] ✅ Uploading local file innodata/innolake/calculation_views/cv_vuro/tasks/landing2gtr.py
[dbx][2022-12-14 13:58:34.091] ⬆ Uploading local file innodata/innolake/calculation_views/cv_vuro/conf/dta/tasks/landing2gtr.yml
[dbx][2022-12-14 13:58:34.403] ✅ Uploading local file innodata/innolake/calculation_views/cv_vuro/conf/dta/tasks/landing2gtr.yml
[dbx][2022-12-14 13:58:34.404] ⬆ Uploading local file dist/innodata-0.0.1-py3-none-any.whl
[dbx][2022-12-14 13:58:34.668] ✅ Uploading local file dist/innodata-0.0.1-py3-none-any.whl
[dbx][2022-12-14 13:58:34.671] ⬆ Uploading local file innodata/innolake/calculation_views/cv_vuro/tasks/gtr2gbr.py
[dbx][2022-12-14 13:58:34.871] ✅ Uploading local file innodata/innolake/calculation_views/cv_vuro/tasks/gtr2gbr.py
[dbx][2022-12-14 13:58:34.874] ⬆ Uploading local file innodata/innolake/calculation_views/cv_vuro/conf/dta/tasks/gtr2gbr.yml
[dbx][2022-12-14 13:58:35.005] ✅ Uploading local file innodata/innolake/calculation_views/cv_vuro/conf/dta/tasks/gtr2gbr.yml
[dbx][2022-12-14 13:58:35.009] Traversal process finished, all provided references were resolved
[dbx][2022-12-14 13:58:35.011] 🤖 Applying workflow definitions via API
[dbx][2022-12-14 13:58:35.275] 🪄  Updating existing workflow with name cv_vuro_source2gbr and id: 961911341177929
[dbx][2022-12-14 13:58:35.463] ✅ Applying workflow definitions - done
[dbx][2022-12-14 13:58:35.755] ✨ Deployment for environment default finished successfully