Added support for Databricks Apps in DABs

[andrewnester]

Changes

Now it's possible to configure new app resource in bundle and point it to the custom source_code_path location where Databricks App code is defined.

On databricks bundle deploy DABs will create an app. All consecutive databricks bundle deploy execution will update an existing app if there are any updated

On databricks bundle run <my_app> DABs will execute app deployment. If the app is not started yet, it will start the app first.

Bundle configuration

bundle:
  name: apps

variables:
  my_job_id:
    description: "ID of job to run app"
    lookup:
      job: "My Job"
  databricks_name:
    description: "Name for app user"
  additional_flags:
    description: "Additional flags to run command app"
    default: ""
  my_app_config:
    type: complex
    description: "Configuration for my Databricks App"
    default:
      command:
        - flask
        - --app
        - hello
        - run
        - ${var.additional_flags}
      env:
        - name: DATABRICKS_NAME
          value: ${var.databricks_name}

resources:
  apps:
    my_app:
      name: "anester-app" # required and has to be unique
      description: "My App"
      source_code_path: ./app # required and points to location of app code
      config: ${var.my_app_config}
      resources:
        - name: "my-job"
          description: "A job for app to be able to run"
          job:
            id: ${var.my_job_id}
            permission: "CAN_MANAGE_RUN"
      permissions:
        - user_name: "foo@bar.com"
          level: "CAN_VIEW"
        - service_principal_name: "my_sp"
          level: "CAN_MANAGE"

targets:
  dev:
    variables:
      databricks_name: "Andrew (from dev)"
      additional_flags: --debug
  
  prod:
    variables:
      databricks_name: "Andrew (from prod)"

Execution

1. databricks bundle deploy -t dev
2. databricks bundle run my_app -t dev

If app is started

✓ Getting the status of the app my-app
✓ App is in RUNNING state
✓ Preparing source code for new app deployment.
✓ Deployment is pending
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

If app is not started

✓ Getting the status of the app my-app
✓ App is in UNAVAILABLE state
✓ Starting the app my-app
✓ App is starting...
....
✓ App is starting...
✓ App is started!
✓ Preparing source code for new app deployment.
✓ Downloading source code from /Workspace/Users/...
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

Tests

Added unit and config tests + manual test.

--- PASS: TestAccDeployBundleWithApp (404.59s)
PASS
coverage: 36.8% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       405.035s        coverage: 36.8% of statements in ./...

[pietern]

That works for me.

The test now uses the TF notation, so it will fail when that is changed.

[pietern]

@fjakobs @ilyakuz-db FYI, this is a potential problem for source-linked deployments. We assume here that files are deployed under ${workspace.file_path} and then write a file into it. If we were to make this work for source-linked deployments, this would have to write a file to the source folder itself, which in turn means we'll get validation errors (we check that the user does not have both an app.yml and a configuration section in the bundle configuration for their app.

The fact that we have to think about this (and potentially work around it) illustrates the problem that this duality brings.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 1928
• Commit SHA: 7693f0add0ab88bd49c109047cedc75cc16edd42

Checks will be approved automatically on success.

[pietern]

For later; this duplicates the entries in SupportedResources. Can it be de-duped?

[pietern]

Could you regenerate the OpenAPI annotations as well? That will populate these descriptions.

Can be in a separate PR.

[andrewnester]

@pietern will do anyway as part of Go SDK upgarde