This template lets you get started using Dagster Cloud with a Hybrid agent.
Note It is recommended to first deploy the example project included in this repository and then replace it with your own Dagster project.
What you need to start using this template:
-
A Dagster Cloud account set up for Hybrid deployments.
-
A Hybrid agent up and running.
-
A Docker container registry accessible from the hybrid agent and from your GitHub workflows.
Click the Use this Template
button and provide details for your new repo.
The dagster_cloud.yaml
file defines the configuration for building and deploying your code locations. For the quickstart_etl
, specify the Docker registry in the registry:
key:
Edit the GitHub Workflow at
.github/workflows/dagster-cloud-deploy.yml
to configure your Dagster Cloud account as well as Docker registry access.
-
Set the
DAGSTER_CLOUD_ORGANIZATION
environment to the name of your Dagster Cloud organization. If you access Dagster Cloud at https://acme.dagster.cloud then your organization is acme. -
Set the
IMAGE_REGISTRY
environment to the same registry specified indagster_cloud.yaml
: -
Uncomment one of the options for logging into the Docker registry:
Set up secrets on your newly created repository by navigating to the Settings
panel in your repo, clicking Secrets
on the sidebar, and selecting Actions
. Then, click New repository secret
. The following secrets are needed.
Name | Description |
---|---|
DAGSTER_CLOUD_API_TOKEN |
An agent token, for more details see the Dagster Cloud docs. |
Docker access secrets | Depending on which Docker registry you are using, you must define the credentials listed in the workflow file. |
Here is an example screenshot showing the secrets for AWS ECR.
At this point, the workflow run should complete successfully and you should see the quickstart_etl
location in https://dagster.cloud. If builds are failing, ensure that your secrets are properly set up.
Once you have the quickstart_etl
example deployed, you can replace the sample code with your Dagster project. You will then need to update the dagster_cloud.yaml
file:
-
Update
dagster_cloud.yaml
. See documentation for details. -
If you have more than one code location, duplicate the
build-docker-image
and the"ci set-build-output"
steps indagster-cloud-deploy.yaml
for the new code locations.
Branch Deployments are enabled by default. To disable them comment out the for your Hybrid agent, comment out the pull_request
section in dagster_cloud.yaml
:
A standard Dockerfile
is included in this project and used to build the quickstart_etl
. This file is used by the build-push-action
:
dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml
Lines 123 to 129 in fa0a0d3
To customize the Docker image, modify the build-push-action
and update the Dockerfile
as needed:
- To use a different directory for the
Dockerfile
, use thecontext:
input. See build-push-action for more details. - To reuse a Docker image for multiple code locations, use a single
build-push-action
and multiple"ci set-build-output"
steps, all using the same image tag.
The ci-init
step accepts a location_names
input string containing a JSON list of location names to be deployed. To deploy only specific locations provide the location_names:
input, for example:
- name: Initialize build session
id: ci-init
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/ci-init@v0.1
with:
project_dir: ${{ env.DAGSTER_PROJECT_DIR }}
dagster_cloud_yaml_path: ${{ env.DAGSTER_CLOUD_YAML_PATH }}
deployment: 'prod'
location_names: '["quickstart_etl1", "location2"]' # only deploy these two locations