Clean databricks credentials in test/CI #969
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
tatiana
force-pushed
the
clean-databricks-credentials
branch
from
e975678 to
b5ce8a8
Compare
tatiana
commented
May 15, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #969 +/- ##
=======================================
Coverage 95.69% 95.69%
=======================================
Files 59 59
Lines 2880 2880
=======================================
Hits 2756 2756
Misses 124 124 ☔ View full report in Codecov by Sentry. |
tatiana
commented
May 15, 2024
tatiana
force-pushed
the
clean-databricks-credentials
branch
from
b22c2a0 to
88c844a
Compare
[2024-05-15T15:00:02.751+0000] {config.py:291} INFO - Creating temporary profiles.yml with use_mock_values=False at /tmp/tmpheo8f1ns/profiles.yml with the following contents:
default:
outputs:
dev:
connect_retries: 3
host: adb-2703548196728655.15.azuredatabricks.net
http_path: /sql/1.0/warehouses/368fe30b92228713
schema: cosmos_9097739441
token: '{{ env_var(''COSMOS_CONN_DATABRICKS_TOKEN'') }}'
type: databricks
target: dev
[2024-05-15T15:00:02.751+0000] {local.py:281} INFO - Trying to run dbtRunner with:
['run', '--models', 'customers', '--project-dir', '/tmp/tmpid0qcz29', '--profiles-dir', '/tmp/tmpheo8f1ns', '--profile', 'default', '--target', 'dev']
in /tmp/tmpid0qcz29
15:00:02 Running with dbt=1.7.14
15:00:02 Registered adapter: databricks=1.7.14
15:00:02 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 538 macros, 0 groups, 0 semantic models
15:00:02
15:00:04 Concurrency: 1 threads (target='dev')
15:00:04
15:00:04 1 of 1 START python table model cosmos_9097739441.customers .................... [RUN]
15:00:04 Unhandled error while executing target/run/jaffle_shop/models/customers.py
Databricks `http_path` or `cluster_id` of an all-purpose cluster is required for the `all_purpose_cluster` submission method.
15:00:04 1 of 1 ERROR creating python table model cosmos_9097739441.customers ........... [ERROR in 0.12s]
15:00:05
15:00:05 Finished running 1 table model in 0 hours 0 minutes and 2.13 seconds (2.13s).
15:00:05
15:00:05 Completed with 1 error and 0 warnings:
15:00:05
15:00:05 Databricks `http_path` or `cluster_id` of an all-purpose cluster is required for the `all_purpose_cluster` submission method.
As seen in:
https://github.com/astronomer/astronomer-cosmos/actions/runs/9097739441/job/25026817210
dosubot
bot
added
size:M
pankajastro
approved these changes
May 16, 2024
Merged
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
We're changing the credentials for the Databricks cluster used to validate dbt Python models, and I noticed there were a few unnecessary credentials set in the CI. This PR aims to clean them up, making it easier for us to manage them. All the tests passed in: https://github.com/astronomer/astronomer-cosmos/actions/runs/9109622858 The push tests failed because they relied on workflow configuration from the `main` branch.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We're changing the credentials for the Databricks cluster used to validate dbt Python models, and I noticed there were a few unnecessary credentials set in the CI. This PR aims to clean them up, making it easier for us to manage them.
All the tests passed in: https://github.com/astronomer/astronomer-cosmos/actions/runs/9109622858
The push tests failed because they relied on workflow configuration from the
mainbranch.