Add AWS ECS task run execution mode #1507
Conversation
…inside `DbtAwsEcsBaseOperator`
dosubot
bot
added
the
size:XL
✅ Deploy Preview for sunny-pastelito-5ecb04 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
dosubot
bot
added
the
area:execution
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1507 +/- ##
==========================================
- Coverage 97.05% 96.88% -0.18%
==========================================
Files 77 78 +1
Lines 4483 4552 +69
==========================================
+ Hits 4351 4410 +59
- Misses 132 142 +10 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Hi @CarlosGitto, this is a great contribution, thank you very much!
Some early feedback:
Code coverage
- Please, feel free to use
# pragma: no coverin the import statements that codecov mentioned that are untested. This is one of the few exceptions we open to using# pragma: no cover. This should increase the current 85.50% of diff hit (target 97.05%).
Pre-commit
There were some minor linting details that pre-commit complained:
ix end of files..............................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook
Fixing docs/getting_started/aws-container-run-job.rst
trim trailing whitespace......................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook
Fixing docs/getting_started/aws-container-run-job.rst
ruff..........................................................................Failed
- hook id: ruff
- files were modified by this hook
Found 1 error (1 fixed, 0 remaining).
black.........................................................................Failed
- hook id: black
- files were modified by this hook
reformatted cosmos/__init__.py
To learn more about how we use pre-commit, please, check https://github.com/astronomer/astronomer-cosmos/blob/c344eb4750f0269ecd267326373814abf56ec18e/docs/contributing.rst#pre-commit
I'm optimistic we'll be able to ship this as part of Cosmos 1.9 - we'll do a more thorough review in the upcoming days
| DbtRunOperationMixin, | ||
| DbtSeedMixin, | ||
| DbtSnapshotMixin, | ||
| DbtTestMixin, |
There was a problem hiding this comment.
We'll probably also want to import DbtSourceMixin and implement the correspondent class to avoid this type issue:
#1290
| super().__init__(*args, **kwargs) | ||
|
|
||
|
|
||
| class DbtTestAwsEcsOperator(DbtTestMixin, DbtAwsEcsBaseOperator): |
There was a problem hiding this comment.
Some operators can leverage the full_refresh configuration. You may want to follow something similar to #1501 to make sure this works for this execution mode.
| @@ -0,0 +1,175 @@ | |||
| .. title:: Getting Started with Astronomer Cosmos on AWS ECS | |||
|
|
|||
| Getting Started with Astronomer Cosmos on AWS ECS | |||
There was a problem hiding this comment.
In Addition to this excellent documentation, it would be great if you could also update the page that gives an overview on the execution modes available, referencing this newly introduced page:
https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html
Description
This PR introduces an AWS ECS operator that allows Cosmos to run dbt tasks in AWS ECS, pulling images from Amazon ECR. This ensures that dbt executions are consistent with the existing Airflow DAGs running in ECS.
Similar to the Azure Container Instance execution mode, this enhancement provides an ECS-native execution mode for Cosmos users.
What changes?
AwsEcsOperatorclasses (inheriting fromAbstractDbtBaseOperator) - Based on the originalEcsRunTaskOperatorRelated Issue(s)
resolves #1355
Breaking Change?
No, just an added feature
Checklist