Update getting started guide with an example dbt project #778
Conversation
dosubot
bot
added
the
size:XS
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
dosubot
bot
added
area:docs
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #778 +/- ##
=======================================
Coverage 93.28% 93.28%
=======================================
Files 55 55
Lines 2502 2502
=======================================
Hits 2334 2334
Misses 168 168 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Hi @RNHTTR, thanks for pointing this out! There is room for us to improve the onboarding of Cosmos in Astro.
Some broader questions before we merge this PR (please, @jlaneve & @harels, share your thoughts!)
-
The last versions of dbt and Airflow don't conflict (https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes-local-conflicts.html#execution-modes-local-conflicts). Do we still want to encourage Astro users to create an independent virtualenv for dbt when using Cosmos? Providers/adapters may conflict, so this may be the safest option. Please let me know what your thoughts are.Unfortunately since the recent Airflow 2.8 release, we're back to having several conflicts between Airflow (2.8) & dbt (1.0-1.4, 1.6, 1.7): https://github.com/astronomer/astronomer-cosmos/pull/779/files So the answer to this question is yes, we should stick to the separate virtualenv for dbt for now. -
What is the goal of this getting started? Is it a step-by-step for someone trying out Cosmos and Astro for the first time? This Astro+Cosmos demo by @jlaneve is great:
https://github.com/astronomer/cosmos-demo
It is straightforward, self-contained, and uses Jaffle Shop. Would it make sense to use it / reference it?
| - A dbt project. The `jaffle shop example <https://github.com/dbt-labs/jaffle_shop>`_ is a great first step. To use the jaffle shop example to demo Cosmos... | ||
|
|
||
| - Clone the jaffle shop project using ``git clone`` into the ``/dags`` directory created by ``astro dev init`` | ||
| - Add ``dbt-postgres`` to the requirements.txt file created by ``astro dev init`` |
There was a problem hiding this comment.
@RNHTTR in this example, dbt is being installed in an independent Python virtual env, different from Airflow's Python deps.
Therefore, this dependency should be in the Create a virtualenv environment section, in the Dockerfile
There was a problem hiding this comment.
What is the goal of this getting started?
I didn't create it! I just found it. I'm likely in the minority for cosmos users, but I don't have dbt experience and I don't have a dbt project laying around, so the motivation for this change was to include a sample dbt project. @jlaneve 's cosmos-demo works nicely -- I hadn't seen that before. I think the docs should link to that though maybe?
The current "quickstart" link just goes to the cosmos docs
Based on: #778 Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
Based on: #778 Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
|
Closed after minor docs changes: #951 |
Based on: astronomer#778 Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
Description
I was going through the getting started guide, and it took me a little while to find a simple getting started project that I could use to demo Cosmos.
Related Issue(s)
Breaking Change?
Checklist