CI failing on cosmos main branch with error cannot load CLI commands from auth manager

[pankajastro]

CI: https://github.com/astronomer/astronomer-cosmos/actions/runs/12254777354/job/34186460687

Logs

File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/__init__.py", line 12, in <module>
    from cosmos.airflow.dag import DbtDag
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/airflow/dag.py", line 9, in <module>
    from airflow.models.dag import DAG
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8/lib/python3.11/site-packages/airflow/models/dag.py", line 94, in <module>
    from airflow.jobs.job import run_job
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8/lib/python3.11/site-packages/airflow/jobs/job.py", line 35, in <module>
    from airflow.serialization.pydantic.job import JobPydantic
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8/lib/python3.11/site-packages/airflow/serialization/pydantic/job.py", line 33, in <module>
    class JobPydantic(BaseModelPydantic):
  File "pydantic/main.py", line 221, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 504, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 434, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 544, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 568, in pydantic.fields.ModelField._set_default_and_type
  File "pydantic/fields.py", line 437, in pydantic.fields.ModelField.get_default
  File "pydantic/utils.py", line 694, in pydantic.utils.smart_deepcopy
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.RLock' object

[dosubot]

The error you're encountering is related to a TypeError that occurs when Pydantic's smart_deepcopy function attempts to deep copy an object that includes a _thread.RLock, which is not serializable. This issue arises during the serialization process in Airflow's job model handling.

To address this issue, you can consider the following approaches:

1. Identify and Remove Non-Serializable Objects: Review the Pydantic models or any data structures being used in the Airflow DAGs or tasks to identify where the _thread.RLock or similar non-serializable objects are being included. These should be removed or replaced with serializable alternatives.

2. Custom Serialization Logic: If the use of locks or similar objects is necessary, implement custom serialization logic to handle these objects. This might involve excluding them from the serialization process or using a proxy object that can be serialized.

3. Update Dependencies: Ensure that all dependencies, including Pydantic and Airflow, are up to date. Sometimes, newer versions of libraries include fixes for known serialization issues.

4. Refactor Code: If the issue is related to recent changes, such as the transition from dbt-sqlite to dbt-postgres, review the changes to ensure that they do not introduce non-serializable objects into the models or configurations.

By addressing the serialization of non-serializable objects, you should be able to resolve the TypeError and allow the CI process to complete successfully.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[pankajastro]

maybe related to pydantic version

apache-airflow 2.8.0 requires pydantic>=2.3.0, but you have pydantic 1.10.19 which is incompatible

[pankajastro]

Look like dbt-databricks is the root cause of this issue.

dbt-databricks has pin the pydantic https://github.com/databricks/dbt-databricks/blob/main/pyproject.toml#L33-L33
Added in PR: databricks/dbt-databricks#843

But newer version of Airflow require pydantic 2.0

[pankajastro]

For Airflow < 2.8 working fine: https://github.com/astronomer/astronomer-cosmos/actions/runs/12259335118/job/34201293598

But the newer version of Airflow require pydantic 2.0

[pankajastro]

PR: #1376