[CT-3325] [Tech Debt][Spike+] Test dbt-core with Python 3.12

[aranke]

Housekeeping

• I am a maintainer of dbt-core

Short description

Python 3.12 was released last month, let's test that dbt-core works with this

https://www.python.org/downloads/release/python-3120/

Acceptance criteria

• Bundles can be created with Python 3.12
  • Removed since the current bundle integration tests are failing (link), will test with dbt-core 1.7.9 and 1.8.0b1
• dbt-common also supports python 3.12
• Update testing matrix to use python 3.12

Impact to Other Teams

• upstream dependencies that will also need to be made compatible with 3.12: dbt-adapters dbt-semantic-interfaces
• 1p adapters

Will backports be required?

No

Context

https://docs.getdbt.com/faqs/Core/install-python-compatibility

“New dbt minor versions will add support for new Python3 minor versions as soon as all dependencies can support it. In turn, dbt minor versions will drop support for old Python3 minor versions right before they reach end of life.”

[sdebruyn]

Deprecation notices appear when you run dbt init with dbt-postgres with Python 3.12:

dbt init -h
/Users/sam/.pyenv/versions/3.12.0/envs/myproject-dbt/lib/python3.12/site-packages/dateutil/tz/tz.py:37: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
Usage: dbt init [OPTIONS] [PROJECT_NAME]
...

[bgarcevic]

I get the warning when I run dbt debug - also using dbt-postgres and Python 3.12

dbt debug
DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

[edgarrmondragon]

I get the warning when I run dbt debug - also using dbt-postgres and Python 3.12

dbt debug
DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

FWIW those seem to come from dependencies rather than dbt-core itself.

[SorenSpicknall]

Hi folks - I identified a new problem with dbt-core and Python 3.12.1.

3.12 formally removed distutils, which was deprecated in 3.10. This removal wasn't fully implemented until 3.12.1, which released this week. The distutils import strtobool is used in dbt-core in two places - here and here. While some other functions of distutils were moved to other standard Python packages/functions/modules, strtobool was not and the PEP 632 migration guide suggests that project maintainers reimplement its functionality.

[lew181818]

Hello - is there an update on this issue? It doesn't look to be a major reimplementation in the dbt-core logic - isolated to this file I believe: dbt-core/blob/main/core/dbt/task/test.py. I believe this logic would just have to be implemented directly instead of using strtobool: https://docs.python.org/3.9/distutils/apiref.html#distutils.util.strtobool

We'd like to stay on the current version of Python within a reasonable timeframe to reduce exposure to vulnerabilities. Thanks so much!!

[aranke]

Hi all,

Python 3.12 support will be included with upcoming dbt-core releases 1.7.9 and 1.8.0b1.
I'm closing this issue for now, please stay tuned for the release.

Thanks!