Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9

[lucaswiley]

Describe the bug

This issue originally started when I merged a change into production to begin using the Shopify DBT package. The merge broke because our prod machine was still on dbt 17. So when I went to upgrade I hit the below issues.

Currently, I get the below stack trace when running dbt --version while trying to upgrade to dbt 18.1. I imagine this has something to do with my brew/python setup environment but am not quite sure exactly how to debug after about a day of brute forcing various solutions. Thanks for your help.

Steps To Reproduce

1. Install brew
2. Install Git and DBT via Brew
3. Run dbt --version

Expected behavior

dbt --version to return the updated version

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 33, in <module>
    sys.exit(load_entry_point('dbt-core==0.18.1', 'console_scripts', 'dbt')())
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/main.py", line 11, in <module>
    import dbt.version
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 100, in <module>
    installed = get_installed_version()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 29, in get_installed_version
    return dbt.semver.VersionSpecifier.from_version_string(__version__)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/semver.py", line 100, in from_version_string
    return cls.from_dict(matched)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 618, in from_dict
    cls.validate(data)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 957, in validate
    schema = _validate_schema(cls)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 183, in _validate_schema
    schema = schema_cls.json_schema()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 942, in json_schema
    return cls._json_schema_recursive(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 913, in _json_schema_recursive
    schema = cls._schema_defs_from_cache(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 900, in _schema_defs_from_cache
    schema = cls._collect_json_schema(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 863, in _collect_json_schema
    properties[target_field], is_required = cls._get_field_schema(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 809, in _get_field_schema
    field_schema, required = cls._get_schema_for_type(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 783, in _get_schema_for_type
    raise ValidationError(f"Unable to create schema for '{type_name}'")
hologram.ValidationError: Unable to create schema for 'Optional'

The operating system you're using:
Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1028-aws x86_64)

The output of python --version:
Python 3.9.0 (I do have multiple installs of python which may be influencing the issue)

Additional context

$ python --version
Python 3.9.0

$ brew --version
Homebrew 2.5.8
Homebrew/linuxbrew-core (git revision 071501; last commit 2020-11-03)

My best assumption is the issue lies here with the billion python installs, but not sure.

$ whereis dbt
dbt: /home/linuxbrew/.linuxbrew/bin/dbt

$ whereis python
python: /usr/bin/python3.6 /usr/bin/python3.6-config /usr/bin/python2.7 /usr/bin/python2.7-config /usr/bin/python /usr/bin/python3.6m-config /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib/python3.8 /usr/lib/python2.7 /usr/lib/python3.7 /etc/python3.6 /etc/python2.7 /etc/python /usr/local/lib/python3.6 /usr/local/lib/python2.7 /usr/include/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/python /home/linuxbrew/.linuxbrew/bin/python3.9 /home/linuxbrew/.linuxbrew/bin/python3.9-config /home/airflow/onedrop-scripts/transform/dbt/dbt-env/bin/python /usr/share/man/man1/python.1.gz

$ whereis brew
brew: /home/linuxbrew/.linuxbrew/bin/brew

$ which dbt
/home/linuxbrew/.linuxbrew/bin/dbt

$ which python
/usr/bin/python

$ which brew
/home/linuxbrew/.linuxbrew/bin/brew

Thank you so much

[gshank]

It's possible that hologram version doesn't support Python 3.9. There were some changes in a recent hologram commit to support 3.9, but it hasn't been packaged yet and dbt support for 3.9 hasn't been done yet.

[lucaswiley]

Thanks, you're right. Resolved via instructions in this thread to use pip3 instead of brew: https://getdbt.slack.com/archives/C2JRRQDTL/p1594318582280200?thread_ts=1594253045.233300&cid=C2JRRQDTL

[mycaule]

Experimenting the same bug inside a GitHub Action (python 3.9).
@lucaswiley can you provide instructions on how to solve here (without having to log in to Slack) ? Thank you.

      - name: Setup Python
        uses: actions/setup-python@v2.1.4

      - name: Setup dbt
      - run: |
          pip install dbt
          dbt --version

[lucaswiley]

There are broken dependencies on the DBT brew installation for Python 3.9. Instead use pip3:

brew remove dbt
pip3 install virtualenv virtualenvwrapper
virtualenv foo
pip3 install dbt

[mrafayaleem]

I am using Conda and this is what I did to resolve this issue.

Downgrade python from 3.9 to 3.8:

conda install python=3.8

Do a force reinstall of dbt:

pip3 install --upgrade --force-reinstall dbt

Run dbt --version:

(dbt-poc) ➜  dbt-poc git:(master) dbt --version
installed version: 0.18.1
   latest version: 0.18.1

Up to date!

Plugins:
  - bigquery: 0.18.1
  - snowflake: 0.18.1
  - redshift: 0.18.1
  - postgres: 0.18.1

[mycaule]

This bug hasn't been solved, please reopen the ticket.

Here is the GitHub action workflow file running Python < 3.9

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ '3.6', '3.8' ]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/setup-python@v2.1.4

      - name: Setup dbt
        run: |
          python3 -m venv dbt-env
          source dbt-env/bin/activate
          pip3 install dbt
          dbt --version

[jtcohen6]

The latest version of dbt (v0.18.1) does not support python 3.9. We plan to add py39 support (#2822) for the next release of dbt (v0.19.0). If it proves straightforward, I'd also like to release a patch version (v0.18.2) that is compatible with python 3.9 as well.

In the meantime, I am going to close this issue as a duplicate of #2822.

[Startouf]

Hello, I still have this issue with sqlfluff, dbt 0.19 and python 3.9

pip3 --version
pip 21.1.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
python3 --version
Python 3.9.4
dbt --version
installed version: 0.19.1
sqlfluff --version
sqlfluff, version 0.5.6

When running sqlfluff lint with templater = dbt. WHen I remove template = dbt it runs, but jinja seems to be broken. Is that to be expected ?

[jtcohen6]

Hey @Startouf, dbt-core==0.19.1 is compatible with py39, but one of the plugins included by pip install dbt (dbt-snowflake==0.19.1) is not compatible with py39. The next minor version (v0.20) will be.

I'm not positive if that clarifies the jinja issue you're seeing, but suffice to say, it is expected that pip install dbt==0.19.1 + py39 don't yet work together. If you know which adapter you're using, however, and it isn't Snowflake, you can install that in a py39 environment just fine, e.g. pip install dbt-postgres.

[mike-seekwell]

confirmed this works in a clean py39 env

pip install dbt-postgres==0.19.1
dbt --version
installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - postgres: 0.19.1

[jeroenvlek]

Still encountering this issue for 0.19.1 with a fresh dbt install on Arch Linux and python 3.9. Creating a virtualenv with 3.7 fixed it for me.

[levietha92]

pip3 install --upgrade --force-reinstall dbt

thanks! this works for me 👍🏽 was trying to install dbt 0.18.0