Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to uv for Python management #635

Merged
merged 17 commits into from
Nov 6, 2024
Merged

Switch to uv for Python management #635

merged 17 commits into from
Nov 6, 2024

Conversation

dfsnow
Copy link
Member

@dfsnow dfsnow commented Nov 6, 2024

This PR switches to using uv for environment and dependency management in our data architecture repo and Actions.

Locally, this (optionally) allows users to:

  • Use uv for virtual env management (via uv venv && source .venv/bin/activate) with a per-subdirectory virtualenv (i.e. dbt has its own dbt/.venv directory)
  • Install the pinned Python version (3.11) into the virtualenv using uv python install
  • Install packages into that virtualenv using uv pip install .

For Actions:

  • uv now handles caching and installing pip dependencies
  • The standard, GitHub-provided setup Action uses the pinned Python version in each project
  • We ignore virtual environments, instead using uv to install directly to the system library (just as we did before with plain pip)

Why make this switch?

  • uv is significantly faster when installing Python dependencies in Actions (10-100x faster)
  • It versions and isolates the dependencies used by each subproject (dbt/, socrata/, etc.) via uv.lock files
  • It pins the Python version used by each subproject and uses the same version within Actions via .python-version files

The downsides are:

  • It's something new to learn for users. However, the commands are simpler than standard venv and users can still use plain pip and venv if they want to
  • It's still slightly hacky to install into the system environment within Actions using uv

@dfsnow dfsnow force-pushed the dfsnow/switch-to-uv branch from b43bcdd to 4333014 Compare November 6, 2024 18:01
Comment on lines +15 to +17
cache-dependency-glob: |
${{ inputs.dbt_project_dir }}/pyproject.toml
${{ inputs.dbt_project_dir }}/uv.lock
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike before, we have to explicitly specify the cache keys for pip dependencies. This is actually probably a good thing insofar as it lets us isolate each of the subproject dependencies in their own cache.

Comment on lines +28 to +29
env:
UV_SYSTEM_PYTHON: 1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This env var is required if we want to install the pip dependencies into the system library rather than a uv virtual env. We could use the uv virtual env but we'd need to switch all the run commands from python $SCRIPT to uv run $SCRIPT and I thought this was easier.

- name: Install Python
uses: actions/setup-python@v5
with:
python-version-file: ${{ inputs.dbt_project_dir }}/.python-version
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having this file lets us keep our Python version synchronized with uv. If you want to use the Python version associated with the project locally, you just uv python install with the virtualenv active.

@dfsnow dfsnow force-pushed the dfsnow/switch-to-uv branch from 8fc1df2 to e841932 Compare November 6, 2024 18:47
@dfsnow dfsnow requested a review from jeancochrane November 6, 2024 18:48
@dfsnow dfsnow self-assigned this Nov 6, 2024
@dfsnow dfsnow marked this pull request as ready for review November 6, 2024 18:48
@dfsnow dfsnow requested a review from a team as a code owner November 6, 2024 18:48
Copy link
Contributor

@jeancochrane jeancochrane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice! I tested this out locally and it works great. No concerns from my end.

@dfsnow dfsnow merged commit a8def19 into master Nov 6, 2024
8 checks passed
@dfsnow dfsnow deleted the dfsnow/switch-to-uv branch November 6, 2024 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants