Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: setup build/publish with uv #573

Merged
merged 6 commits into from
Oct 31, 2024
Merged

ci: setup build/publish with uv #573

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7b818d2
ci: setup build/publish with uv
domoritz Oct 30, 2024
56e4457
chore: move complicated script into package.json
domoritz Oct 30, 2024
97aa3db
Update pyproject.toml
domoritz Oct 30, 2024
f1187be
chore: update dev specification
domoritz Oct 30, 2024
c8fbba4
ci: use workspace for uv and clean up gitignores
domoritz Oct 31, 2024
9314395
chore: combine ruff config
domoritz Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,29 +41,29 @@ jobs:
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
- uses: astral-sh/setup-uv@v3
with:
python-version-file: ".python-version"
cache: "pip"
cache-dependency-path: "**/pyproject.toml"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install hatch
version: ">=0.4.0"
enable-cache: true
cache-dependency-glob: "**/pyproject.toml"

- name: Build and lint Widget
run: |
cd packages/widget
hatch build
hatch fmt --check

uv build
uv run ruff check
uv run ruff format --check

- name: Build, lint, and test Server
run: |
cd packages/duckdb-server
hatch build
hatch fmt --check
hatch run test:cov

uv build
uv run ruff check
uv run ruff format --check
uv run mypy
uv run --with pytest-cov pytest --cov-report=term-missing --color=yes --cov=pkg
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on how you want to split out these, I believe you can run:

uv run ruff check
uv run ruff format --check

from the root of the repo.

I usally have a separate workflow for "Python Lint" e.g.,:

 python_lint:
    name: Lint Python
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v3
        with:
          python-version-file: ".python-version"
          cache: "pip"
          cache-dependency-path: "**/pyproject.toml"

      - name: Ruff
        run: |
          uv run ruff check
          uv run ruff format --check
          
  python_test:
    name: Test Python
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v3
        with:
          python-version-file: ".python-version"
          cache: "pip"
          cache-dependency-path: "**/pyproject.toml"

      - name: Widget
        run: |
          cd packages/widget
          uv build

      - name: Server
        run: |
          cd packages/duckdb-server
          uv run mypy
          uv run --with pytest-cov pytest --cov-report=term-missing --color=yes --cov=pkg

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just moved out up. I didn't set up a separate workflow since the python one is plenty fast.

setup-uv doesn't seem to like the python version file.

Screenshot 2024-10-31 at 10 43 57

Copy link
Collaborator

@manzt manzt Oct 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setup-uv doesn't seem to like the python version file.

Yeah, uv is a python installer and by default i think picks a "managed" mode where it will determine and install interpreters rather than relying on the system. If there is a .python-version root of the project (like we've done in this PR) it will respect that when running various uv commands. You can specify the version with UV_PYTHON= in the env if you want to test multple python versions in matrix.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could setup-python with a specific version and then setup-uv and run uv commands with --system to use the "system python" (i.e., whatever setup-python did), but i find the uv-only workflow most simple.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/manzt/anywidget/blob/478e742eaaf85d8fab68b006210680232863a488/.github/workflows/ci.yml#L56-L69

For example, I should change this to:

  TestPython:
    name: Python / Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version:
          - "3.8"
          - "3.9"
          - "3.10"
          - "3.11"
          - "3.12"
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with:
          run_install: true
      - uses: astral-sh/setup-uv@v3
        with:
          version: "0.4.x"
      - name: Run tests
        run: uv run --with pytest-cov pytest ./tests --color=yes --cov anywidget --cov-report xml
        env:
          UV_PYTHON: ${{ matrix.python-version }}


rust:
name: Test in Rust
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ To run local interactive examples:

To launch a local DuckDB server:

* Install [hatch](https://hatch.pypa.io/latest/install/), if not already present.
* Install [uv](https://docs.astral.sh/uv/), if not already present.
* Run `npm run server` to launch the [`duckdb-server`](https://github.com/uwdata/mosaic/tree/main/packages/duckdb-server). This runs the server in development mode, so the server will restart if you change its code.

To use Mosaic with DuckDB Python in Jupyter Notebooks:
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"build": "lerna run build",
"lint": "lerna run lint",
"test": "lerna run test",
"server": "cd packages/duckdb-server && hatch run serve",
"server": "cd packages/duckdb-server && npm run dev",
"server:rust": "cd packages/duckdb-server-rust && cargo run",
"server:node": "nodemon packages/duckdb/bin/run-server.js",
"dev": "vite",
Expand Down
2 changes: 2 additions & 0 deletions packages/duckdb-server/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,7 @@ dist
__pycache__
.ruff_cache
.pytest_cache
.mypy_cache
.coverage
.venv

16 changes: 10 additions & 6 deletions packages/duckdb-server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,25 @@ _Note:_ This package provides a local DuckDB server. To instead use DuckDB-WASM

## Installation and usage

We recommend running the server in an isolated environment with [pipx](https://github.com/pypa/pipx). For example, to directly run the server, use:
We recommend running the server in an isolated environment with [uvx](https://docs.astral.sh/uv/). For example, to directly run the server, use:

```bash
pipx run duckdb-server
uvx duckdb-server
```

Alternatively, you can install the server with `pip install duckdb-server`. Then you can start the server with `duckdb-server`.

## Developer Setup

We use [hatch](https://hatch.pypa.io/latest/) to manage our development setup.
We use [uv](https://docs.astral.sh/uv/) to manage our development setup.

Start the server in development with `hatch run serve`. The server restarts when you change the code.
Start the server with `uv run duckdb-server`. The server will not restart when the code changes.

To run the tests, use `hatch run test:cov`.
Start the server in development with `npm run dev`. The server restarts when you change the code.

Run `uv run ruff check --fix` and `uv run ruff format` to lint the code.

To run the tests, use `uv run pytest`.

To set up a local certificate for SSL, use https://github.com/FiloSottile/mkcert.

Expand Down Expand Up @@ -54,4 +58,4 @@ Loads the bundled results.

## Publishing

Run the build with `hatch build`. Then publish with `hatch publish`. We publish using tokens so when asked, set the username to `__token__` and then use your token as the password. Alternatively, create a [`.pypirc` file](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#create-an-account).
Run the build with `uv build`. Then publish with `uvx twine upload --skip-existing dist/*`. We publish using tokens so when asked, set the username to `__token__` and then use your token as the password. Alternatively, create a [`.pypirc` file](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#create-an-account).
8 changes: 5 additions & 3 deletions packages/duckdb-server/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
"version": "0.11.0",
"private": true,
"scripts": {
"prepublishOnly": "rimraf dist && mkdir dist && hatch run test:cov && hatch fmt --check && hatch build",
"publish": "hatch publish --user __token__",
"release": "npm run prepublishOnly && npm run publish"
"prepublishOnly": "uv run pytest && uv run ruff check && uv run ruff format --check",
"publish": "uv build && uvx twine upload --skip-existing dist/*",
Copy link
Collaborator

@manzt manzt Oct 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't mind this at all! We do something similar in anywidget. The funny thing is that uv is so flexible that you could probably publish a package to PyPI that just reads scripts from somewhere in the pyproject.toml and calls it (e.g., a python package called project-scripts that fulfills the following):

uvx project-scripts <script-name>

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, that's a nice hack. I hope uv will support scripts/shortcuts/commands/dev-scripts (whatever they will call it) eventually to clean this up.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. IIRC, there has been some discussion around this, but I don't remember the thread or where I saw it.

"release": "npm run prepublishOnly && npm run publish",
"dev": "uv run watchmedo auto-restart --pattern '*.py' --recursive --signal SIGTERM uv run duckdb-server",
"clean": "rimraf dist"
}
}
8 changes: 6 additions & 2 deletions packages/duckdb-server/pkg/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,9 @@ def handle_query(handler: Handler, con, cache, query):
json = retrieve(cache, query, partial(get_json, con))
handler.json(json)
elif command == "create-bundle":
create_bundle(con, cache, query.get("queries"), BUNDLE_DIR / query.get("name"))
create_bundle(
con, cache, query.get("queries"), BUNDLE_DIR / query.get("name")
)
handler.done()
elif command == "load-bundle":
load_bundle(con, cache, BUNDLE_DIR / query.get("name"))
Expand Down Expand Up @@ -164,7 +166,9 @@ async def http_handler(res, req):
{
"compression": CompressOptions.SHARED_COMPRESSOR,
"message": ws_message,
"drain": lambda ws: logger.warning(f"WebSocket backpressure: {ws.get_buffered_amount()}"),
"drain": lambda ws: logger.warning(
f"WebSocket backpressure: {ws.get_buffered_amount()}"
),
},
)

Expand Down
5 changes: 4 additions & 1 deletion packages/duckdb-server/pkg/tests/test_query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@


def test_key():
assert get_key("SELECT 1", "arrow") == "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.arrow"
assert (
get_key("SELECT 1", "arrow")
== "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.arrow"
)


def test_query_json():
Expand Down
48 changes: 22 additions & 26 deletions packages/duckdb-server/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "duckdb-server"
description = "A DuckDB server for Mosaic"
Expand All @@ -23,36 +19,36 @@ duckdb-server = "pkg.__main__:serve"
[project.urls]
homepage = "https://github.com/uwdata/mosaic"

[project.optional-dependencies]
dev = [
"watchdog[watchmedo]"
]

[tool.hatch.envs.default]
python = "3.11"
features = ["dev"]
installer = "uv"

[tool.hatch.envs.default.scripts]
serve = "watchmedo auto-restart --pattern '*.py' --recursive --signal SIGTERM python pkg/__main__.py"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.envs.test]
installer = "uv"
dependencies = [
"coverage[toml]",
"pytest",
"pytest-cov",
[tool.uv]
dev-dependencies = [
"watchdog[watchmehatch]",
"mypy>=1.11.1",
"pytest>=7.4.4",
"ruff>=0.6.1",
"types-ujson",
]

domoritz marked this conversation as resolved.
Show resolved Hide resolved
[tool.hatch.envs.test.scripts]
cov = 'pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=pkg'

[tool.hatch.build.targets.wheel]
packages = ["pkg"]

[tool.hatch.version]
path = "package.json"
pattern = "\"version\": \"(?P<version>.+?)\""

# https://github.com/charliermarsh/ruff
[tool.ruff]
lint.ignore = ["G004", "TRY301", "EM102", "EM101", "TRY003", "FBT002", "S608", "TRY002", "ARG001"]
src = ["pkg"]

[tool.ruff.lint]
pydocstyle = { convention = "numpy" }
ignore = ["G004", "TRY301", "EM102", "EM101", "TRY003", "FBT002", "S608", "TRY002", "ARG001"]

# https://mypy.readthedocs.io/en/stable/config_file.html
[tool.mypy]
files = "pkg/**/*.py"
strict = false
ignore_missing_imports = true
Loading
Loading