-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Recommend poetry in docs #12475
Recommend poetry in docs #12475
Changes from 2 commits
a7e6c3e
ffe1108
833056d
72fc3d4
2c9dc49
8f167f4
c55d9cc
a8da697
2ed6364
0649ac0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Strongly recommend `poetry` for development. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -48,19 +48,27 @@ can find many good git tutorials on the web. | |
|
||
# 4. Install the dependencies | ||
|
||
Once you have installed Python 3 and added the source, please open a terminal and | ||
setup a *virtualenv*, as follows: | ||
Synapse uses the [poetry](https://python-poetry.org/) project to manage its dependencies | ||
and development environment. Once you have installed Python 3 and added the | ||
source, you should install `poetry`. We recommend | ||
DMRobertson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Of their installation methods, we recommend installing `poetry` using `pipx`, | ||
|
||
```shell | ||
pip install --user pipx | ||
pipx install poetry | ||
``` | ||
|
||
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation) | ||
for more details. | ||
|
||
Next, open a terminal and install dependencies as follows: | ||
|
||
```sh | ||
cd path/where/you/have/cloned/the/repository | ||
python3 -m venv ./env | ||
source ./env/bin/activate | ||
pip install wheel | ||
pip install -e ".[all,dev]" | ||
pip install tox | ||
poetry install --extras all | ||
``` | ||
|
||
This will install the developer dependencies for the project. | ||
This will install the runtime and developer dependencies for the project. | ||
|
||
|
||
# 5. Get in touch. | ||
|
@@ -117,11 +125,10 @@ The linters look at your code and do two things: | |
- ensure that your code follows the coding style adopted by the project; | ||
- catch a number of errors in your code. | ||
|
||
The linter takes no time at all to run as soon as you've [downloaded the dependencies into your python virtual environment](#4-install-the-dependencies). | ||
The linter takes no time at all to run as soon as you've [downloaded the dependencies](#4-install-the-dependencies). | ||
|
||
```sh | ||
source ./env/bin/activate | ||
./scripts-dev/lint.sh | ||
poetry run ./scripts-dev/lint.sh | ||
``` | ||
|
||
Note that this script *will modify your files* to fix styling errors. | ||
|
@@ -131,15 +138,13 @@ If you wish to restrict the linters to only the files changed since the last com | |
(much faster!), you can instead run: | ||
|
||
```sh | ||
source ./env/bin/activate | ||
./scripts-dev/lint.sh -d | ||
poetry run ./scripts-dev/lint.sh -d | ||
``` | ||
|
||
Or if you know exactly which files you wish to lint, you can instead run: | ||
|
||
```sh | ||
source ./env/bin/activate | ||
./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder | ||
poetry run ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder | ||
``` | ||
|
||
## Run the unit tests (Twisted trial). | ||
|
@@ -148,16 +153,14 @@ The unit tests run parts of Synapse, including your changes, to see if anything | |
was broken. They are slower than the linters but will typically catch more errors. | ||
|
||
```sh | ||
source ./env/bin/activate | ||
trial tests | ||
poetry run trial tests | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does poetry not have a concept of activating the virtualenv? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The closest thing is There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (I say "but" because it means it's not a 1:1 replacement) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess you can still just manually activate the virtualenv in the same way, if you wanted to. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we include a cross-reference to the direnv bits somewhere? (Or mention |
||
``` | ||
|
||
If you wish to only run *some* unit tests, you may specify | ||
another module instead of `tests` - or a test class or a method: | ||
|
||
```sh | ||
source ./env/bin/activate | ||
trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite | ||
poetry run trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite | ||
``` | ||
|
||
If your tests fail, you may wish to look at the logs (the default log level is `ERROR`): | ||
|
@@ -169,7 +172,7 @@ less _trial_temp/test.log | |
To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`: | ||
|
||
```sh | ||
SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests | ||
SYNAPSE_TEST_LOG_LEVEL=DEBUG poetry run trial tests | ||
``` | ||
|
||
By default, tests will use an in-memory SQLite database for test data. For additional | ||
|
@@ -180,7 +183,7 @@ database state to be stored in a file named `test.db` under the trial process' | |
working directory. Typically, this ends up being `_trial_temp/test.db`. For example: | ||
|
||
```sh | ||
SYNAPSE_TEST_PERSIST_SQLITE_DB=1 trial tests | ||
SYNAPSE_TEST_PERSIST_SQLITE_DB=1 poetry run trial tests | ||
``` | ||
|
||
The database file can then be inspected with: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,138 @@ | ||
# Managing dependencies with Poetry | ||
|
||
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be worth some explanatory/additional background here. E.g.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TBH I think the only item from this list I might add would be the link to the issue where you recommend poetry-the rest feel a little more like blog-post material vs docs material and may risk overwhelming the user. That being said, this doc is already great and I think people will find it quite helpful. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've added a "background section" which address some of this. It might be overwhelming but I think it's useful to have examples from the project file and the lockfile. I didn't mention anything in the fourth bullet, and only added a link for the third. Hopefully this is a reasonable compromise? /shrug |
||
|
||
# Recommendation: direnv | ||
|
||
[`direnv`](https://direnv.net/) is a tool for activating environments in your | ||
shell inside a given directory. Its support for poetry is unofficial (a | ||
community wiki recipe only), but works solidly in our experience. We thoroughly | ||
recommend it for daily use. To use it: | ||
|
||
1. [Install `direnv`](https://direnv.net/docs/installation.html) - it's likely | ||
packaged for your system already. | ||
2. Teach direnv about poetry. The [shell config here](https://github.com/direnv/direnv/wiki/Python#poetry) | ||
needs to be added to `~/.config/direnv/direnvrc` (or more generally `$XDG_CONFIG_HOME/direnv/direnvrc`). | ||
3. Mark the synapse checkout as a poetry project: `echo layout poetry > .envrc`. | ||
4. Convince yourself that you trust this `.envrc` configuration and project. | ||
Then formally confirm this to `direnv` by running `direnv allow`. | ||
|
||
Then whenever you navigate to the synapse checkout, you should be able to run | ||
e.g. `mypy` instead of `poetry run mypy`; `python` instead of | ||
`poetry run python`; and your shell commands will automatically be ran in the | ||
context of poetry's venv, without having to run `poetry shell` beforehand. | ||
|
||
# Rules of thumb: | ||
clokep marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- `poetry install --extras all` gets you into a good state. | ||
- `poetry install --extras all --remove-untracked` gets you into the precise | ||
locked state with no undeclared dependencies floating around. | ||
- Whenever you edit `pyproject.toml`, run `poetry lock --no-update` to keep them | ||
in sync. | ||
- `poetry run cmd args` when you need the python virtualenv context. | ||
`poetry shell` is roughly the same as activating the virtualenv, execpt it | ||
starts a new shell. (Though see the direnv recommendation above, which makes | ||
both commands unnecessary.) | ||
|
||
# How do I... | ||
|
||
## ...reset my venv to the locked environment? | ||
|
||
```shell | ||
poetry install --extras all --remove-untracked | ||
``` | ||
|
||
## ... inspect the `poetry` virtualenv? | ||
|
||
Some suggestions: | ||
|
||
```shell | ||
# Current env only | ||
poetry env info | ||
# All envs: this allows you to have e.g. a poetry managed venv for Python 3.7, | ||
# and another for Python 3.10. | ||
poetry env list --full-path | ||
poetry run pip list | ||
``` | ||
|
||
Note that `poetry show` describes the abstract *lock file* rather than your | ||
on-disk environment. With that said, `poetry show --tree` can sometimes be | ||
useful. | ||
|
||
|
||
## ...add a new dependency? | ||
|
||
Either: | ||
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else | ||
- `poetry add packagename`. See `poetry add --help`; note the `--dev`, | ||
`--extras` and `--optional` flags in particular. | ||
|
||
Include the updated `pyproject.toml` and `poetry.lock` files in your commit. | ||
|
||
## ...remove a dependency? | ||
|
||
I don't think we do this often, and I've not tried it myself, but | ||
DMRobertson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```shell | ||
poetry remove packagename | ||
``` | ||
|
||
ought to do the trick. Alternatively, manually update `pyproject.toml` and | ||
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock` | ||
files in your commit. | ||
|
||
## ...update the version range for an existing dependency? | ||
|
||
Best done by manually editing `pyproject.toml`, then `poetry lock --no-update`. | ||
Include the updated `pyproject.toml` and `poetry.lock` in your commit. | ||
|
||
## ...update a dependency in the locked environment? | ||
|
||
Use | ||
|
||
```shell | ||
poetry update packagename | ||
``` | ||
|
||
to use the latest version of `packagename` in the locked environment, without | ||
affecting the broad dependencies listed in the wheel. | ||
|
||
There doesn't seem to be away to do this whilst locking a _specific_ version of | ||
`packagename`. We can workaround this (crudely) as follows: | ||
|
||
```shell | ||
poetry add packagename==1.2.3 | ||
# This should update pyproject.lock. | ||
|
||
# Now undo the changes to pyproject.toml. For example | ||
# git restore pyproject.toml | ||
|
||
# Get poetry to recompute the content-hash of pyproject.toml without changing | ||
# the locked package versions. | ||
poetry lock --no-update | ||
``` | ||
|
||
Either way, include the updated `poetry.lock` file in your commit. | ||
|
||
## ...export a `requirements.txt` file? | ||
|
||
```shell | ||
poetry export --extras all | ||
``` | ||
|
||
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`. | ||
|
||
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may | ||
be required. | ||
|
||
## ...build a test wheel? | ||
|
||
I usually use | ||
|
||
```shell | ||
`poetry run pip install build && poetry run python -m build | ||
``` | ||
|
||
because [`build`](https://github.com/pypa/build) is a standardish tool which | ||
doesn't require poetry. (It's what we use in CI too). However, you could try | ||
`poetry build` too. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My only issue with sending people to the poetry docs is that installing by
pipx
is not recommended by them, which people (beginners especially) might find confusing, especially if something isn't working. Maybe add some words that acknowledge this?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed it's worth acknowledging this.
FWIW It is mentioned further down on that link, if not their primary recommendation: see https://python-poetry.org/docs/#installing-with-pipx. Though as you say that's a point of confusion.
Their main recommendation is to download a script with curl and pipe it into Python. (That allows poetry to upgrade itself, among other things.) I'm sure it's safe---the downloaded script is small, and we trust the poetry project---but I didn't want to encourage that behaviour.