This code should work equivalently on MacOS, Linux, and Windows. However, it only works on Python 3.10 and earlier, because the asynctest library depends on @asyncio.coroutine, which was removed in Python 3.11.
This project uses Poetry to manage Python packaging and dependencies. Most day-to-day tasks (such as running unit tests from the command line) are orchestrated through Poetry.
A coding standard is enforced using Black, isort and Pylint. Python 3 type hinting is validated using MyPy. To reduce boilerplate, classes are defined using Attrs (see this rationale).
To add dependencies use poetry add package (for runtime dependencies) or poetry add --group dev package (for development environment dependencies).
To update dependencies, use poetry update. This will update all of the dependencies without taking you past any major version changes that are likely to be incompatible. If you want to update a single package, use poetry update package. If you want to update past a major version change, either specify the version like poetry add package=^2.0.3 or get the latest with poetry add package@latest. PEP 508 specifications are also supported.
I rely on pre-commit hooks to ensure that the code is properly-formatted,
clean, and type-safe when it's checked in. The run install step described
below installs the project pre-commit hooks into your repository. These hooks
are configured in .pre-commit-config.yaml.
If necessary, you can temporarily disable a hook using Git's --no-verify
switch. However, keep in mind that the CI build on GitHub enforces these
checks, so the build will fail.
The .gitattributes file controls line endings for the files
in this repository. It would be simplest to have files in the Git working copy
use native line endings. However, I develop this code on multiple platforms,
and the files in the published PyPI package get the line endings from the
working copy. If we use native line endings, the format of the published
package will vary depending on where the publish step was run. This is
confusing, and can cause problems for downstream users who expect the PyPI
package to have a consistent format. Instead of relying on automatic behavior,
the .gitattributes file forces most files to have UNIX line endings.
This generally works ok, except for the docs/requirements.txt file
generated by Poetry. Unfortunately, all of the files that Poetry generates
have native platform line endings, and you can't override that behavior. Even
with sane configuration in .gitattributes, you sometimes still get spurious
differences, where Git says that a file has changed but then git diff shows
an empty result. The run script and the pre-commit hooks both normalize the
line endings for requirements.txt. I wish there were a standard way to do
this in Poetry or in Python, but there isn't as of this writing.
Nearly all prerequisites are managed by Poetry. All you need to do is make sure that you have a working Python 3 enviroment and install Poetry itself.
The project is designed to work with Poetry >= 1.2.0. If you already have an older version of Poetry installed on your system, uninstall it before following the setup process below:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 - --uninstall
See the Announcing Poetry 1.2.0 blog post for more information.
On MacOS, it's easiest to use Homebrew to install Python:
brew install python3
Once that's done, make sure the python on your $PATH is Python 3 from
Homebrew (in /usr/local), rather than the standard Python 2 that comes with
MacOS.
Although Poetry can also be installed from Homebrew, it works better to use to official installer:
curl -sSL https://install.python-poetry.org | python3 -
Note: The installer prints the location of the installed
poetryscript. Make sure to add this to your$PATH, otherwise you won't be able to run it.
First, install Python 3 and related tools:
sudo apt-get install python3 python3-venv python3-pip
Next, make sure that the python interpreter on your $PATH is Python 3.
If you are using Debian bullseye or later (first released in August 2021),
then Python 2 is deprecated and Python 3 is the primary Python on your system.
However, by default there is only a python3 interpreter on your $PATH, not
a python interpreter. To add the python interpreter, use:
sudo apt-get install python-is-python3
For earlier releases of Debian where both Python 2 and Python 3 are available,
the process is a little more complicated. The approach I used before upgrading
to bullseye was based on update-alternatives, as discussed on
StackExchange.
Next, install Poetry using the official installer:
curl -sSL https://install.python-poetry.org | python3 -
Note: The installer prints the location of the installed
poetryscript. Make sure to add this to your$PATH, otherwise you won't be able to run it.
First, install Python 3 from your preferred source, either a standard
installer or a meta-installer like Chocolatey. Make sure the python
on your $PATH is Python 3.
Next, install Poetry using the official installer:
curl -sSL https://install.python-poetry.org | python -
Note: The installer prints the location of the installed
poetryscript. Make sure to add this to your$PATH, otherwise you won't be able to run it.
The development environment (the run script, etc.) expects a bash shell
to be available. On Windows, it works fine with the standard Git Bash.
The run script provides shortcuts for common developer tasks:
$ ./run --help
------------------------------------
Shortcuts for common developer tasks
------------------------------------
Basic tasks:
- run install: Setup the virtualenv via Poetry and install pre-commit hooks
- run format: Run the code formatters
- run checks: Run the code checkers
- run build: Build artifacts in the dist/ directory
- run test: Run the unit tests
- run test -c: Run the unit tests with coverage
- run test -ch: Run the unit tests with coverage and open the HTML report
- run suite: Run the complete test suite, as for the GitHub Actions CI build
Additional tasks:
- run demo: Run the demo client (requires running server)
- run docs: Build the Sphinx documentation for readthedocs.io
- run docs -o: Build the Sphinx documentation and open in a browser
- run publish: Publish the current code to PyPI and push to GitHub
- run release: Release a specific version and tag the code
- run requirements: Regenerate the docs/requirements.txt file
- run server: Start the websockets server
To run the server from the codebase for local testing, use the run server
command. This is equivalent to the installed apologies-server script.
$ ./run server --help
usage: apologies-server [-h] [--quiet] [--verbose] [--debug] [--config CONFIG]
[--logfile LOGFILE] [--override OVERRIDE]
Start the apologies server and let it run forever.
optional arguments:
-h, --help show this help message and exit
--quiet decrease log verbosity from INFO to ERROR
--verbose increase log verbosity from INFO to DEBUG
--debug like --verbose but also include websockets logs
--config CONFIG path to configuration on disk
--logfile LOGFILE path to logfile on disk (default is stdout)
--override OVERRIDE override a config parameter as "param:value"
By default, the server writes logs to stdout. If you prefer, you can specify
the path to a logfile, and logs will be written there instead. The default
configuration file is "/Users/kpronovici/.apologiesrc". If the default
configuration file is not found, default values will be set. If you override
the default config file, it must exist. You may override any individual config
parameter with "--override param:value".
The simplest way to start the server is with no arguments:
$ ./run server
2020-06-10 14:31:39,831Z --> [INFO ] Apologies server started
2020-06-10 14:31:39,832Z --> [INFO ] Configuration: {
"logfile_path": null,
"server_host": "localhost",
"server_port": 8080,
"close_timeout_sec": 10,
"websocket_limit": 1000,
"total_game_limit": 1000,
"in_progress_game_limit": 25,
"registered_player_limit": 100,
"websocket_idle_thresh_min": 2,
"websocket_inactive_thresh_min": 5,
"player_idle_thresh_min": 15,
"player_inactive_thresh_min": 30,
"game_idle_thresh_min": 10,
"game_inactive_thresh_min": 20,
"game_retention_thresh_min": 2880,
"idle_websocket_check_period_sec": 120,
"idle_websocket_check_delay_sec": 300,
"idle_player_check_period_sec": 120,
"idle_player_check_delay_sec": 300,
"idle_game_check_period_sec": 120,
"idle_game_check_delay_sec": 300,
"obsolete_game_check_period_sec": 300,
"obsolete_game_check_delay_sec": 300
}
2020-06-10 14:31:39,832Z --> [INFO ] Adding signal handlers...
2020-06-10 14:31:39,832Z --> [INFO ] Scheduling tasks...
2020-06-10 14:31:39,832Z --> [INFO ] Completed starting websocket server
The server displays its configuration when it boots. You can override any of
this configuration using the switches on the run server command.
While this is primarily a websockets server, it includes a quick'n'dirty demo that plays a game as a websockets client, to demonstrate the protocol and the websockets client code.
$ ./run demo --help
usage: demo [-h] [--quiet] [--verbose] [--debug] [--logfile LOGFILE]
[--host HOST] [--port PORT]
Start the apologies server demo client.
optional arguments:
-h, --help show this help message and exit
--quiet decrease log verbosity from INFO to ERROR
--verbose increase log verbosity from INFO to DEBUG
--debug like --verbose but also include websockets logs
--logfile LOGFILE path to logfile on disk (default is stdout)
--host HOST host where the server is running
--port PORT port where the server is running on the host
The client requires that you already have the server running. By default, the
client writes logs to stdout. If you prefer, you can specify the path to a
logfile, and logs will be written there instead.
To run the demo, you must also have a server running elsewhere. Your simplest option is to start the server in one window:
./run server
and the demo in another window:
./run demo
The demo registers a "human" player, starts a 4-player game (getting 3 programmatic players as opponents), and then runs through the entire game until it completes. To simulate "human" game play, each time it is the human player's turn, a move is chosen randomly. The programmatic opponents are played as usual by the server using its reward-based play choice algorithm.
The demo assumes it can register a player with handle leela. You should be
able to run the demo multiple times in a row against the server without any
problems, because it cleans up after itself properly. However, if you have
been making changes to the demo - or if it crashed or was interrupted - and the
leela handle is still registered, the demo will fail to run.
Currently, I use PyCharm Community Edition as my day-to-day IDE. By integrating Black and Pylint, most everything important that can be done from a shell environment can also be done right in PyCharm.
PyCharm offers a good developer experience. However, the underlying configuration on disk mixes together project policy (i.e. preferences about which test runner to use) with system-specific settings (such as the name and version of the active Python interpreter). This makes it impossible to commit complete PyCharm configuration to the Git repository. Instead, the repository contains partial configuration, and there are instructions below about how to manually configure the remaining items.
Before going any further, make sure sure that you have installed all of the system prerequisites discussed above. Then, make sure your environment is in working order. In particular, if you do not run the install step, there will be no virtualenv for PyCharm to use:
./run install && ./run suite
Once you have a working shell development environment, Open (do not
Import) the apologies-server directory in PyCharm, then follow the remaining
instructions below. By using Open, the existing .idea directory will be
retained and all of the existing settings will be used.
As a security precaution, PyCharm does not trust any virtual environment
installed within the repository, such as the Poetry .venv directory. In the
status bar on the bottom right, PyCharm will report No interpreter. Click
on this error and select Add Interpreter. In the resulting dialog, click
Ok to accept the selected environment, which should be the Poetry virtual
environment.
Go to the PyCharm settings and find the apologies-server project. Under
Project Structure, mark both src and tests as source folders. In
the Exclude Files box, enter the following:
LICENSE;NOTICE;PyPI.md;.coverage;.coveragerc;.github;.gitignore;.gitattributes;.htmlcov;.idea;.isort.cfg;.mypy.ini;.mypy_cache;.pre-commit-config.yaml;.pylintrc;.pytest_cache;.readthedocs.yml;.tabignore;build;dist;docs/_build;out;poetry.lock;poetry.toml;run;.run;.venv
When you're done, click Ok. Then, go to the gear icon in the project panel and uncheck Show Excluded Files. This will hide the files and directories in the list above.
In the PyCharm settings, go to Editor > Inspections and be sure that the Project Default profile is selected.
Unit tests are written using Pytest, and API documentation is written using Google Style Python Docstring. However, neither of these is the default in PyCharm. In the PyCharm settings, go to Tools > Python Integrated Tools. Under Testing > Default test runner, select pytest. Under Docstrings > Docstring format, select Google.
Right click on the tests folder in the project explorer and choose Run
'pytest in tests'. Make sure that all of the tests pass. If you see a slightly
different option (i.e. for "Unittest" instead of "pytest") then you probably
skipped the preferences setup discussed above. You may need to remove the
run configuration before PyCharm will find the right test suite.
Optionally, you might want to set up external tools for some of common developer tasks: code reformatting and the PyLint and MyPy checks. One nice advantage of doing this is that you can configure an output filter, which makes the Pylint and MyPy errors clickable. To set up external tools, go to PyCharm settings and find Tools > External Tools. Add the tools as described below.
On Linux or MacOS, you can set up the external tools to invoke the run script
directly.
For this to work, it's important that tools like poetry are on the system
path used by PyCharm. On Linux, depending on how you start PyCharm, your
normal shell environment may or may not be inherited. For instance, I had to
adjust the target of my LXDE desktop shortcut to be the script below, which
sources my profile before running the pycharm.sh shell script:
#!/bin/bash
source ~/.bash_profile
/opt/local/lib/pycharm/pycharm-community-2020.3.2/bin/pycharm.sh| Field | Value |
|---|---|
| Name | Format Code |
| Description | Run the code formatters |
| Group | Developer Tools |
| Program | $ProjectFileDir$/run |
| Arguments | format |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Checked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Unchecked |
| Make console active on message in stderr | Unchecked |
| Output filters | Empty |
| Field | Value |
|---|---|
| Name | Run MyPy Checks |
| Description | Run the MyPy code checks |
| Group | Developer Tools |
| Program | $ProjectFileDir$/run |
| Arguments | mypy |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN$:.* |
| Field | Value |
|---|---|
| Name | Run Pylint Checks |
| Description | Run the Pylint code checks |
| Group | Developer Tools |
| Program | $ProjectFileDir$/run |
| Arguments | pylint |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN.* |
On Windows, PyCharm has problems invoking the run script. The trick is to
invoke the Bash interpreter and tell it to invoke the run script. The
examples below assume that you have installed Git Bash in its standard location
under C:\Program Files\Git. If it is somewhere else on your system, just
change the path for bash.exe.
| Field | Value |
|---|---|
| Name | Format Code |
| Description | Run the code formatters |
| Group | Developer Tools |
| Program | powershell.exe |
| Arguments | `& 'C:\Program Files\Git\bin\bash.exe' "./run" format |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Checked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Unchecked |
| Make console active on message in stderr | Unchecked |
| Output filters | Empty |
| Field | Value |
|---|---|
| Name | Run MyPy Checks |
| Description | Run the MyPy code checks |
| Group | Developer Tools |
| Program | powershell.exe |
| Arguments | `& 'C:\Program Files\Git\bin\bash.exe' "./run" mypy |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN$:.* |
| Field | Value |
|---|---|
| Name | Run Pylint Checks |
| Description | Run the Pylint code checks |
| Group | Developer Tools |
| Program | powershell.exe |
| Arguments | `& 'C:\Program Files\Git\bin\bash.exe' "./run" pylint |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN.* |
Documentation at Read the Docs is generated via a GitHub hook each time code is pushed to master. So, there is no formal release process for the documentation.
Code is released to PyPI. There is a partially-automated process to publish a new release.
Note: In order to publish code, you must must have push permissions to the GitHub repo and be a collaborator on the PyPI project. Before running this process for the first time, you must set up a PyPI API token and configure Poetry to use it. (See notes below.)
Ensure that you are on the master branch. Releases must always be done from
master.
Ensure that the Changelog is up-to-date and reflects all of the changes that
will be published. The top line must show your version as unreleased:
Version 0.1.29 unreleased
Run the release step:
./run release 0.1.29
This updates pyproject.toml and the Changelog to reflect the released
version, then commits those changes and tags the code. Nothing has been pushed
or published yet, so you can always remove the tag (i.e. git tag -d v0.1.29)
and revert your commit (git reset HEAD~1) if you made a mistake.
Finally, publish the release:
./run publish
This builds the deployment artifacts, publishes the artifacts to PyPI, and pushes the repo to GitHub. The code will be available on PyPI for others to use after a little while.
In order to publish to PyPI, you must configure Poetry to use a PyPI API token. Once you have the token, you will configure Poetry to use it. Poetry relies on the Python keyring to store this secret. On MacOS and Windows, it will use the system keyring, and no other setup is required. If you are using Debian, the process is more complicated. See the notes below.
First, in your PyPI account settings, create an API token with upload permissions for the apologiesserver project. Once you have a working keyring, configure Poetry following the instructions:
poetry config pypi-token.pypi <the PyPI token>
Note that this leaves your actual secret in the command-line history, so make sure to scrub it once you're done.
On Debian, the process really only works from an X session. There is a way to manipulate the keyring without being in an X session, and I used to document it here. However, it's so ugly that I don't want to encourage anyone to use it. If you want to dig in on your own, see the keyring documentation under the section Using Keyring on headless Linux systems.
Some setup is required to initialize the keyring in your Debian system. First,
install the gnome-keyring package, and then log out:
$ sudo apt-get install gnome-keyring
$ exit
Log back in and initialize your keyring by setting and then removing a dummy value:
$ keyring set testvalue "user"
Password for 'user' in 'testvalue':
Please enter password for encrypted keyring:
$ keyring get testvalue "user"
Please enter password for encrypted keyring:
password
$ keyring del testvalue "user"
Deleting password for 'user' in 'testvalue':
At this point, the keyring should be fully functional and it should be ready for use with Poetry. Whenever Poetry needs to read a secret from the keyring, you'll get a popup window where you need to enter the keyring password.