Setting up a new worker

Adding pyproject.toml and generating a poetry.lock based on it

This assumes that you already created a GitHub repo for the new worker.

1. If you are running Windows, set up WSL
2. Make sure Poetry is installed in WSL.
3. Clone the repository and navigate to it within WSL.
4. Copy the pyproject.toml found in this repo to your repo and change the details found underneath [tool.poetry] to match your project's description
5. Run poetry install to generate a virtual environment and a poetry.lock file
6. Add any additional dependencies required - either using poetry add <package-name> or by adding them to pyproject.toml and rerunning poetry install.

Setting up workflows/GitHub actions

1. Copy the folder .github from this repository (including its contents)
2. Comment out the deploy job from main-branch.yml and manual-deploy.yml (if Docker is not set up yet - once it's set up, this needs to be uncommented and refined accordingly)
3. Comment out the pytest task inside _test.yml (if there's no config file for it yet - uncomment and refine once (unit) tests are added)
4. Add an empty .py file (recommended: main_data_processor.py) so that mypy doesn't make the workflow pipeline fail

Adding Dockerfile and testing if it runs

• Copy the Dockerfile from dane-audio-extraction-worker

• Copy also the docker-entrypoint.sh from that same repo

• For Docker to work with WSL, I followed this

• Additionally, I had to do the following steps to be able to run docker build:

  1. Create the docker group: sudo groupadd docker

  2. Add your user to the docker group: sudo usermod -aG docker ${USER}

  3. You would need to log out and log back in so that your group membership is re-evaluated. You can just exit WSL, then start it up again and it will work.

• Lastly, to test whether the image builds or not, do:

docker build . -t <your-dane-worker>

Inside your repository's directory.