[ci] move CI logic into shell scripts (fixes #114)

[jameslamb]

Fixes #114.

This PR proposes moving the logic currently contained in the project's CI (.circleci/config.yml) into shell scripts, for the reasons mentioned in #114:

• reduced code duplication across jobs
• reduced effort required to add, remove, and customize jobs
• lower-friction local develoopment

Changes

• moves commands for CI jobs into 2 shell scripts
  • .ci/setup.sh = install dependencies used by every CI job
  • .ci/test.sh = install hamilton's Python dependencies and run tests
• adds docs in developer_setup.md explaining how to re-use these scripts to run tests in Docker locally during development
• replaces uses of apt install with apt-get install
• switches pyspark job to using a circleci/python image for consistency with all the other CI jobs + ease of use in development

How I tested this

Ran shellcheck on the new scripts.

shellcheck .ci/setup.sh
shellcheck .ci/test.sh

Ran yamllint on the new CircleCI config.

yamllint .circleci/config.yml

Tried running some of the project's tests with different combinations of TASK and PYTHON_VERSION.

For example:

Dask tests on Python 3.9 (click me)

docker run \
  --rm \
  --entrypoint="" \
  -v "$(pwd)":/opt/testing \
  --workdir /opt/testing \
  --env TASK=dask \
  -it circleci/python:3.9 \
  /bin/bash -c '.ci/setup.sh && .ci/test.sh'

main tests on Python 3.10 (click me)

docker run \
  --rm \
  --entrypoint="" \
  -v "$(pwd)":/opt/testing \
  --workdir /opt/testing \
  --env TASK=tests \
  -it circleci/python:3.10 \
  /bin/bash -c '.ci/setup.sh && .ci/test.sh'

Notes

References on re-using configuration in CircleCI configs:

• https://circleci.com/docs/reusing-config/
• https://circleci.com/docs/configuration-reference/#matrix
• https://circleci.com/docs/env-vars/

Reference on why apt-get is preferred to apt for things like CI scripts:

• https://askubuntu.com/a/829869

Thanks for your time and consideration.

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[skrawcz]

Looking good! :) just minor nits.

[skrawcz]

@jameslamb mind if I squash merge? Or do you want to rebase to resolve conflicts?

[jameslamb]

@skrawcz sure, you can squash them! I just added you as a collaborator on my fork so you can do whatever you want there.

---

For your consideration... I personally feel like open source projects (at least here on GitHub) should just always use squash merging exclusively (one commit per PR, commit message = PR title) so you never have to think about commits on PRs.

Also means that if a PR gets behind the target branch, you can just git merge main to update it (instead of caring about rebasing).

That's been working great for us on LightGBM for years. Look how pretty this commit history is: https://github.com/microsoft/LightGBM/commits/master.

[jameslamb]

And to be specific, I mean doing this in settings:

[skrawcz]

Thanks this @jameslamb !

[jameslamb]

thanks so much for considering it!

[elijahbenizzy]

thanks so much for considering it!

Thanks for the contribution!